2020-11-16T18:38:12.477 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-16T18:38:12.478 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631108 --description rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/osd} --block --verbose --lock --name kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi -- /tmp/teuthology-worker.w5je1xml.tmp 2020-11-16T18:38:12.497 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-16T18:38:12.552 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631108 branch: wip-kefu-testing-2020-11-16-1845 description: rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/osd} email: tchaikov@gmail.com first_in_suite: false job_id: '5631108' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-kefu-testing-2020-11-16-1845 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - Health detail log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - Health detail sha1: 911cc35794d871071a8bebfb0919482263b1d18b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 911cc35794d871071a8bebfb0919482263b1d18b workunit: branch: wip-kefu-testing-2020-11-16-1845 sha1: 911cc35794d871071a8bebfb0919482263b1d18b owner: scheduled_kchai@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 911cc35794d871071a8bebfb0919482263b1d18b sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2020-11-16-1845 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-11-16-1845/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 911cc35794d871071a8bebfb0919482263b1d18b tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master timestamp: 2020-11-16_12:11:03 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.11583 2020-11-16T18:38:12.552 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-11-16-1845/qa; will attempt to use it 2020-11-16T18:38:12.553 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-11-16-1845/qa/tasks 2020-11-16T18:38:12.553 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-16T18:38:12.555 INFO:teuthology.task.internal:Checking packages... 2020-11-16T18:38:12.575 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '911cc35794d871071a8bebfb0919482263b1d18b' 2020-11-16T18:38:12.575 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-16T18:38:12.576 INFO:teuthology.packaging:ref: None 2020-11-16T18:38:12.576 INFO:teuthology.packaging:tag: None 2020-11-16T18:38:12.576 INFO:teuthology.packaging:branch: wip-kefu-testing-2020-11-16-1845 2020-11-16T18:38:12.577 INFO:teuthology.packaging:sha1: 911cc35794d871071a8bebfb0919482263b1d18b 2020-11-16T18:38:12.577 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-kefu-testing-2020-11-16-1845 2020-11-16T18:38:12.737 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7219-g911cc357-1bionic 2020-11-16T18:38:12.738 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-16T18:38:12.749 INFO:teuthology.task.internal:no buildpackages task found 2020-11-16T18:38:12.749 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-16T18:38:16.560 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-16T18:38:16.561 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-16T18:38:16.618 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:38:26.635 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:38:36.650 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:38:46.676 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:38:56.689 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:39:06.707 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:39:16.725 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:39:26.739 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:39:36.756 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:39:46.772 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:39:56.790 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:40:06.808 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:40:16.821 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:40:26.830 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:40:36.838 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:40:46.847 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:40:56.855 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:41:06.872 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:41:16.886 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:41:26.902 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:41:36.918 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:41:46.933 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:41:56.949 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:42:06.965 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:42:16.981 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:42:26.997 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:42:37.014 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:42:47.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:42:57.045 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:43:07.061 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:43:17.078 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:43:27.093 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:43:37.109 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:43:47.126 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:43:57.137 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T18:44:07.154 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T19:22:23.788 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T19:22:33.805 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T19:22:44.023 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T19:22:54.037 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 1, have 0)... 2020-11-16T19:23:04.054 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_kchai@teuthology', 'count': 1, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631108'} 2020-11-16T19:23:04.071 DEBUG:teuthology.lock.ops:locked ubuntu@smithi189.front.sepia.ceph.com 2020-11-16T19:23:04.102 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1c24378268>, signals=[15]) 2020-11-16T19:23:04.289 INFO:teuthology.provision.fog.smithi189:Scheduling deploy of ubuntu 18.04 2020-11-16T19:23:04.542 INFO:teuthology.orchestra.console:Power off smithi189 2020-11-16T19:23:04.543 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-16T19:23:04.568 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T19:23:08.594 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T19:23:12.706 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T19:23:16.818 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T19:23:17.030 INFO:teuthology.orchestra.console:Power off for smithi189 completed 2020-11-16T19:23:17.131 INFO:teuthology.orchestra.console:Power on smithi189 2020-11-16T19:23:17.131 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-16T19:23:17.153 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T19:23:21.178 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T19:23:25.290 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T19:23:25.501 INFO:teuthology.orchestra.console:Power on for smithi189 completed 2020-11-16T19:23:25.603 INFO:teuthology.provision.fog.smithi189:Waiting for deploy to finish 2020-11-16T19:26:26.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:26:51.316 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:27:00.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:27:09.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:27:18.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:27:27.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:27:36.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:28:30.498 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-16T19:28:30.500 INFO:teuthology.orchestra.run.smithi189:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-16T19:28:30.847 INFO:teuthology.provision.fog.smithi189:Node is ready 2020-11-16T19:28:30.847 INFO:teuthology.orchestra.run.smithi189:> hostname 2020-11-16T19:28:30.894 INFO:teuthology.orchestra.run.smithi189.stdout:smithi189.front.sepia.ceph.com 2020-11-16T19:28:30.895 INFO:teuthology.orchestra.run.smithi189:> grep smithi189.front.sepia.ceph.com /etc/hosts 2020-11-16T19:28:30.943 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189 smithi189.front.sepia.ceph.com smithi189 2020-11-16T19:28:30.943 INFO:teuthology.orchestra.run.smithi189:> sudo hostname smithi189 2020-11-16T19:28:31.074 INFO:teuthology.orchestra.run.smithi189:> sudo sed -i -e 's/smithi189.front.sepia.ceph.com/smithi189/g' /etc/hosts 2020-11-16T19:28:31.087 INFO:teuthology.orchestra.run.smithi189:> sudo sed -i -e 's/172.21.15.189/172.21.15.189/g' /etc/hosts 2020-11-16T19:28:31.144 INFO:teuthology.orchestra.run.smithi189:> sudo sed -i -e 's/smithi189.front.sepia.ceph.com/smithi189/g' /etc/hostname 2020-11-16T19:28:31.198 INFO:teuthology.orchestra.run.smithi189:> sudo hostname smithi189 2020-11-16T19:28:31.254 INFO:teuthology.provision.fog.smithi189:Deploy complete! 2020-11-16T19:28:31.255 DEBUG:teuthology.parallel:result is None 2020-11-16T19:28:31.382 INFO:teuthology.lock.ops:Checking smithi189.front.sepia.ceph.com 2020-11-16T19:28:31.382 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-16T19:28:31.400 INFO:teuthology.lock.ops:Updating [smithi189.front.sepia.ceph.com]: set os type and version on server 2020-11-16T19:28:31.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T19:28:31.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:28:31.469 INFO:teuthology.orchestra.run.smithi189:> true 2020-11-16T19:28:31.639 INFO:teuthology.orchestra.run.smithi189:> uname -m 2020-11-16T19:28:31.686 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2020-11-16T19:28:31.687 INFO:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2020-11-16T19:28:31.734 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="Ubuntu" 2020-11-16T19:28:31.734 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-16T19:28:31.734 INFO:teuthology.orchestra.run.smithi189.stdout:ID=ubuntu 2020-11-16T19:28:31.735 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE=debian 2020-11-16T19:28:31.735 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-16T19:28:31.735 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="18.04" 2020-11-16T19:28:31.735 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-16T19:28:31.735 INFO:teuthology.orchestra.run.smithi189.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-16T19:28:31.735 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-16T19:28:31.735 INFO:teuthology.orchestra.run.smithi189.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-16T19:28:31.736 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_CODENAME=bionic 2020-11-16T19:28:31.736 INFO:teuthology.orchestra.run.smithi189.stdout:UBUNTU_CODENAME=bionic 2020-11-16T19:28:31.737 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2020-11-16T19:28:31.753 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-11-16T19:28:31.769 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi189.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDj0wu+uAGwsapoYQIycPqbqbdkRwifV3HNXAqci94rFdBWlzz3MLAT1pYpZSXqczEcIOi1LZMsMbpzyU0A3lNjqMjjJGhXsrgKAQnnIxz8rGvJ8cPc7xjCC6hL7lCicsIPunNVz84NreL5puh13SN3dJf1bFaVrooCnDQO1vPzrqNDUjCE62gUSvAlA+elO0ixgFX5nC9Jpr63sLhr6AxBpdK8i4KjP1Yp7/sIRitic13VXBpjMsMyhIKYsvQFaP+2lzYPMaPXOlPSHjs2J2PfWOYCnF0pr8IHfdAebf+Y9NYH9qKTpDbpgqJ0q9/8phBBnWRLFjIhb1EdgWj7NV+L 2020-11-16T19:28:31.769 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-16T19:28:31.814 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-16T19:28:31.823 INFO:teuthology.task.internal:Saving configuration 2020-11-16T19:28:31.832 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-16T19:28:31.857 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-16T19:28:31.872 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-16 19:23:04.060853', 'description': '/home/teuthworker/archive/kchai-2020-11-16_12:11:03-rados-wip-kefu-testing-2020-11-16-1845-distro-basic-smithi/5631108', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIERZP/pYIwzqQKSgUj2cqybeJylIzmvF6bl/xpJ5bdde', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi189.front.sepia.ceph.com'} 2020-11-16T19:28:31.872 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-16T19:28:31.879 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-11-16T19:28:31.879 INFO:teuthology.run_tasks:Running task console_log... 2020-11-16T19:28:31.918 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1c27e02a60>, signals=[15]) 2020-11-16T19:28:31.919 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-16T19:28:31.926 INFO:teuthology.task.internal:Opening connections... 2020-11-16T19:28:31.926 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2020-11-16T19:28:31.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:28:32.001 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-16T19:28:32.007 INFO:teuthology.orchestra.run.smithi189:> uname -m 2020-11-16T19:28:32.243 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2020-11-16T19:28:32.244 INFO:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2020-11-16T19:28:32.290 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="Ubuntu" 2020-11-16T19:28:32.291 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-16T19:28:32.291 INFO:teuthology.orchestra.run.smithi189.stdout:ID=ubuntu 2020-11-16T19:28:32.291 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE=debian 2020-11-16T19:28:32.291 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-16T19:28:32.291 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="18.04" 2020-11-16T19:28:32.291 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-16T19:28:32.291 INFO:teuthology.orchestra.run.smithi189.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-16T19:28:32.292 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-16T19:28:32.292 INFO:teuthology.orchestra.run.smithi189.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-16T19:28:32.292 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_CODENAME=bionic 2020-11-16T19:28:32.292 INFO:teuthology.orchestra.run.smithi189.stdout:UBUNTU_CODENAME=bionic 2020-11-16T19:28:32.292 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2020-11-16T19:28:32.308 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-16T19:36:46.565 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-16T19:36:46.577 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-16T19:36:46.577 INFO:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2020-11-16T19:36:46.584 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-16T19:36:46.609 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-16T19:36:46.610 INFO:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2020-11-16T19:36:46.647 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-16T19:36:46.675 INFO:teuthology.run_tasks:Running task kernel... 2020-11-16T19:36:46.693 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2020-11-16T19:36:46.694 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-16T19:36:46.694 INFO:teuthology.orchestra.run.smithi189:> uname -r 2020-11-16T19:36:46.700 INFO:teuthology.orchestra.run.smithi189.stdout:4.15.0-123-generic 2020-11-16T19:36:46.701 INFO:teuthology.task.kernel:Running kernel on smithi189: 4.15.0-123-generic 2020-11-16T19:36:46.701 INFO:teuthology.orchestra.run.smithi189:> sudo apt-get clean 2020-11-16T19:36:46.823 INFO:teuthology.orchestra.run.smithi189:> sudo apt-get update 2020-11-16T19:36:47.104 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-16T19:36:47.104 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-16T19:36:47.168 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-16T19:36:47.439 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-16T19:36:47.619 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,162 kB] 2020-11-16T19:36:47.970 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,761 kB] 2020-11-16T19:36:48.088 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [373 kB] 2020-11-16T19:36:48.104 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [17.4 kB] 2020-11-16T19:36:48.104 INFO:teuthology.orchestra.run.smithi189.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [196 kB] 2020-11-16T19:36:48.113 INFO:teuthology.orchestra.run.smithi189.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [26.3 kB] 2020-11-16T19:36:48.114 INFO:teuthology.orchestra.run.smithi189.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,692 kB] 2020-11-16T19:36:48.165 INFO:teuthology.orchestra.run.smithi189.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,547 kB] 2020-11-16T19:36:48.214 INFO:teuthology.orchestra.run.smithi189.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [355 kB] 2020-11-16T19:36:48.223 INFO:teuthology.orchestra.run.smithi189.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [36.2 kB] 2020-11-16T19:36:48.224 INFO:teuthology.orchestra.run.smithi189.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.4 kB] 2020-11-16T19:36:48.224 INFO:teuthology.orchestra.run.smithi189.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,308 B] 2020-11-16T19:36:49.020 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 7,439 kB in 2s (3,803 kB/s) 2020-11-16T19:36:50.529 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-16T19:36:50.574 INFO:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-16T19:36:50.655 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-16T19:36:50.885 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2020-11-16T19:36:50.885 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2020-11-16T19:36:50.964 INFO:teuthology.orchestra.run.smithi189.stderr:E: Unable to locate package linux-image-current-generic 2020-11-16T19:36:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-16T19:36:50.967 INFO:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-16T19:36:51.047 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-16T19:36:51.287 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2020-11-16T19:36:51.288 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2020-11-16T19:36:51.462 INFO:teuthology.orchestra.run.smithi189.stdout:The following additional packages will be installed: 2020-11-16T19:36:51.462 INFO:teuthology.orchestra.run.smithi189.stdout: linux-generic linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-16T19:36:51.462 INFO:teuthology.orchestra.run.smithi189.stdout: linux-headers-generic linux-image-4.15.0-124-generic 2020-11-16T19:36:51.463 INFO:teuthology.orchestra.run.smithi189.stdout: linux-modules-4.15.0-124-generic linux-modules-extra-4.15.0-124-generic 2020-11-16T19:36:51.463 INFO:teuthology.orchestra.run.smithi189.stdout:Suggested packages: 2020-11-16T19:36:51.464 INFO:teuthology.orchestra.run.smithi189.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-11-16T19:36:51.464 INFO:teuthology.orchestra.run.smithi189.stdout:Recommended packages: 2020-11-16T19:36:51.464 INFO:teuthology.orchestra.run.smithi189.stdout: thermald 2020-11-16T19:36:51.490 INFO:teuthology.orchestra.run.smithi189.stdout:The following NEW packages will be installed: 2020-11-16T19:36:51.490 INFO:teuthology.orchestra.run.smithi189.stdout: linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-16T19:36:51.490 INFO:teuthology.orchestra.run.smithi189.stdout: linux-image-4.15.0-124-generic linux-modules-4.15.0-124-generic 2020-11-16T19:36:51.491 INFO:teuthology.orchestra.run.smithi189.stdout: linux-modules-extra-4.15.0-124-generic 2020-11-16T19:36:51.491 INFO:teuthology.orchestra.run.smithi189.stdout:The following packages will be upgraded: 2020-11-16T19:36:51.492 INFO:teuthology.orchestra.run.smithi189.stdout: linux-generic linux-headers-generic linux-image-generic 2020-11-16T19:36:51.667 INFO:teuthology.orchestra.run.smithi189.stdout:3 upgraded, 5 newly installed, 0 to remove and 1 not upgraded. 2020-11-16T19:36:51.667 INFO:teuthology.orchestra.run.smithi189.stdout:Need to get 65.9 MB of archives. 2020-11-16T19:36:51.668 INFO:teuthology.orchestra.run.smithi189.stdout:After this operation, 333 MB of additional disk space will be used. 2020-11-16T19:36:51.668 INFO:teuthology.orchestra.run.smithi189.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-124-generic amd64 4.15.0-124.127 [13.1 MB] 2020-11-16T19:36:52.648 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-124-generic amd64 4.15.0-124.127 [8,012 kB] 2020-11-16T19:36:52.895 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-124-generic amd64 4.15.0-124.127 [32.8 MB] 2020-11-16T19:36:53.812 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.124.111 [1,868 B] 2020-11-16T19:36:53.813 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.124.111 [2,624 B] 2020-11-16T19:36:53.813 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124 all 4.15.0-124.127 [10.9 MB] 2020-11-16T19:36:54.126 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124-generic amd64 4.15.0-124.127 [1,084 kB] 2020-11-16T19:36:54.138 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.124.111 [2,520 B] 2020-11-16T19:36:54.629 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 65.9 MB in 3s (25.0 MB/s) 2020-11-16T19:36:54.791 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-modules-4.15.0-124-generic. 2020-11-16T19:36:54.829 INFO:teuthology.orchestra.run.smithi189.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102077 files and directories currently installed.) 2020-11-16T19:36:54.832 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../0-linux-modules-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-16T19:36:54.876 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-16T19:36:56.485 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-image-4.15.0-124-generic. 2020-11-16T19:36:56.497 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../1-linux-image-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-16T19:36:56.595 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-16T19:36:57.138 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-124-generic. 2020-11-16T19:36:57.150 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-16T19:36:57.188 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-16T19:37:00.940 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../3-linux-generic_4.15.0.124.111_amd64.deb ... 2020-11-16T19:37:01.092 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-16T19:37:01.415 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.124.111_amd64.deb ... 2020-11-16T19:37:01.537 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-image-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-16T19:37:01.873 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-headers-4.15.0-124. 2020-11-16T19:37:01.885 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../5-linux-headers-4.15.0-124_4.15.0-124.127_all.deb ... 2020-11-16T19:37:01.915 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-16T19:37:06.397 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-headers-4.15.0-124-generic. 2020-11-16T19:37:06.419 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../6-linux-headers-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-16T19:37:06.471 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-16T19:37:08.786 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.124.111_amd64.deb ... 2020-11-16T19:37:08.908 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-16T19:37:09.279 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-16T19:37:09.820 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-16T19:37:10.267 INFO:teuthology.orchestra.run.smithi189.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-124-generic 2020-11-16T19:37:10.267 INFO:teuthology.orchestra.run.smithi189.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-124-generic 2020-11-16T19:37:10.406 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-16T19:37:10.915 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-16T19:37:11.042 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-image-generic (4.15.0.124.111) ... 2020-11-16T19:37:11.167 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-16T19:37:11.294 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-generic (4.15.0.124.111) ... 2020-11-16T19:37:11.419 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-generic (4.15.0.124.111) ... 2020-11-16T19:37:11.546 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-16T19:37:11.768 INFO:teuthology.orchestra.run.smithi189.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-11-16T19:37:11.768 INFO:teuthology.orchestra.run.smithi189.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-124-generic 2020-11-16T19:37:26.829 INFO:teuthology.orchestra.run.smithi189.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-11-16T19:37:26.830 INFO:teuthology.orchestra.run.smithi189.stdout:Sourcing file `/etc/default/grub' 2020-11-16T19:37:26.858 INFO:teuthology.orchestra.run.smithi189.stdout:Generating grub configuration file ... 2020-11-16T19:37:27.210 INFO:teuthology.orchestra.run.smithi189.stdout:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-16T19:37:27.238 INFO:teuthology.orchestra.run.smithi189.stdout:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-16T19:37:27.555 INFO:teuthology.orchestra.run.smithi189.stdout:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-16T19:37:27.582 INFO:teuthology.orchestra.run.smithi189.stdout:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-16T19:37:28.445 INFO:teuthology.orchestra.run.smithi189.stdout:done 2020-11-16T19:37:28.813 INFO:teuthology.orchestra.run.smithi189:> dpkg -s linux-image-generic 2020-11-16T19:37:28.838 INFO:teuthology.orchestra.run.smithi189.stdout:Package: linux-image-generic 2020-11-16T19:37:28.838 INFO:teuthology.orchestra.run.smithi189.stdout:Status: install ok installed 2020-11-16T19:37:28.839 INFO:teuthology.orchestra.run.smithi189.stdout:Priority: optional 2020-11-16T19:37:28.839 INFO:teuthology.orchestra.run.smithi189.stdout:Section: kernel 2020-11-16T19:37:28.839 INFO:teuthology.orchestra.run.smithi189.stdout:Installed-Size: 16 2020-11-16T19:37:28.839 INFO:teuthology.orchestra.run.smithi189.stdout:Maintainer: Ubuntu Kernel Team 2020-11-16T19:37:28.839 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: amd64 2020-11-16T19:37:28.839 INFO:teuthology.orchestra.run.smithi189.stdout:Source: linux-meta 2020-11-16T19:37:28.839 INFO:teuthology.orchestra.run.smithi189.stdout:Version: 4.15.0.124.111 2020-11-16T19:37:28.840 INFO:teuthology.orchestra.run.smithi189.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-16T19:37:28.840 INFO:teuthology.orchestra.run.smithi189.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-16T19:37:28.840 INFO:teuthology.orchestra.run.smithi189.stdout:Recommends: thermald 2020-11-16T19:37:28.840 INFO:teuthology.orchestra.run.smithi189.stdout:Description: Generic Linux kernel image 2020-11-16T19:37:28.840 INFO:teuthology.orchestra.run.smithi189.stdout: This package will always depend on the latest generic kernel image 2020-11-16T19:37:28.841 INFO:teuthology.orchestra.run.smithi189.stdout: available. 2020-11-16T19:37:28.841 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-123-generic Expected: 4.15.0-124-generic 2020-11-16T19:37:28.841 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-11-16T19:37:28.842 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-11-16T19:37:28.842 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-11-16T19:37:28.842 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=None, version=distro) 2020-11-16T19:37:28.843 INFO:teuthology.orchestra.run.smithi189:> sudo apt-get clean 2020-11-16T19:37:28.937 INFO:teuthology.orchestra.run.smithi189:> sudo apt-get update 2020-11-16T19:37:29.203 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-16T19:37:29.204 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-16T19:37:29.271 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-16T19:37:29.353 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-16T19:37:31.072 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-16T19:37:31.115 INFO:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-16T19:37:31.240 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-16T19:37:31.485 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2020-11-16T19:37:31.486 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2020-11-16T19:37:31.571 INFO:teuthology.orchestra.run.smithi189.stderr:E: Unable to locate package linux-image-current-generic 2020-11-16T19:37:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-16T19:37:31.574 INFO:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-16T19:37:31.652 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-16T19:37:31.911 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2020-11-16T19:37:31.912 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2020-11-16T19:37:32.112 INFO:teuthology.orchestra.run.smithi189.stdout:linux-image-generic is already the newest version (4.15.0.124.111). 2020-11-16T19:37:32.112 INFO:teuthology.orchestra.run.smithi189.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2020-11-16T19:37:32.113 INFO:teuthology.orchestra.run.smithi189:> dpkg -s linux-image-generic 2020-11-16T19:37:32.137 INFO:teuthology.orchestra.run.smithi189.stdout:Package: linux-image-generic 2020-11-16T19:37:32.138 INFO:teuthology.orchestra.run.smithi189.stdout:Status: install ok installed 2020-11-16T19:37:32.138 INFO:teuthology.orchestra.run.smithi189.stdout:Priority: optional 2020-11-16T19:37:32.138 INFO:teuthology.orchestra.run.smithi189.stdout:Section: kernel 2020-11-16T19:37:32.139 INFO:teuthology.orchestra.run.smithi189.stdout:Installed-Size: 16 2020-11-16T19:37:32.139 INFO:teuthology.orchestra.run.smithi189.stdout:Maintainer: Ubuntu Kernel Team 2020-11-16T19:37:32.139 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: amd64 2020-11-16T19:37:32.139 INFO:teuthology.orchestra.run.smithi189.stdout:Source: linux-meta 2020-11-16T19:37:32.140 INFO:teuthology.orchestra.run.smithi189.stdout:Version: 4.15.0.124.111 2020-11-16T19:37:32.140 INFO:teuthology.orchestra.run.smithi189.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-16T19:37:32.140 INFO:teuthology.orchestra.run.smithi189.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-16T19:37:32.140 INFO:teuthology.orchestra.run.smithi189.stdout:Recommends: thermald 2020-11-16T19:37:32.140 INFO:teuthology.orchestra.run.smithi189.stdout:Description: Generic Linux kernel image 2020-11-16T19:37:32.141 INFO:teuthology.orchestra.run.smithi189.stdout: This package will always depend on the latest generic kernel image 2020-11-16T19:37:32.141 INFO:teuthology.orchestra.run.smithi189.stdout: available. 2020-11-16T19:37:32.141 INFO:teuthology.orchestra.run.smithi189:> mktemp 2020-11-16T19:37:32.186 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.ygOWrbmnSb 2020-11-16T19:37:32.187 INFO:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ygOWrbmnSb 2020-11-16T19:37:32.244 INFO:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.ygOWrbmnSb 2020-11-16T19:37:32.366 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.ygOWrbmnSb is 8KB 2020-11-16T19:37:32.378 INFO:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.ygOWrbmnSb 2020-11-16T19:37:32.385 INFO:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-11-16T19:37:32.444 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-16T19:37:32.445 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-16T19:37:32.446 INFO:teuthology.orchestra.run.smithi189:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-16T19:37:32.490 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-16T19:37:32.504 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-16T19:37:32.505 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-16T19:37:32.505 INFO:teuthology.orchestra.run.smithi189.stderr:93 bytes copied, 3.4001e-05 s, 2.7 MB/s 2020-11-16T19:37:32.505 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-16T19:37:32.511 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-124-generic 2020-11-16T19:37:32.511 INFO:teuthology.orchestra.run.smithi189:> sudo update-grub 2020-11-16T19:37:33.816 INFO:teuthology.orchestra.run.smithi189.stderr:Sourcing file `/etc/default/grub' 2020-11-16T19:37:33.830 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2020-11-16T19:37:34.065 INFO:teuthology.orchestra.run.smithi189.stderr:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-16T19:37:34.077 INFO:teuthology.orchestra.run.smithi189.stderr:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-16T19:37:34.384 INFO:teuthology.orchestra.run.smithi189.stderr:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-16T19:37:34.397 INFO:teuthology.orchestra.run.smithi189.stderr:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-16T19:37:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:done 2020-11-16T19:37:34.987 INFO:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2020-11-16T19:38:05.019 INFO:teuthology.misc:Re-opening connections... 2020-11-16T19:38:05.019 INFO:teuthology.misc:trying to connect to ubuntu@smithi189.front.sepia.ceph.com 2020-11-16T19:38:05.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T19:38:05.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:38:23.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2020-11-16T19:38:23.423 DEBUG:teuthology.misc:waited 18.40405535697937 2020-11-16T19:38:24.424 INFO:teuthology.misc:trying to connect to ubuntu@smithi189.front.sepia.ceph.com 2020-11-16T19:38:24.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T19:38:24.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:38:27.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2020-11-16T19:38:27.487 DEBUG:teuthology.misc:waited 22.467379093170166 2020-11-16T19:38:28.487 INFO:teuthology.misc:trying to connect to ubuntu@smithi189.front.sepia.ceph.com 2020-11-16T19:38:28.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T19:38:28.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:38:30.559 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2020-11-16T19:38:30.560 DEBUG:teuthology.misc:waited 25.540536165237427 2020-11-16T19:38:31.560 INFO:teuthology.misc:trying to connect to ubuntu@smithi189.front.sepia.ceph.com 2020-11-16T19:38:31.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T19:38:31.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:38:33.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2020-11-16T19:38:33.630 DEBUG:teuthology.misc:waited 28.611205577850342 2020-11-16T19:38:34.631 INFO:teuthology.misc:trying to connect to ubuntu@smithi189.front.sepia.ceph.com 2020-11-16T19:38:34.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T19:38:34.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:38:36.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2020-11-16T19:38:36.702 DEBUG:teuthology.misc:waited 31.683197498321533 2020-11-16T19:38:37.703 INFO:teuthology.misc:trying to connect to ubuntu@smithi189.front.sepia.ceph.com 2020-11-16T19:38:37.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T19:38:37.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:38:39.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2020-11-16T19:38:39.774 DEBUG:teuthology.misc:waited 34.755213022232056 2020-11-16T19:38:40.775 INFO:teuthology.misc:trying to connect to ubuntu@smithi189.front.sepia.ceph.com 2020-11-16T19:38:40.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T19:38:40.777 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:38:42.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2020-11-16T19:38:42.846 DEBUG:teuthology.misc:waited 37.82717227935791 2020-11-16T19:38:43.847 INFO:teuthology.misc:trying to connect to ubuntu@smithi189.front.sepia.ceph.com 2020-11-16T19:38:43.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T19:38:43.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:39:43.850 DEBUG:teuthology.orchestra.remote:timed out 2020-11-16T19:39:43.852 DEBUG:teuthology.misc:waited 98.83222937583923 2020-11-16T19:39:44.852 INFO:teuthology.misc:trying to connect to ubuntu@smithi189.front.sepia.ceph.com 2020-11-16T19:39:44.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T19:39:44.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T19:39:44.930 INFO:teuthology.orchestra.run.smithi189:> true 2020-11-16T19:39:45.275 DEBUG:teuthology.misc:waited 100.25527310371399 2020-11-16T19:39:46.276 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-11-16T19:39:46.276 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.15.0-124-generic"... 2020-11-16T19:39:46.277 INFO:teuthology.orchestra.run.smithi189:> uname -r 2020-11-16T19:39:46.283 INFO:teuthology.orchestra.run.smithi189.stdout:4.15.0-124-generic 2020-11-16T19:39:46.283 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-124-generic vs 4.15.0-124-generic 2020-11-16T19:39:46.283 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-11-16T19:39:46.284 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-11-16T19:39:47.285 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-11-16T19:39:47.285 INFO:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-16T19:39:47.327 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2020-11-16T19:39:47.357 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-16T19:39:47.365 INFO:teuthology.task.internal:Creating test directory... 2020-11-16T19:39:47.365 INFO:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-16T19:39:47.404 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-16T19:39:47.414 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-16T19:39:47.426 INFO:teuthology.task.internal:Creating archive directory... 2020-11-16T19:39:47.426 INFO:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-16T19:39:47.461 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-16T19:39:47.469 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-16T19:39:47.469 INFO:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-16T19:39:47.522 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T19:39:47.530 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T19:39:47.531 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-16T19:39:47.542 INFO:teuthology.task.internal:Configuring sudo... 2020-11-16T19:39:47.542 INFO:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-16T19:39:47.590 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-16T19:39:47.600 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-16T19:39:47.601 INFO:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-16T19:39:47.636 INFO:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-16T19:39:47.685 INFO:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-16T19:39:47.732 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-16T19:39:47.733 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-16T19:39:47.778 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-16T19:39:47.799 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-16T19:39:47.799 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-16T19:39:47.800 INFO:teuthology.orchestra.run.smithi189.stderr:152 bytes copied, 3.2055e-05 s, 4.7 MB/s 2020-11-16T19:39:47.800 INFO:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2020-11-16T19:39:47.884 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-16T19:39:47.894 INFO:teuthology.task.internal:Starting timer... 2020-11-16T19:39:47.894 INFO:teuthology.run_tasks:Running task pcp... 2020-11-16T19:39:47.924 INFO:teuthology.run_tasks:Running task selinux... 2020-11-16T19:39:47.939 DEBUG:teuthology.task.selinux:Excluding smithi189: OS 'ubuntu' does not support SELinux 2020-11-16T19:39:47.939 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-16T19:39:47.940 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-11-16T19:39:47.940 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-16T19:39:47.940 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-16T19:39:47.953 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-11-16T19:39:47.960 INFO:teuthology.repo_utils:Fetching master from origin 2020-11-16T19:39:48.008 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-16T19:39:48.023 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-16T19:39:48.025 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-16T19:39:52.678 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-16T19:40:02.116 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-16T19:40:08.788 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-11-16T19:40:08.810 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:40:08 +0000 (0:00:00.075) 0:00:00.075 ******* 2020-11-16T19:48:50.488 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi189.front.sepia.ceph.com] 2020-11-16T19:48:50.560 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:48:50 +0000 (0:08:41.749) 0:08:41.824 ******* 2020-11-16T19:48:50.815 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-11-16T19:48:50.818 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-16T19:48:50.873 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:48:50 +0000 (0:00:00.310) 0:08:42.134 ******* 2020-11-16T19:48:51.511 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-11-16T19:48:51.513 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-16T19:48:51.546 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:48:51 +0000 (0:00:00.676) 0:08:42.811 ******* 2020-11-16T19:52:48.295 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.102643", "end": "2020-11-16 19:52:48.270885", "rc": 0, "start": "2020-11-16 19:52:48.168242", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-16T19:52:48.328 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:52:48 +0000 (0:03:56.782) 0:12:39.593 ******* 2020-11-16T19:52:48.561 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-11-16T19:52:48.563 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-16T19:52:48.601 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:52:48 +0000 (0:00:00.272) 0:12:39.866 ******* 2020-11-16T19:54:30.756 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-11-16T19:54:30.759 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605556368.6165514-82512228813590/source", "state": "file", "uid": 0} 2020-11-16T19:54:30.790 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:30 +0000 (0:01:42.188) 0:14:22.055 ******* 2020-11-16T19:54:35.310 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-11-16T19:54:35.312 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-16T19:54:35.314 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-16T19:54:35.409 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-11-16T19:54:35.417 INFO:teuthology.task.ansible.out: 2020-11-16T19:54:35.419 INFO:teuthology.task.ansible.out:PLAY [testnodes] *************************************************************** 2020-11-16T19:54:35.550 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:35 +0000 (0:00:04.759) 0:14:26.815 ******* 2020-11-16T19:54:36.350 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi189.front.sepia.ceph.com] 2020-11-16T19:54:36.463 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Monday 16 November 2020 19:54:36 +0000 (0:00:00.903) 0:14:27.718 ******* 2020-11-16T19:54:40.998 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi189.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.252379", "end": "2020-11-16 19:54:40.959136", "rc": 0, "start": "2020-11-16 19:54:40.706757", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-16T19:54:41.058 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:41 +0000 (0:00:04.589) 0:14:32.308 ******* 2020-11-16T19:54:49.316 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-11-16T19:54:49.318 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.012522", "end": "2020-11-16 19:54:49.292139", "failed_when_result": false, "rc": 0, "start": "2020-11-16 19:54:49.279617", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-16T19:54:49.350 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:49 +0000 (0:00:08.306) 0:14:40.614 ******* 2020-11-16T19:54:53.382 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-16T19:54:53.419 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:53 +0000 (0:00:04.069) 0:14:44.684 ******* 2020-11-16T19:54:53.452 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.486 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:53 +0000 (0:00:00.065) 0:14:44.749 ******* 2020-11-16T19:54:53.514 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.549 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:53 +0000 (0:00:00.060) 0:14:44.810 ******* 2020-11-16T19:54:53.575 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.608 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:53 +0000 (0:00:00.062) 0:14:44.872 ******* 2020-11-16T19:54:53.644 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.673 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:53 +0000 (0:00:00.065) 0:14:44.937 ******* 2020-11-16T19:54:53.703 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.738 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:53 +0000 (0:00:00.064) 0:14:45.002 ******* 2020-11-16T19:54:53.811 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-16T19:54:53.817 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.824 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.836 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.868 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd 2020-11-16T19:54:53.870 INFO:teuthology.task.ansible.out:0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.882 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.892 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.910 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.913 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.926 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.940 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVN 2020-11-16T19:54:53.942 INFO:teuthology.task.ansible.out:TUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.950 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.960 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.963 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.972 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.985 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.987 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:53.997 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.002 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.011 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.017 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.023 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.035 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.038 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.050 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.067 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.074 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.083 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.092 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.103 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.114 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.154 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.170 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.179 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.191 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.203 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.226 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.231 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.250 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.258 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.273 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.291 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.293 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.303 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.314 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.316 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.328 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.339 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.391 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRX 2020-11-16T19:54:54.394 INFO:teuthology.task.ansible.out:lHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.399 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.419 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.427 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.441 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.446 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.452 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.469 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.479 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.490 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.515 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomo 2020-11-16T19:54:54.520 INFO:teuthology.task.ansible.out:n@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.527 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.542 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.550 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.559 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.563 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.574 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.588 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.612 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 2020-11-16T19:54:54.614 INFO:teuthology.task.ansible.out:94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.627 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.638 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.669 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.672 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.681 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.698 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.714 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.721 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.734 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.737 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.746 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.763 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.782 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.794 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dorinda', 'ovpn': 'dorinda@ideapad /BLvrh5FMYG1lXzwak5Z+w d028380a40a95f162324dfbb024cfa98b5771a1b90741fbfc908cad93ef09a14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dorinda", "ovpn": "dorinda@ideapad /BLvrh5FMYG1lXzwak5Z+w d028380a40a95f162324dfbb024cfa98b5771a1b90741fbfc908cad93ef09a14"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:54.862 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:54 +0000 (0:00:01.123) 0:14:46.126 ******* 2020-11-16T19:54:54.895 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-16T19:54:54.935 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:54 +0000 (0:00:00.073) 0:14:46.199 ******* 2020-11-16T19:54:54.978 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:55.017 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:55 +0000 (0:00:00.080) 0:14:46.280 ******* 2020-11-16T19:54:55.057 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:55.091 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:55 +0000 (0:00:00.076) 0:14:46.356 ******* 2020-11-16T19:54:59.204 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.244 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:59 +0000 (0:00:04.152) 0:14:50.509 ******* 2020-11-16T19:54:59.272 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.310 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:59 +0000 (0:00:00.062) 0:14:50.571 ******* 2020-11-16T19:54:59.336 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-11-16T19:54:59.339 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.378 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:59 +0000 (0:00:00.071) 0:14:50.643 ******* 2020-11-16T19:54:59.430 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-16T19:54:59.432 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.441 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.452 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.464 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.466 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.478 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.479 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.489 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.491 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.501 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.512 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.513 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.523 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.525 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.534 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.535 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.545 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.547 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.556 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.558 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.568 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.569 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.580 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.580 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.582 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.592 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.593 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.604 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.623 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.624 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.635 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.636 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.646 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.648 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.658 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.660 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.669 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-16T19:54:59.712 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:54:59 +0000 (0:00:00.332) 0:14:50.975 ******* 2020-11-16T19:55:08.328 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-11-16T19:55:08.330 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-11-16T19:55:08.362 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:08 +0000 (0:00:08.652) 0:14:59.627 ******* 2020-11-16T19:55:08.389 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-16T19:55:08.390 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => { "msg": "Host smithi189.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-16T19:55:08.423 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:08 +0000 (0:00:00.060) 0:14:59.688 ******* 2020-11-16T19:55:08.460 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-16T19:55:08.461 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-16T19:55:08.493 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:08 +0000 (0:00:00.069) 0:14:59.758 ******* 2020-11-16T19:55:16.614 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-16T19:55:16.616 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:16.648 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:16 +0000 (0:00:08.154) 0:15:07.913 ******* 2020-11-16T19:55:16.675 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-16T19:55:16.677 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:16.707 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:16 +0000 (0:00:00.059) 0:15:07.972 ******* 2020-11-16T19:55:16.734 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:16.768 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:16 +0000 (0:00:00.059) 0:15:08.032 ******* 2020-11-16T19:55:16.793 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:16.828 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:16 +0000 (0:00:00.060) 0:15:08.093 ******* 2020-11-16T19:55:16.854 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-16T19:55:16.856 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:16.888 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:16 +0000 (0:00:00.059) 0:15:08.152 ******* 2020-11-16T19:55:16.914 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-16T19:55:16.916 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:16.947 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:16 +0000 (0:00:00.059) 0:15:08.212 ******* 2020-11-16T19:55:16.974 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-16T19:55:16.975 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.008 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.060) 0:15:08.273 ******* 2020-11-16T19:55:17.036 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.077 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.069) 0:15:08.342 ******* 2020-11-16T19:55:17.115 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi189.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T19:55:17.150 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.072) 0:15:08.415 ******* 2020-11-16T19:55:17.178 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.214 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.064) 0:15:08.479 ******* 2020-11-16T19:55:17.242 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-16T19:55:17.243 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.295 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.080) 0:15:08.559 ******* 2020-11-16T19:55:17.324 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T19:55:17.357 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.062) 0:15:08.622 ******* 2020-11-16T19:55:17.384 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-16T19:55:17.386 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.418 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.060) 0:15:08.683 ******* 2020-11-16T19:55:17.445 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-11-16T19:55:17.447 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T19:55:17.479 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.061) 0:15:08.744 ******* 2020-11-16T19:55:17.506 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.543 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.064) 0:15:08.808 ******* 2020-11-16T19:55:17.571 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-16T19:55:17.573 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.607 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.063) 0:15:08.872 ******* 2020-11-16T19:55:17.635 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.669 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.061) 0:15:08.934 ******* 2020-11-16T19:55:17.697 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-16T19:55:17.699 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.732 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.062) 0:15:08.997 ******* 2020-11-16T19:55:17.764 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T19:55:17.797 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.065) 0:15:09.062 ******* 2020-11-16T19:55:17.825 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-16T19:55:17.826 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.861 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.064) 0:15:09.126 ******* 2020-11-16T19:55:17.888 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.922 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.061) 0:15:09.187 ******* 2020-11-16T19:55:17.949 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:17.985 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:17 +0000 (0:00:00.062) 0:15:09.250 ******* 2020-11-16T19:55:18.012 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-16T19:55:18.014 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:18.047 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:18 +0000 (0:00:00.061) 0:15:09.311 ******* 2020-11-16T19:55:18.074 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:18.107 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:18 +0000 (0:00:00.060) 0:15:09.372 ******* 2020-11-16T19:55:18.134 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-16T19:55:18.136 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:18.169 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:18 +0000 (0:00:00.061) 0:15:09.433 ******* 2020-11-16T19:55:18.195 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-16T19:55:18.197 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T19:55:18.230 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:18 +0000 (0:00:00.061) 0:15:09.495 ******* 2020-11-16T19:55:18.258 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-16T19:55:18.260 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:18.305 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:18 +0000 (0:00:00.075) 0:15:09.570 ******* 2020-11-16T19:55:30.405 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:30.438 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:30 +0000 (0:00:12.132) 0:15:21.703 ******* 2020-11-16T19:55:30.464 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-16T19:55:30.466 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:30.498 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:30 +0000 (0:00:00.060) 0:15:21.763 ******* 2020-11-16T19:55:30.536 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:30.538 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:30.571 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:30 +0000 (0:00:00.072) 0:15:21.835 ******* 2020-11-16T19:55:30.597 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-16T19:55:30.599 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-16T19:55:30.609 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-16T19:55:30.643 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:30 +0000 (0:00:00.072) 0:15:21.908 ******* 2020-11-16T19:55:30.670 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-16T19:55:30.671 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:30.706 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:30 +0000 (0:00:00.062) 0:15:21.970 ******* 2020-11-16T19:55:30.847 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-16T19:55:30.849 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.018137", "end": "2020-11-16 19:55:30.829528", "rc": 0, "start": "2020-11-16 19:55:30.811391", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-16T19:55:30.901 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:30 +0000 (0:00:00.195) 0:15:22.166 ******* 2020-11-16T19:55:30.930 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-16T19:55:30.932 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:30.970 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:30 +0000 (0:00:00.064) 0:15:22.231 ******* 2020-11-16T19:55:30.993 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-16T19:55:30.995 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:31.028 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:31 +0000 (0:00:00.062) 0:15:22.293 ******* 2020-11-16T19:55:31.054 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-16T19:55:31.056 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:31.089 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:31 +0000 (0:00:00.060) 0:15:22.354 ******* 2020-11-16T19:55:31.116 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:31.159 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:31 +0000 (0:00:00.069) 0:15:22.424 ******* 2020-11-16T19:55:31.188 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-16T19:55:31.189 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:31.222 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:31 +0000 (0:00:00.063) 0:15:22.487 ******* 2020-11-16T19:55:31.249 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-16T19:55:31.251 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:31.283 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:31 +0000 (0:00:00.060) 0:15:22.548 ******* 2020-11-16T19:55:31.310 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-16T19:55:31.312 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:31.343 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:31 +0000 (0:00:00.060) 0:15:22.608 ******* 2020-11-16T19:55:31.518 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.053412", "end": "2020-11-16 19:55:31.500364", "rc": 0, "start": "2020-11-16 19:55:31.446952", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T19:55:31.557 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:31 +0000 (0:00:00.213) 0:15:22.822 ******* 2020-11-16T19:55:31.586 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-16T19:55:31.588 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:31.621 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:31 +0000 (0:00:00.063) 0:15:22.886 ******* 2020-11-16T19:55:31.814 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-16T19:55:31.816 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.059541", "end": "2020-11-16 19:55:31.797033", "rc": 0, "start": "2020-11-16 19:55:31.737492", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T19:55:31.852 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:31 +0000 (0:00:00.230) 0:15:23.117 ******* 2020-11-16T19:55:55.733 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-16T19:55:55.735 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T19:55:55.781 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:55 +0000 (0:00:23.929) 0:15:47.046 ******* 2020-11-16T19:55:56.603 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-16T19:55:56.604 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T19:55:56.637 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:56 +0000 (0:00:00.855) 0:15:47.901 ******* 2020-11-16T19:55:56.663 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-16T19:55:56.665 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:55:56.697 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:56 +0000 (0:00:00.060) 0:15:47.962 ******* 2020-11-16T19:55:56.993 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-16T19:55:56.995 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-16T19:55:57.027 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:57 +0000 (0:00:00.330) 0:15:48.292 ******* 2020-11-16T19:55:57.883 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-16T19:55:57.884 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T19:55:57.917 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:55:57 +0000 (0:00:00.889) 0:15:49.181 ******* 2020-11-16T19:55:58.255 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-16T19:55:58.256 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-16T19:55:58.549 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-16T19:56:34.265 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-16T19:56:34.316 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:56:34 +0000 (0:00:36.399) 0:16:25.581 ******* 2020-11-16T19:56:34.441 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-16T19:56:34.474 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:56:34 +0000 (0:00:00.157) 0:16:25.739 ******* 2020-11-16T19:56:34.792 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-16T19:56:34.795 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-16T19:56:35.101 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-16T19:56:35.408 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-16T19:56:35.454 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:56:35 +0000 (0:00:00.979) 0:16:26.718 ******* 2020-11-16T19:56:35.493 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-16T19:56:35.495 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-16T19:56:35.533 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:56:35 +0000 (0:00:00.079) 0:16:26.798 ******* 2020-11-16T19:56:35.560 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-16T19:56:35.562 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:56:35.593 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:56:35 +0000 (0:00:00.060) 0:16:26.858 ******* 2020-11-16T19:56:35.620 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-16T19:56:35.622 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T19:56:35.658 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:56:35 +0000 (0:00:00.064) 0:16:26.923 ******* 2020-11-16T19:56:35.696 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-16T19:56:35.698 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-16T19:56:35.708 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-16T19:56:35.709 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-16T19:56:35.719 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-16T19:56:35.756 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:56:35 +0000 (0:00:00.097) 0:16:27.021 ******* 2020-11-16T19:56:35.783 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-16T19:56:35.785 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:56:35.817 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:56:35 +0000 (0:00:00.061) 0:16:27.082 ******* 2020-11-16T19:56:36.732 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-16T19:56:36.734 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T19:56:36.767 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:56:36 +0000 (0:00:00.949) 0:16:28.032 ******* 2020-11-16T19:57:16.527 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-16T19:57:16.529 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-16T19:57:16.561 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:16 +0000 (0:00:39.793) 0:17:07.826 ******* 2020-11-16T19:57:16.900 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-16T19:57:16.902 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-16T19:57:16.933 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:16 +0000 (0:00:00.372) 0:17:08.198 ******* 2020-11-16T19:57:17.103 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ ok: [smithi189.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-16T19:57:17.211 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-16T19:57:17.264 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:17 +0000 (0:00:00.329) 0:17:08.527 ******* 2020-11-16T19:57:17.292 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:17.330 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:17 +0000 (0:00:00.067) 0:17:08.595 ******* 2020-11-16T19:57:17.361 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-16T19:57:17.363 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-16T19:57:17.397 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:17 +0000 (0:00:00.065) 0:17:08.660 ******* 2020-11-16T19:57:17.436 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:17.475 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:17 +0000 (0:00:00.073) 0:17:08.734 ******* 2020-11-16T19:57:17.834 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-16T19:57:17.875 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:17 +0000 (0:00:00.399) 0:17:09.133 ******* 2020-11-16T19:57:18.506 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-16T19:57:18.507 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-16 19:39:22 UTC", "ActiveEnterTimestampMonotonic": "22226124", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target var-run.mount systemd-tmpfiles-setup.service -.mount basic.target sysinit.target nss-lookup.target systemd-journald.socket local-fs.target network.target system.slice time-sync.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-16 19:39:22 UTC", "AssertTimestampMonotonic": "22224286", "Before": "shutdown.target xdm.service getty@tty1.service multi-user.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-16 19:39:22 UTC", "ConditionTimestampMonotonic": "22224280", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-16T19:57:18.509 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1311", "ExecMainStartTimestamp": "Mon 2020-11-16 19:39:22 UTC", "ExecMainStartTimestampMonotonic": "22226077", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Mon 2020-11-16 19:39:22 UTC] ; stop_time=[n/a] ; pid=1311 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-16 19:39:22 UTC", "InactiveExitTimestampMonotonic": "22226124", "InvocationID": "55dbf12fa3b747cf964434e52511d8b6", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-11-16T19:57:18.510 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1311", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-11-16T19:57:18.512 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-16 19:39:22 UTC", "StateChangeTimestampMonotonic": "22226124", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-16 19:39:22 UTC", "WatchdogTimestampMonotonic": "22226123", "WatchdogUSec": "0"}} 2020-11-16T19:57:18.541 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:18 +0000 (0:00:00.672) 0:17:09.805 ******* 2020-11-16T19:57:18.567 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-11-16T19:57:18.569 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:18.600 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:18 +0000 (0:00:00.059) 0:17:09.865 ******* 2020-11-16T19:57:18.629 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-11-16T19:57:18.631 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:18.661 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:18 +0000 (0:00:00.061) 0:17:09.926 ******* 2020-11-16T19:57:18.689 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:18.722 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:18 +0000 (0:00:00.061) 0:17:09.987 ******* 2020-11-16T19:57:18.750 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-16T19:57:18.752 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:18.783 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:18 +0000 (0:00:00.060) 0:17:10.048 ******* 2020-11-16T19:57:18.810 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-16T19:57:18.812 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:18.843 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:18 +0000 (0:00:00.060) 0:17:10.108 ******* 2020-11-16T19:57:18.870 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-16T19:57:18.872 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:18.903 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:18 +0000 (0:00:00.059) 0:17:10.168 ******* 2020-11-16T19:57:18.932 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-16T19:57:18.933 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:18.964 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:18 +0000 (0:00:00.060) 0:17:10.229 ******* 2020-11-16T19:57:18.991 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-11-16T19:57:18.992 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:19.023 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:19 +0000 (0:00:00.059) 0:17:10.288 ******* 2020-11-16T19:57:19.050 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-16T19:57:19.052 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:19.083 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:19 +0000 (0:00:00.059) 0:17:10.348 ******* 2020-11-16T19:57:19.111 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:19.143 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:19 +0000 (0:00:00.060) 0:17:10.408 ******* 2020-11-16T19:57:19.170 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-16T19:57:19.172 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-16T19:57:19.206 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:19 +0000 (0:00:00.062) 0:17:10.471 ******* 2020-11-16T19:57:19.243 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-16T19:57:19.245 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-16T19:57:19.278 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:19 +0000 (0:00:00.071) 0:17:10.542 ******* 2020-11-16T19:57:19.315 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-16T19:57:19.316 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-16T19:57:19.353 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:19 +0000 (0:00:00.075) 0:17:10.618 ******* 2020-11-16T19:57:19.390 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-16T19:57:19.392 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-16T19:57:19.427 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:19 +0000 (0:00:00.073) 0:17:10.692 ******* 2020-11-16T19:57:19.548 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-16T19:57:19.549 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-16T19:57:19.581 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:19 +0000 (0:00:00.154) 0:17:10.846 ******* 2020-11-16T19:57:19.712 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-16T19:57:19.714 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-16T19:57:19.745 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:19 +0000 (0:00:00.163) 0:17:11.010 ******* 2020-11-16T19:57:19.772 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-16T19:57:19.774 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:19.806 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:19 +0000 (0:00:00.060) 0:17:11.071 ******* 2020-11-16T19:57:19.833 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-16T19:57:19.835 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:19.884 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:19 +0000 (0:00:00.078) 0:17:11.149 ******* 2020-11-16T19:57:20.038 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-16T19:57:20.040 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-16T19:57:20.072 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:20 +0000 (0:00:00.188) 0:17:11.337 ******* 2020-11-16T19:57:20.443 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-16T19:57:20.445 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-16T19:57:20.478 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:20 +0000 (0:00:00.405) 0:17:11.743 ******* 2020-11-16T19:57:20.504 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-16T19:57:20.506 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi189"}, "changed": false} 2020-11-16T19:57:20.541 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:20 +0000 (0:00:00.062) 0:17:11.805 ******* 2020-11-16T19:57:21.465 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-11-16T19:57:21.467 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi189.front.sepia.ceph.com", "ansible_hostname": "smithi189", "ansible_nodename": "smithi189"}, "changed": true, "name": "smithi189"} 2020-11-16T19:57:21.499 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:21 +0000 (0:00:00.958) 0:17:12.764 ******* 2020-11-16T19:57:21.822 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-11-16T19:57:21.824 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-16T19:57:21.858 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:21 +0000 (0:00:00.357) 0:17:13.122 ******* 2020-11-16T19:57:26.180 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-11-16T19:57:26.181 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-16T19:57:26.217 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:26 +0000 (0:00:04.358) 0:17:17.481 ******* 2020-11-16T19:57:26.738 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-16T19:57:26.740 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-16T19:57:26.774 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:26 +0000 (0:00:00.557) 0:17:18.039 ******* 2020-11-16T19:57:26.802 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:26.838 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:26 +0000 (0:00:00.064) 0:17:18.103 ******* 2020-11-16T19:57:26.865 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-16T19:57:26.867 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:26.900 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:26 +0000 (0:00:00.060) 0:17:18.164 ******* 2020-11-16T19:57:30.892 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-16T19:57:30.896 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:30.935 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:30 +0000 (0:00:04.036) 0:17:22.200 ******* 2020-11-16T19:57:30.971 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:31.002 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:31 +0000 (0:00:00.067) 0:17:22.267 ******* 2020-11-16T19:57:31.053 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-16T19:57:31.055 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:31.098 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:31 +0000 (0:00:00.093) 0:17:22.361 ******* 2020-11-16T19:57:31.124 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-16T19:57:31.125 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:31.158 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:31 +0000 (0:00:00.062) 0:17:22.423 ******* 2020-11-16T19:57:31.189 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-16T19:57:31.191 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:31.226 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:31 +0000 (0:00:00.065) 0:17:22.489 ******* 2020-11-16T19:57:31.252 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:31.294 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:31 +0000 (0:00:00.069) 0:17:22.559 ******* 2020-11-16T19:57:31.328 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-16T19:57:31.362 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:31 +0000 (0:00:00.067) 0:17:22.626 ******* 2020-11-16T19:57:31.394 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-16T19:57:31.433 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:31 +0000 (0:00:00.067) 0:17:22.694 ******* 2020-11-16T19:57:31.458 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-16T19:57:31.501 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:31 +0000 (0:00:00.071) 0:17:22.765 ******* 2020-11-16T19:57:31.535 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:31.569 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:31 +0000 (0:00:00.066) 0:17:22.832 ******* 2020-11-16T19:57:31.617 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi189.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-16T19:57:31.626 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-16T19:57:31.660 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:31 +0000 (0:00:00.092) 0:17:22.925 ******* 2020-11-16T19:57:35.649 INFO:teuthology.task.ansible.out: 2020-11-16T19:57:35.651 INFO:teuthology.task.ansible.out:TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:35.653 INFO:teuthology.task.ansible.out: 2020-11-16T19:57:35.698 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:35 +0000 (0:00:04.036) 0:17:26.962 ******* 2020-11-16T19:57:35.736 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T19:57:35.738 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:35.776 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:35 +0000 (0:00:00.079) 0:17:27.041 ******* 2020-11-16T19:57:39.773 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:39.808 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:39 +0000 (0:00:04.032) 0:17:31.073 ******* 2020-11-16T19:57:39.835 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-16T19:57:39.837 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:39.872 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:39 +0000 (0:00:00.062) 0:17:31.135 ******* 2020-11-16T19:57:39.898 INFO:teuthology.task.ansible.out: 2020-11-16T19:57:39.899 INFO:teuthology.task.ansible.out:TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:39.932 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:39 +0000 (0:00:00.061) 0:17:31.197 ******* 2020-11-16T19:57:39.963 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-16T19:57:39.964 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.001 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:39 +0000 (0:00:00.068) 0:17:31.266 ******* 2020-11-16T19:57:40.028 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-16T19:57:40.029 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.062 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.061) 0:17:31.327 ******* 2020-11-16T19:57:40.089 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-16T19:57:40.091 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.127 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.064) 0:17:31.392 ******* 2020-11-16T19:57:40.166 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.200 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.072) 0:17:31.465 ******* 2020-11-16T19:57:40.226 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-16T19:57:40.228 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.262 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.061) 0:17:31.526 ******* 2020-11-16T19:57:40.288 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.324 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.062) 0:17:31.589 ******* 2020-11-16T19:57:40.351 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-16T19:57:40.352 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.389 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.060) 0:17:31.649 ******* 2020-11-16T19:57:40.412 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-16T19:57:40.413 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.446 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.061) 0:17:31.711 ******* 2020-11-16T19:57:40.473 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-16T19:57:40.475 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.515 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.064) 0:17:31.775 ******* 2020-11-16T19:57:40.549 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.577 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.065) 0:17:31.841 ******* 2020-11-16T19:57:40.604 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-11-16T19:57:40.606 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.640 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.062) 0:17:31.904 ******* 2020-11-16T19:57:40.667 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.702 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.062) 0:17:31.967 ******* 2020-11-16T19:57:40.735 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.769 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.066) 0:17:32.033 ******* 2020-11-16T19:57:40.796 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-16T19:57:40.798 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.840 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.071) 0:17:32.105 ******* 2020-11-16T19:57:40.870 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:40.905 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:40 +0000 (0:00:00.064) 0:17:32.170 ******* 2020-11-16T19:57:41.044 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005212", "end": "2020-11-16 19:57:41.027588", "rc": 0, "start": "2020-11-16 19:57:41.022376", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T19:57:41.078 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:41 +0000 (0:00:00.172) 0:17:32.343 ******* 2020-11-16T19:57:41.108 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-16T19:57:41.149 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:41 +0000 (0:00:00.071) 0:17:32.414 ******* 2020-11-16T19:57:57.187 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-16T19:57:57.189 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-16T19:57:57.223 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:57 +0000 (0:00:16.073) 0:17:48.488 ******* 2020-11-16T19:57:57.250 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-16T19:57:57.251 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:57:57.284 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:57:57 +0000 (0:00:00.061) 0:17:48.549 ******* 2020-11-16T19:58:02.461 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-11-16T19:58:02.463 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-16T19:58:15.164 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-16T19:58:15.198 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:15 +0000 (0:00:17.913) 0:18:06.463 ******* 2020-11-16T19:58:16.106 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-16T19:58:16.108 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T19:58:16.139 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:16 +0000 (0:00:00.941) 0:18:07.404 ******* 2020-11-16T19:58:16.167 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-16T19:58:16.200 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:16 +0000 (0:00:00.060) 0:18:07.465 ******* 2020-11-16T19:58:24.237 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-11-16T19:58:24.239 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.029088", "end": "2020-11-16 19:58:24.213212", "rc": 0, "start": "2020-11-16 19:58:24.184124", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T19:58:24.273 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:24 +0000 (0:00:08.072) 0:18:15.538 ******* 2020-11-16T19:58:27.075 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-11-16T19:58:27.076 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T19:58:27.109 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:27 +0000 (0:00:02.835) 0:18:18.373 ******* 2020-11-16T19:58:48.131 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-11-16T19:58:48.133 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 1 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137606 files and directories 2020-11-16T19:58:48.135 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 1 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-11-16T19:58:48.142 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137606 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-16T19:58:48.165 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:48 +0000 (0:00:21.056) 0:18:39.430 ******* 2020-11-16T19:58:49.040 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T19:58:49.073 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:49 +0000 (0:00:00.908) 0:18:40.338 ******* 2020-11-16T19:58:49.100 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-16T19:58:49.101 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:58:49.133 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:49 +0000 (0:00:00.059) 0:18:40.398 ******* 2020-11-16T19:58:58.430 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-16T19:58:58.432 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 1 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9393 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-11-16T19:58:58.434 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137464 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-11-16T19:58:58.435 INFO:teuthology.task.ansible.out:1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 1 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9393 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-11-16T19:58:58.437 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137464 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-16T19:58:58.465 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:58 +0000 (0:00:09.332) 0:18:49.730 ******* 2020-11-16T19:58:59.333 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-16T19:58:59.334 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T19:58:59.368 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:59 +0000 (0:00:00.902) 0:18:50.633 ******* 2020-11-16T19:58:59.499 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-11-16T19:58:59.500 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-16T19:58:59.532 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:59 +0000 (0:00:00.164) 0:18:50.797 ******* 2020-11-16T19:58:59.569 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-16T19:58:59.571 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:58:59.603 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:59 +0000 (0:00:00.070) 0:18:50.868 ******* 2020-11-16T19:58:59.733 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-16T19:58:59.735 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-16T19:58:59.767 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:58:59 +0000 (0:00:00.163) 0:18:51.032 ******* 2020-11-16T19:59:00.073 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-16T19:59:00.075 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-16T19:59:00.107 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:00 +0000 (0:00:00.339) 0:18:51.372 ******* 2020-11-16T19:59:00.257 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-16T19:59:00.259 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-16T19:59:00.290 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:00 +0000 (0:00:00.183) 0:18:51.555 ******* 2020-11-16T19:59:00.424 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-16T19:59:00.426 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-16T19:59:00.457 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:00 +0000 (0:00:00.166) 0:18:51.722 ******* 2020-11-16T19:59:00.744 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-16T19:59:00.746 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Mon 2020-11-16 19:39:23 UTC", "ActiveEnterTimestampMonotonic": "22701287", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket system.slice systemd-tmpfiles-setup.service remote-fs.target sysinit.target basic.target network.target -.mount nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-16 19:39:22 UTC", "AssertTimestampMonotonic": "22218457", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-16 19:39:22 UTC", "ConditionTimestampMonotonic": "22218457", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-16T19:59:00.748 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1401", "ExecMainStartTimestamp": "Mon 2020-11-16 19:39:23 UTC", "ExecMainStartTimestampMonotonic": "22701259", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-16 19:39:22 UTC", "InactiveExitTimestampMonotonic": "22220149", "InvocationID": "9ba06089a1694e2e9d4ce49ab2f2c378", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-16T19:59:00.749 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1401", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-16T19:59:00.750 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-16 19:39:23 UTC", "StateChangeTimestampMonotonic": "22701287", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-16 19:39:23 UTC", "WatchdogTimestampMonotonic": "22701285", "WatchdogUSec": "0"}} 2020-11-16T19:59:00.780 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:00 +0000 (0:00:00.323) 0:18:52.045 ******* 2020-11-16T19:59:00.807 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-16T19:59:00.809 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:00.841 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:00 +0000 (0:00:00.060) 0:18:52.106 ******* 2020-11-16T19:59:00.871 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-16T19:59:00.873 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:00.907 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:00 +0000 (0:00:00.066) 0:18:52.172 ******* 2020-11-16T19:59:00.935 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-16T19:59:00.937 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:00.971 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:00 +0000 (0:00:00.063) 0:18:52.236 ******* 2020-11-16T19:59:00.999 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:01.034 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:01 +0000 (0:00:00.063) 0:18:52.299 ******* 2020-11-16T19:59:01.062 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-16T19:59:01.065 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:01.096 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:01 +0000 (0:00:00.062) 0:18:52.361 ******* 2020-11-16T19:59:01.124 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-16T19:59:01.126 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:01.165 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:01 +0000 (0:00:00.068) 0:18:52.430 ******* 2020-11-16T19:59:01.195 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-16T19:59:01.198 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:01.228 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:01 +0000 (0:00:00.063) 0:18:52.493 ******* 2020-11-16T19:59:01.256 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-16T19:59:01.258 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:01.289 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:01 +0000 (0:00:00.061) 0:18:52.554 ******* 2020-11-16T19:59:01.315 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T19:59:01.317 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:01.349 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:01 +0000 (0:00:00.059) 0:18:52.613 ******* 2020-11-16T19:59:05.293 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-16T19:59:05.295 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:05.328 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:05 +0000 (0:00:03.979) 0:18:56.593 ******* 2020-11-16T19:59:05.355 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-16T19:59:05.357 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:05.389 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:05 +0000 (0:00:00.060) 0:18:56.653 ******* 2020-11-16T19:59:05.417 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-16T19:59:05.419 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:05.451 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:05 +0000 (0:00:00.062) 0:18:56.716 ******* 2020-11-16T19:59:05.481 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-16T19:59:05.482 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:05.518 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:05 +0000 (0:00:00.066) 0:18:56.782 ******* 2020-11-16T19:59:05.544 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-16T19:59:05.547 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:05.586 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:05 +0000 (0:00:00.065) 0:18:56.848 ******* 2020-11-16T19:59:05.630 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:05.664 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:05 +0000 (0:00:00.079) 0:18:56.927 ******* 2020-11-16T19:59:05.691 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-16T19:59:05.693 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:05.728 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:05 +0000 (0:00:00.063) 0:18:56.990 ******* 2020-11-16T19:59:05.755 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-16T19:59:05.757 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:05.794 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:05 +0000 (0:00:00.067) 0:18:57.058 ******* 2020-11-16T19:59:05.831 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:05.865 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:05 +0000 (0:00:00.072) 0:18:57.130 ******* 2020-11-16T19:59:05.897 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:05.938 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:05 +0000 (0:00:00.068) 0:18:57.198 ******* 2020-11-16T19:59:05.971 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:06.013 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:06 +0000 (0:00:00.078) 0:18:57.277 ******* 2020-11-16T19:59:06.043 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:06.073 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:06 +0000 (0:00:00.060) 0:18:57.338 ******* 2020-11-16T19:59:06.115 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-16T19:59:06.117 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:06.149 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:06 +0000 (0:00:00.075) 0:18:57.413 ******* 2020-11-16T19:59:06.178 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:06.212 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:06 +0000 (0:00:00.063) 0:18:57.477 ******* 2020-11-16T19:59:06.241 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:06.276 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:06 +0000 (0:00:00.064) 0:18:57.541 ******* 2020-11-16T19:59:06.303 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-16T19:59:06.306 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:06.340 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:06 +0000 (0:00:00.062) 0:18:57.604 ******* 2020-11-16T19:59:06.367 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:06.401 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:06 +0000 (0:00:00.061) 0:18:57.665 ******* 2020-11-16T19:59:06.541 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-11-16T19:59:06.576 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:06 +0000 (0:00:00.173) 0:18:57.838 ******* 2020-11-16T19:59:06.910 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-16T19:59:06.911 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-16T19:59:06.954 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:06 +0000 (0:00:00.376) 0:18:58.215 ******* 2020-11-16T19:59:06.984 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-16T19:59:06.986 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:07.021 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:07 +0000 (0:00:00.069) 0:18:58.285 ******* 2020-11-16T19:59:07.361 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-16T19:59:07.396 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:07 +0000 (0:00:00.376) 0:18:58.661 ******* 2020-11-16T19:59:07.739 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-11-16T19:59:07.740 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-16T19:59:07.792 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:07 +0000 (0:00:00.395) 0:18:59.057 ******* 2020-11-16T19:59:08.119 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-16T19:59:08.121 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-16T19:59:08.153 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:08 +0000 (0:00:00.361) 0:18:59.418 ******* 2020-11-16T19:59:08.292 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-16T19:59:08.294 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-16T19:59:08.329 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:08 +0000 (0:00:00.175) 0:18:59.594 ******* 2020-11-16T19:59:08.358 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.391 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:08 +0000 (0:00:00.062) 0:18:59.656 ******* 2020-11-16T19:59:08.419 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-16T19:59:08.421 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.452 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:08 +0000 (0:00:00.060) 0:18:59.717 ******* 2020-11-16T19:59:08.480 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.514 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:08 +0000 (0:00:00.062) 0:18:59.779 ******* 2020-11-16T19:59:08.548 INFO:teuthology.task.ansible.out: 2020-11-16T19:59:08.551 INFO:teuthology.task.ansible.out:TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.582 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:08 +0000 (0:00:00.067) 0:18:59.847 ******* 2020-11-16T19:59:08.611 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.656 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:08 +0000 (0:00:00.074) 0:18:59.921 ******* 2020-11-16T19:59:08.695 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-16T19:59:08.728 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:08 +0000 (0:00:00.071) 0:18:59.992 ******* 2020-11-16T19:59:08.782 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi189.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.799 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.812 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.821 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.833 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi189.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.844 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.848 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.857 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.879 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.881 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-16T19:59:08.923 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:08 +0000 (0:00:00.193) 0:19:00.186 ******* 2020-11-16T19:59:17.066 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-16T19:59:17.067 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:07.884359", "end": "2020-11-16 19:59:17.040090", "rc": 0, "start": "2020-11-16 19:59:09.155731", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-16T19:59:17.107 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:17 +0000 (0:00:08.185) 0:19:08.372 ******* 2020-11-16T19:59:17.135 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-16T19:59:17.137 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:17.202 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:17 +0000 (0:00:00.095) 0:19:08.467 ******* 2020-11-16T19:59:17.245 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:17.280 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:17 +0000 (0:00:00.076) 0:19:08.544 ******* 2020-11-16T19:59:17.306 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-16T19:59:17.308 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:17.346 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:17 +0000 (0:00:00.067) 0:19:08.611 ******* 2020-11-16T19:59:18.239 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-16T19:59:18.241 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T19:59:18.273 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:18 +0000 (0:00:00.927) 0:19:09.538 ******* 2020-11-16T19:59:18.300 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-16T19:59:18.302 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:18.333 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:18 +0000 (0:00:00.060) 0:19:09.598 ******* 2020-11-16T19:59:18.371 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-16T19:59:18.373 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item={'block_used': 13753604, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341504, 'size_available': 927094800384, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149531, 'block_size': 4096, 'inode_available': 60905445}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341504, "block_size": 4096, "block_total": 240095108, "block_used": 13753604, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905445, "inode_total": 61054976, "inode_used": 149531, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927094800384, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}} 2020-11-16T19:59:18.405 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:18 +0000 (0:00:00.072) 0:19:09.670 ******* 2020-11-16T19:59:22.481 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-16T19:59:22.482 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010116", "end": "2020-11-16 19:59:22.463929", "rc": 0, "start": "2020-11-16 19:59:22.453813", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T19:59:22.513 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:22 +0000 (0:00:04.107) 0:19:13.778 ******* 2020-11-16T19:59:22.550 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-16T19:59:22.552 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'block_used': 13753604, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341504, 'size_available': 927094800384, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149531, 'block_size': 4096, 'inode_available': 60905445}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341504, "block_size": 4096, "block_total": 240095108, "block_used": 13753604, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905445, "inode_total": 61054976, "inode_used": 149531, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927094800384, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:22.585 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:22 +0000 (0:00:00.071) 0:19:13.850 ******* 2020-11-16T19:59:22.611 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-16T19:59:22.644 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:22 +0000 (0:00:00.059) 0:19:13.909 ******* 2020-11-16T19:59:22.693 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-16T19:59:22.695 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:24.356 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfGllstUrpfHEV756B7BFn0U9EzRZVTVNM'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.532036", "end": "2020-11-16 19:59:24.320808", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfGllstUrpfHEV756B7BFn0U9EzRZVTVNM"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:22.788772", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T19:59:24.357 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:24.368 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:25.596 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfeHsMv958PAgFAnwtUrIMlhfqI1QB4pkZ'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.114467", "end": "2020-11-16 19:59:25.576809", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfeHsMv958PAgFAnwtUrIMlhfqI1QB4pkZ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:24.462342", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T19:59:25.608 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKLY', 'wwn-0x5000c50092937477'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKLY-part1', 'wwn-0x5000c50092937477-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092937477', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XKLY', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKLY", "wwn-0x5000c50092937477"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKLY-part1", "wwn-0x5000c50092937477-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XKLY", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092937477"}}, "skip_reason": 2020-11-16T19:59:25.610 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-16T19:59:30.804 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-AcFCd1-u18J-yqW0-qxWf-ttTF-g7NJ-XpvHXu', 'nvme-INTEL_SSDPEDMD400G4_CVFT6232004L400BGN', 'nvme-nvme.8086-43564654363233323030344c34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.142698", "end": "2020-11-16 19:59:30.776937", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-AcFCd1-u18J-yqW0-qxWf-ttTF-g7NJ-XpvHXu", "nvme-INTEL_SSDPEDMD400G4_CVFT6232004L400BGN", "nvme-nvme.8086-43564654363233323030344c34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:29.634239", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-16T19:59:30.806 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T19:59:30.816 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:30.827 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:35.926 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMf4S0coqZfDDAZ7m63yxB2fdHmpM8PrsF8'], 'uuids': ['954b5276-bdbd-4b9e-be29-edc1da06fe64']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.138222", "end": "2020-11-16 19:59:35.893435", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMf4S0coqZfDDAZ7m63yxB2fdHmpM8PrsF8"], "labels": [], "masters": [], "uuids": ["954b5276-bdbd-4b9e-be29-edc1da06fe64"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:34.755213", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T19:59:35.942 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:35.948 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:37.206 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMf3EoOtpMM8eLqorl2GemBQ8GnDMrtQtIF'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.132397", "end": "2020-11-16 19:59:37.176505", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMf3EoOtpMM8eLqorl2GemBQ8GnDMrtQtIF"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:36.044108", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T19:59:37.217 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:38.449 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfaIeIXrkauwKfK4yHqdXdTJUc19njUCzC'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.121482", "end": "2020-11-16 19:59:38.426295", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfaIeIXrkauwKfK4yHqdXdTJUc19njUCzC"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:37.304813", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-16T19:59:38.493 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:38 +0000 (0:00:15.848) 0:19:29.757 ******* 2020-11-16T19:59:38.553 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-16T19:59:38.554 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:38.820 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfGllstUrpfHEV756B7BFn0U9EzRZVTVNM'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.154401", "end": "2020-11-16 19:59:38.792964", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfGllstUrpfHEV756B7BFn0U9EzRZVTVNM"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:38.638563", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149667 s, 771 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149667 s, 771 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T19:59:38.832 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:38.833 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:39.116 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfeHsMv958PAgFAnwtUrIMlhfqI1QB4pkZ'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.158780", "end": "2020-11-16 19:59:39.091015", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfeHsMv958PAgFAnwtUrIMlhfqI1QB4pkZ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:38.932235", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154152 s, 748 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154152 s, 748 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T19:59:39.117 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKLY', 'wwn-0x5000c50092937477'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKLY-part1', 'wwn-0x5000c50092937477-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092937477', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XKLY', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKLY", "wwn-0x5000c50092937477"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKLY-part1", "wwn-0x5000c50092937477-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XKLY", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092937477"}}, "skip_reason": 2020-11-16T19:59:39.119 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-16T19:59:39.289 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-AcFCd1-u18J-yqW0-qxWf-ttTF-g7NJ-XpvHXu', 'nvme-INTEL_SSDPEDMD400G4_CVFT6232004L400BGN', 'nvme-nvme.8086-43564654363233323030344c34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.052693", "end": "2020-11-16 19:59:39.264324", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-AcFCd1-u18J-yqW0-qxWf-ttTF-g7NJ-XpvHXu", "nvme-INTEL_SSDPEDMD400G4_CVFT6232004L400BGN", "nvme-nvme.8086-43564654363233323030344c34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:39.211631", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0481065 s, 2.4 GB/s", "std 2020-11-16T19:59:39.291 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0481065 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T19:59:39.301 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:39.313 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:39.585 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMf4S0coqZfDDAZ7m63yxB2fdHmpM8PrsF8'], 'uuids': ['954b5276-bdbd-4b9e-be29-edc1da06fe64']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.148159", "end": "2020-11-16 19:59:39.555570", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMf4S0coqZfDDAZ7m63yxB2fdHmpM8PrsF8"], "labels": [], "masters": [], "uuids": ["954b5276-bdbd-4b9e-be29-edc1da06fe64"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:39.407411", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145246 s, 794 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145246 s, 794 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T19:59:39.597 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:39.608 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:39.890 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMf3EoOtpMM8eLqorl2GemBQ8GnDMrtQtIF'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.160463", "end": "2020-11-16 19:59:39.860861", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMf3EoOtpMM8eLqorl2GemBQ8GnDMrtQtIF"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:39.700398", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155697 s, 741 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155697 s, 741 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T19:59:39.902 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-16T19:59:40.185 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfaIeIXrkauwKfK4yHqdXdTJUc19njUCzC'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.158448", "end": "2020-11-16 19:59:40.155483", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-mN9AFFf9Dy0ews50mlfGWM1La1rdHzMfaIeIXrkauwKfK4yHqdXdTJUc19njUCzC"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-16 19:59:39.997035", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153721 s, 750 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153721 s, 750 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-16T19:59:40.228 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:40 +0000 (0:00:01.735) 0:19:31.493 ******* 2020-11-16T19:59:52.379 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-11-16T19:59:52.381 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.187407", "end": "2020-11-16 19:59:52.344929", "rc": 0, "start": "2020-11-16 19:59:52.157522", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T19:59:52.413 INFO:teuthology.task.ansible.out:Monday 16 November 2020 19:59:52 +0000 (0:00:12.184) 0:19:43.677 ******* 2020-11-16T20:00:04.378 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-16T20:00:04.380 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.062923", "end": "2020-11-16 20:00:04.352027", "rc": 0, "start": "2020-11-16 20:00:04.289104", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T20:00:04.412 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:04 +0000 (0:00:11.999) 0:19:55.677 ******* 2020-11-16T20:00:04.439 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-16T20:00:04.475 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:04 +0000 (0:00:00.062) 0:19:55.739 ******* 2020-11-16T20:00:04.501 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-16T20:00:04.503 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:04.537 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:04 +0000 (0:00:00.062) 0:19:55.802 ******* 2020-11-16T20:00:04.564 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-16T20:00:04.566 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:04.598 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:04 +0000 (0:00:00.060) 0:19:55.862 ******* 2020-11-16T20:00:04.624 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-16T20:00:04.626 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:04.661 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:04 +0000 (0:00:00.063) 0:19:55.926 ******* 2020-11-16T20:00:04.688 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-16T20:00:04.689 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:04.721 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:04 +0000 (0:00:00.059) 0:19:55.986 ******* 2020-11-16T20:00:04.841 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-16T20:00:04.843 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-16T20:00:04.875 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:04 +0000 (0:00:00.154) 0:19:56.140 ******* 2020-11-16T20:00:04.913 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-16T20:00:04.914 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'block_used': 13753604, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341504, 'size_available': 927094800384, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149531, 'block_size': 4096, 'inode_available': 60905445}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341504, "block_size": 4096, "block_total": 240095108, "block_used": 13753604, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905445, "inode_total": 61054976, "inode_used": 149531, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927094800384, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:04.951 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:04 +0000 (0:00:00.075) 0:19:56.215 ******* 2020-11-16T20:00:04.977 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-11-16T20:00:04.979 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:05.011 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:05 +0000 (0:00:00.060) 0:19:56.275 ******* 2020-11-16T20:00:05.037 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T20:00:05.038 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:05.070 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:05 +0000 (0:00:00.059) 0:19:56.335 ******* 2020-11-16T20:00:05.096 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T20:00:05.098 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:05.130 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:05 +0000 (0:00:00.060) 0:19:56.395 ******* 2020-11-16T20:00:05.157 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-16T20:00:05.159 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:05.191 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:05 +0000 (0:00:00.060) 0:19:56.456 ******* 2020-11-16T20:00:05.217 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-16T20:00:05.219 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:05.251 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:05 +0000 (0:00:00.059) 0:19:56.516 ******* 2020-11-16T20:00:05.277 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-16T20:00:05.279 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:05.310 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:05 +0000 (0:00:00.059) 0:19:56.575 ******* 2020-11-16T20:00:05.762 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-16T20:00:05.764 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-16T20:00:05.799 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:05 +0000 (0:00:00.488) 0:19:57.064 ******* 2020-11-16T20:00:17.903 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-16T20:00:17.905 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-16T20:00:18.142 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-16T20:00:18.392 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-16T20:00:18.634 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-16T20:00:18.873 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-16T20:00:18.914 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:18 +0000 (0:00:13.115) 0:20:10.179 ******* 2020-11-16T20:00:19.047 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-16T20:00:19.049 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-16T20:00:19.084 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:19 +0000 (0:00:00.169) 0:20:10.349 ******* 2020-11-16T20:00:19.238 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-16T20:00:19.355 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-16T20:00:19.465 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-16T20:00:19.582 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-16T20:00:19.607 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-16T20:00:19.643 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:19 +0000 (0:00:00.558) 0:20:10.908 ******* 2020-11-16T20:00:23.678 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-11-16T20:00:23.680 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-11-16T20:00:23.712 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:23 +0000 (0:00:04.069) 0:20:14.977 ******* 2020-11-16T20:00:23.751 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-11-16T20:00:23.753 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-16T20:00:23.802 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:23 +0000 (0:00:00.089) 0:20:15.066 ******* 2020-11-16T20:00:28.020 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-11-16T20:00:28.021 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true} 2020-11-16T20:00:28.055 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:28 +0000 (0:00:04.253) 0:20:19.320 ******* 2020-11-16T20:00:32.434 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-11-16T20:00:32.436 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-16T20:00:32.474 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:32 +0000 (0:00:04.418) 0:20:23.738 ******* 2020-11-16T20:00:32.622 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-16T20:00:32.625 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-16T20:00:32.663 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:32 +0000 (0:00:00.189) 0:20:23.928 ******* 2020-11-16T20:00:32.815 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-16T20:00:32.855 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:32 +0000 (0:00:00.191) 0:20:24.119 ******* 2020-11-16T20:00:40.866 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-16T20:00:40.905 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:40 +0000 (0:00:08.049) 0:20:32.169 ******* 2020-11-16T20:00:52.798 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-16T20:00:52.805 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-16T20:00:52.835 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:00:52 +0000 (0:00:11.930) 0:20:44.100 ******* 2020-11-16T20:01:01.431 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-16T20:01:01.433 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T20:01:01.465 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:01 +0000 (0:00:08.630) 0:20:52.730 ******* 2020-11-16T20:01:01.639 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-16T20:01:01.641 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.045452", "end": "2020-11-16 20:01:01.616214", "rc": 0, "start": "2020-11-16 20:01:01.570762", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-16T20:01:01.674 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:01 +0000 (0:00:00.209) 0:20:52.939 ******* 2020-11-16T20:01:01.702 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:01.737 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:01 +0000 (0:00:00.063) 0:20:53.002 ******* 2020-11-16T20:01:01.764 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-16T20:01:01.766 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:01.799 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:01 +0000 (0:00:00.061) 0:20:53.063 ******* 2020-11-16T20:01:02.690 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-16T20:01:02.692 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T20:01:02.724 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:02 +0000 (0:00:00.925) 0:20:53.989 ******* 2020-11-16T20:01:02.751 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-16T20:01:02.753 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:02.785 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:02 +0000 (0:00:00.060) 0:20:54.049 ******* 2020-11-16T20:01:03.098 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-16T20:01:03.100 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-11-16T20:01:03.134 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:03 +0000 (0:00:00.349) 0:20:54.399 ******* 2020-11-16T20:01:03.162 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-11-16T20:01:03.164 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:03.197 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:03 +0000 (0:00:00.063) 0:20:54.462 ******* 2020-11-16T20:01:03.394 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-16T20:01:03.396 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-16 19:39:29 UTC", "ActiveEnterTimestampMonotonic": "28329356", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target system.slice basic.target systemd-journald.socket systemd-tmpfiles-setup.service sysinit.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-16 19:39:22 UTC", "AssertTimestampMonotonic": "21820134", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-16 19:39:22 UTC", "ConditionTimestampMonotonic": "21820133", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-16T20:01:03.398 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1769", "ExecMainStartTimestamp": "Mon 2020-11-16 19:39:29 UTC", "ExecMainStartTimestampMonotonic": "28329343", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-16 19:39:22 UTC", "InactiveExitTimestampMonotonic": "21821225", "InvocationID": "f4cb2c21184e4501a8056ba431d7744c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-16T20:01:03.399 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1769", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-16 19:39:29 UTC", "S 2020-11-16T20:01:03.401 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "28329356", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-16 19:39:29 UTC", "WatchdogTimestampMonotonic": "28329354", "WatchdogUSec": "0"}} 2020-11-16T20:01:03.451 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:03 +0000 (0:00:00.253) 0:20:54.715 ******* 2020-11-16T20:01:03.480 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T20:01:03.481 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-16T20:01:03.514 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:03 +0000 (0:00:00.063) 0:20:54.779 ******* 2020-11-16T20:01:03.541 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-16T20:01:03.542 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:03.582 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:03 +0000 (0:00:00.067) 0:20:54.846 ******* 2020-11-16T20:01:03.610 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-16T20:01:03.612 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:03.646 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:03 +0000 (0:00:00.064) 0:20:54.911 ******* 2020-11-16T20:01:04.472 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605555408, "cache_updated": false, "changed": false} 2020-11-16T20:01:04.506 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:04 +0000 (0:00:00.859) 0:20:55.770 ******* 2020-11-16T20:01:04.536 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:04.588 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:04 +0000 (0:00:00.079) 0:20:55.850 ******* 2020-11-16T20:01:04.720 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-16T20:01:04.722 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-16T20:01:04.758 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:04 +0000 (0:00:00.172) 0:20:56.023 ******* 2020-11-16T20:01:32.031 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-11-16T20:01:32.033 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-16T20:01:32.065 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:32 +0000 (0:00:27.307) 0:21:23.330 ******* 2020-11-16T20:01:33.068 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-16T20:01:33.070 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-16T20:01:33.112 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:33 +0000 (0:00:01.047) 0:21:24.377 ******* 2020-11-16T20:01:33.635 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-16T20:01:33.637 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false} 2020-11-16T20:01:33.679 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:33 +0000 (0:00:00.566) 0:21:24.944 ******* 2020-11-16T20:01:33.707 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-16T20:01:33.759 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:33 +0000 (0:00:00.079) 0:21:25.023 ******* 2020-11-16T20:01:33.905 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-16T20:01:33.906 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1605554905.0042686, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1605554905.0002687, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 2817, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1605554905.0002687, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "2282523401", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-16T20:01:33.942 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:33 +0000 (0:00:00.183) 0:21:25.207 ******* 2020-11-16T20:01:34.344 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-16T20:01:34.346 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-16T20:01:34.380 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:34 +0000 (0:00:00.437) 0:21:25.644 ******* 2020-11-16T20:01:38.295 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-16T20:01:38.297 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-16T20:01:38.331 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:38 +0000 (0:00:03.950) 0:21:29.595 ******* 2020-11-16T20:01:38.358 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-16T20:01:38.360 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:38.393 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:38 +0000 (0:00:00.061) 0:21:29.657 ******* 2020-11-16T20:01:38.514 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-16T20:01:38.516 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-16T20:01:38.551 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:38 +0000 (0:00:00.158) 0:21:29.816 ******* 2020-11-16T20:01:38.685 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-16T20:01:38.686 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-16T20:01:38.728 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:38 +0000 (0:00:00.176) 0:21:29.992 ******* 2020-11-16T20:01:38.766 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-16T20:01:38.768 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:38.808 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:38 +0000 (0:00:00.080) 0:21:30.072 ******* 2020-11-16T20:01:38.856 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-16T20:01:38.858 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-16T20:01:38.872 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-16T20:01:38.904 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:38 +0000 (0:00:00.096) 0:21:30.169 ******* 2020-11-16T20:01:38.946 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-16T20:01:38.947 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:38.948 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:38.959 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:38.961 INFO:teuthology.task.ansible.out: 2020-11-16T20:01:38.961 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:38.972 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:39.007 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:39 +0000 (0:00:00.103) 0:21:30.272 ******* 2020-11-16T20:01:39.036 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:39.071 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:39 +0000 (0:00:00.063) 0:21:30.336 ******* 2020-11-16T20:01:39.097 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-16T20:01:39.099 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:39.132 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:39 +0000 (0:00:00.060) 0:21:30.396 ******* 2020-11-16T20:01:39.160 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-16T20:01:39.162 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:39.195 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:39 +0000 (0:00:00.063) 0:21:30.460 ******* 2020-11-16T20:01:39.223 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-16T20:01:39.225 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:39.256 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:39 +0000 (0:00:00.060) 0:21:30.521 ******* 2020-11-16T20:01:39.283 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-16T20:01:39.285 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:39.322 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:39 +0000 (0:00:00.065) 0:21:30.586 ******* 2020-11-16T20:01:39.352 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:39.390 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:39 +0000 (0:00:00.067) 0:21:30.654 ******* 2020-11-16T20:01:39.418 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:39.455 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:39 +0000 (0:00:00.065) 0:21:30.719 ******* 2020-11-16T20:01:39.494 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-16T20:01:39.533 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:39 +0000 (0:00:00.076) 0:21:30.796 ******* 2020-11-16T20:01:39.706 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi189.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010854", "end": "2020-11-16 20:01:39.672728", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-16 20:01:39.661874", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T20:01:39.821 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005977", "end": "2020-11-16 20:01:39.792877", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-16 20:01:39.786900", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T20:01:39.937 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003678", "end": "2020-11-16 20:01:39.907529", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-16 20:01:39.903851", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-16T20:01:40.093 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.035121", "end": "2020-11-16 20:01:40.058627", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-16 20:01:40.023506", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-16T20:01:40.133 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:40 +0000 (0:00:00.601) 0:21:31.398 ******* 2020-11-16T20:01:40.162 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-16T20:01:40.196 INFO:teuthology.task.ansible.out:Monday 16 November 2020 20:01:40 +0000 (0:00:00.062) 0:21:31.461 ******* 2020-11-16T20:01:40.328 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-16T20:01:40.329 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-16T20:01:40.433 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-11-16T20:01:40.437 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-11-16T20:01:40.440 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** 2020-11-16T20:01:40.441 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY RECAP ********************************************************************* smithi189.front.sepia.ceph.com : ok=108 changed=27 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 Monday 16 November 2020 20:01:40 +0000 (0:00:00.244) 0:21:31.705 ******* =============================================================================== Gathering Facts ------------------------------------------------------- 521.75s ansible-managed : Delete the ansible users password. ------------------ 236.78s ansible-managed : Create the cephlab_sudo sudoers.d file. ------------- 102.19s common : Upload nagios sudoers.d for raid utilities. ------------------- 39.79s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. -- 36.40s testnode : Create pip.conf and configure it to use our mirror ---------- 27.31s common : Update apt cache ---------------------------------------------- 23.93s testnode : Ensure packages are not present. ---------------------------- 21.06s testnode : Install apt keys -------------------------------------------- 17.91s testnode : Set apt preferences ----------------------------------------- 16.07s testnode : Zap all non-root disks -------------------------------------- 15.85s testnode : Create logical volume(s) ------------------------------------ 13.12s testnode : Remove all LVM data ----------------------------------------- 12.18s common : Remove old apt-mirror repository definition. ------------------ 12.13s testnode : Check for physical volumes ---------------------------------- 12.00s testnode : Upload CPAN config for root. -------------------------------- 11.93s testnode : Install packages --------------------------------------------- 9.33s users : Write /keys-repo-sha1 sentinel file ----------------------------- 8.65s testnode : Ensure perl-doc is installed on apt systems. ----------------- 8.63s users : Check host's /keys-repo-sha1 sentinel fi 2020-11-16T20:01:40.443 INFO:teuthology.task.ansible.out:le ---------------------- 8.31s 2020-11-16T20:01:40.787 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2020-11-16T20:01:40.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T20:01:40.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T20:01:40.911 INFO:teuthology.orchestra.run.smithi189:> true 2020-11-16T20:01:41.115 INFO:teuthology.run_tasks:Running task clock... 2020-11-16T20:01:41.131 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-11-16T20:01:41.131 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-16T20:01:41.132 INFO:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-16T20:01:41.212 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-16T20:01:41.212 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: Command line: ntpd -gq 2020-11-16T20:01:41.213 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: proto: precision = 0.123 usec (-23) 2020-11-16T20:01:41.213 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-16T20:01:41.213 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: restrict ::: KOD does nothing without LIMITED. 2020-11-16T20:01:41.214 INFO:teuthology.orchestra.run.smithi189.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-16T20:01:41.214 INFO:teuthology.orchestra.run.smithi189.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-16T20:01:41.215 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: Listen and drop on 0 v6wildcard [::]:123 2020-11-16T20:01:41.215 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-16T20:01:41.215 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: Listen normally on 2 lo 127.0.0.1:123 2020-11-16T20:01:41.216 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: Listen normally on 3 enp3s0f1 172.21.15.189:123 2020-11-16T20:01:41.216 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: Listen normally on 4 lo [::1]:123 2020-11-16T20:01:41.216 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6df1%5]:123 2020-11-16T20:01:41.216 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:41 ntpd[5849]: Listening on routing socket on fd #22 for interface updates 2020-11-16T20:01:42.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:42 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:42.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:42 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:42.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:42 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:42.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:42 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:42.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:42 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:42.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:42 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:42.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:42 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:42.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:42 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:43.212 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:43 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:43.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:43 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:43.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:43 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:43.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:43 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:44.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:44 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:44.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:44 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:44.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:44 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:44.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:44 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:44.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:44 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:44.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:44 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:44.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:44 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:44.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:44 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:45.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:45 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:45.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:45 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:45.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:45 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:45.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:45 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:45.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:45 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:45.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:45 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:45.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:45 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:45.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:45 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:46 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:46 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:46 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:46 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:46.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:46 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:46.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:46 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:46.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:46 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:46.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:46 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:47.212 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:47 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:47.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:47 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:47.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:47 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:47.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:47 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:47.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:47 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:47.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:47 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:47.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:47 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:47.214 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:47 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:48.212 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:48 ntpd[5849]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-16T20:01:48.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:48 ntpd[5849]: can't open /var/log/ntpstats/rawstats.20201116: Permission denied 2020-11-16T20:01:48.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:48 ntpd[5849]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-16T20:01:48.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:48 ntpd[5849]: can't open /var/log/ntpstats/peerstats.20201116: Permission denied 2020-11-16T20:01:48.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:48 ntpd[5849]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-16T20:01:48.213 INFO:teuthology.orchestra.run.smithi189.stderr:16 Nov 20:01:48 ntpd[5849]: can't open /var/log/ntpstats/loopstats.20201116: Permission denied 2020-11-16T20:01:48.214 INFO:teuthology.orchestra.run.smithi189.stdout:16 Nov 20:01:48 ntpd[5849]: ntpd: time slew -0.000513 s 2020-11-16T20:01:48.214 INFO:teuthology.orchestra.run.smithi189.stdout:ntpd: time slew -0.000513s 2020-11-16T20:01:48.266 INFO:teuthology.orchestra.run.smithi189.stdout: remote refid st t when poll reach delay offset jitter 2020-11-16T20:01:48.267 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================== 2020-11-16T20:01:48.267 INFO:teuthology.orchestra.run.smithi189.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-16T20:01:48.267 INFO:teuthology.orchestra.run.smithi189.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-16T20:01:48.267 INFO:teuthology.orchestra.run.smithi189.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-16T20:01:48.267 INFO:teuthology.orchestra.run.smithi189.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-16T20:01:48.268 INFO:teuthology.run_tasks:Running task install... 2020-11-16T20:01:48.276 DEBUG:teuthology.task.install:project ceph 2020-11-16T20:01:48.277 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '911cc35794d871071a8bebfb0919482263b1d18b'}} 2020-11-16T20:01:48.277 DEBUG:teuthology.task.install:config {'sha1': '911cc35794d871071a8bebfb0919482263b1d18b'} 2020-11-16T20:01:48.277 INFO:teuthology.task.install:Using flavor: basic 2020-11-16T20:01:48.283 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-11-16T20:01:48.284 INFO:teuthology.task.install:extra packages: [] 2020-11-16T20:01:48.284 INFO:teuthology.orchestra.run.smithi189:> sudo lsb_release -is 2020-11-16T20:01:48.333 INFO:teuthology.orchestra.run.smithi189.stdout:Ubuntu 2020-11-16T20:01:48.338 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-16T20:01:48.339 INFO:teuthology.orchestra.run.smithi189:> sudo apt-key list | grep Ceph 2020-11-16T20:01:48.436 INFO:teuthology.orchestra.run.smithi189.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-16T20:01:48.492 INFO:teuthology.orchestra.run.smithi189.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-16T20:01:48.493 INFO:teuthology.orchestra.run.smithi189.stdout:uid [ unknown] Ceph.com (release key) 2020-11-16T20:01:48.494 INFO:teuthology.orchestra.run.smithi189:> sudo lsb_release -is 2020-11-16T20:01:48.544 INFO:teuthology.orchestra.run.smithi189.stdout:Ubuntu 2020-11-16T20:01:48.549 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-16T20:01:48.549 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-16T20:01:48.550 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=911cc35794d871071a8bebfb0919482263b1d18b 2020-11-16T20:01:48.720 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default/ 2020-11-16T20:01:48.720 INFO:teuthology.task.install.deb:Package version is 16.0.0-7219-g911cc357-1bionic 2020-11-16T20:01:49.039 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-16T20:01:49.039 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-16T20:01:49.044 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-16T20:01:49.053 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-16T20:01:49.054 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-16T20:01:49.054 INFO:teuthology.orchestra.run.smithi189.stderr:168 bytes copied, 7.7821e-05 s, 2.2 MB/s 2020-11-16T20:01:49.054 INFO:teuthology.orchestra.run.smithi189:> sudo apt-get update 2020-11-16T20:01:49.232 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-16T20:01:49.346 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-16T20:01:49.427 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-16T20:01:49.464 INFO:teuthology.orchestra.run.smithi189.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic InRelease 2020-11-16T20:01:49.508 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-16T20:01:49.525 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-16T20:01:49.589 INFO:teuthology.orchestra.run.smithi189.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-16T20:01:49.652 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-16T20:01:49.737 INFO:teuthology.orchestra.run.smithi189.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,954 B] 2020-11-16T20:01:49.836 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 26.8 kB in 1s (46.5 kB/s) 2020-11-16T20:01:51.071 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-16T20:01:51.121 INFO:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7219-g911cc357-1bionic cephadm=16.0.0-7219-g911cc357-1bionic ceph-mds=16.0.0-7219-g911cc357-1bionic ceph-mgr=16.0.0-7219-g911cc357-1bionic ceph-common=16.0.0-7219-g911cc357-1bionic ceph-fuse=16.0.0-7219-g911cc357-1bionic ceph-test=16.0.0-7219-g911cc357-1bionic radosgw=16.0.0-7219-g911cc357-1bionic python3-rados=16.0.0-7219-g911cc357-1bionic python3-rgw=16.0.0-7219-g911cc357-1bionic python3-cephfs=16.0.0-7219-g911cc357-1bionic python3-rbd=16.0.0-7219-g911cc357-1bionic libcephfs2=16.0.0-7219-g911cc357-1bionic libcephfs-dev=16.0.0-7219-g911cc357-1bionic librados2=16.0.0-7219-g911cc357-1bionic librbd1=16.0.0-7219-g911cc357-1bionic rbd-fuse=16.0.0-7219-g911cc357-1bionic 2020-11-16T20:01:51.199 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-16T20:01:51.427 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2020-11-16T20:01:51.428 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2020-11-16T20:01:51.591 INFO:teuthology.orchestra.run.smithi189.stdout:The following additional packages will be installed: 2020-11-16T20:01:51.592 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-16T20:01:51.592 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-16T20:01:51.592 INFO:teuthology.orchestra.run.smithi189.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-16T20:01:51.593 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-16T20:01:51.593 INFO:teuthology.orchestra.run.smithi189.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-16T20:01:51.593 INFO:teuthology.orchestra.run.smithi189.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-16T20:01:51.593 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-16T20:01:51.593 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-16T20:01:51.593 INFO:teuthology.orchestra.run.smithi189.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-16T20:01:51.594 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-16T20:01:51.594 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-16T20:01:51.594 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-16T20:01:51.594 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-16T20:01:51.594 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-16T20:01:51.594 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-16T20:01:51.594 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-16T20:01:51.595 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-16T20:01:51.595 INFO:teuthology.orchestra.run.smithi189.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-16T20:01:51.595 INFO:teuthology.orchestra.run.smithi189.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-16T20:01:51.595 INFO:teuthology.orchestra.run.smithi189.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-16T20:01:51.595 INFO:teuthology.orchestra.run.smithi189.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-16T20:01:51.595 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-16T20:01:51.596 INFO:teuthology.orchestra.run.smithi189.stdout:Suggested packages: 2020-11-16T20:01:51.596 INFO:teuthology.orchestra.run.smithi189.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-16T20:01:51.596 INFO:teuthology.orchestra.run.smithi189.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-16T20:01:51.596 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-16T20:01:51.596 INFO:teuthology.orchestra.run.smithi189.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-16T20:01:51.596 INFO:teuthology.orchestra.run.smithi189.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-16T20:01:51.597 INFO:teuthology.orchestra.run.smithi189.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-16T20:01:51.597 INFO:teuthology.orchestra.run.smithi189.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-16T20:01:51.597 INFO:teuthology.orchestra.run.smithi189.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-16T20:01:51.597 INFO:teuthology.orchestra.run.smithi189.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-16T20:01:51.597 INFO:teuthology.orchestra.run.smithi189.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-16T20:01:51.597 INFO:teuthology.orchestra.run.smithi189.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-16T20:01:51.597 INFO:teuthology.orchestra.run.smithi189.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-16T20:01:51.598 INFO:teuthology.orchestra.run.smithi189.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-16T20:01:51.598 INFO:teuthology.orchestra.run.smithi189.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-16T20:01:51.598 INFO:teuthology.orchestra.run.smithi189.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-16T20:01:51.692 INFO:teuthology.orchestra.run.smithi189.stdout:The following NEW packages will be installed: 2020-11-16T20:01:51.692 INFO:teuthology.orchestra.run.smithi189.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-16T20:01:51.693 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-16T20:01:51.693 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-16T20:01:51.694 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-16T20:01:51.694 INFO:teuthology.orchestra.run.smithi189.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-16T20:01:51.694 INFO:teuthology.orchestra.run.smithi189.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-16T20:01:51.694 INFO:teuthology.orchestra.run.smithi189.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-16T20:01:51.694 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-16T20:01:51.694 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-16T20:01:51.695 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-16T20:01:51.695 INFO:teuthology.orchestra.run.smithi189.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-16T20:01:51.695 INFO:teuthology.orchestra.run.smithi189.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-16T20:01:51.695 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-16T20:01:51.695 INFO:teuthology.orchestra.run.smithi189.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-16T20:01:51.695 INFO:teuthology.orchestra.run.smithi189.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-16T20:01:51.695 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-16T20:01:51.696 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-16T20:01:51.696 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-16T20:01:51.696 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-16T20:01:51.696 INFO:teuthology.orchestra.run.smithi189.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-16T20:01:51.696 INFO:teuthology.orchestra.run.smithi189.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-16T20:01:51.696 INFO:teuthology.orchestra.run.smithi189.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-16T20:01:51.696 INFO:teuthology.orchestra.run.smithi189.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-16T20:01:51.697 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-16T20:01:51.697 INFO:teuthology.orchestra.run.smithi189.stdout:The following packages will be upgraded: 2020-11-16T20:01:51.697 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 librbd1 2020-11-16T20:01:51.882 INFO:teuthology.orchestra.run.smithi189.stdout:2 upgraded, 110 newly installed, 0 to remove and 1 not upgraded. 2020-11-16T20:01:51.882 INFO:teuthology.orchestra.run.smithi189.stdout:Need to get 168 MB of archives. 2020-11-16T20:01:51.882 INFO:teuthology.orchestra.run.smithi189.stdout:After this operation, 782 MB of additional disk space will be used. 2020-11-16T20:01:51.882 INFO:teuthology.orchestra.run.smithi189.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-16T20:01:51.961 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-16T20:01:52.110 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7219-g911cc357-1bionic [1,921 kB] 2020-11-16T20:01:52.212 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-16T20:01:52.226 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-16T20:01:52.241 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-16T20:01:52.340 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-16T20:01:52.342 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-16T20:01:52.367 INFO:teuthology.orchestra.run.smithi189.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-16T20:01:52.373 INFO:teuthology.orchestra.run.smithi189.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-16T20:01:52.381 INFO:teuthology.orchestra.run.smithi189.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-16T20:01:52.393 INFO:teuthology.orchestra.run.smithi189.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-16T20:01:52.402 INFO:teuthology.orchestra.run.smithi189.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-16T20:01:52.426 INFO:teuthology.orchestra.run.smithi189.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-16T20:01:52.429 INFO:teuthology.orchestra.run.smithi189.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-16T20:01:52.449 INFO:teuthology.orchestra.run.smithi189.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-16T20:01:52.460 INFO:teuthology.orchestra.run.smithi189.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-16T20:01:52.461 INFO:teuthology.orchestra.run.smithi189.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-16T20:01:52.528 INFO:teuthology.orchestra.run.smithi189.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-16T20:01:52.528 INFO:teuthology.orchestra.run.smithi189.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-16T20:01:52.538 INFO:teuthology.orchestra.run.smithi189.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-16T20:01:52.538 INFO:teuthology.orchestra.run.smithi189.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-16T20:01:52.538 INFO:teuthology.orchestra.run.smithi189.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-16T20:01:52.538 INFO:teuthology.orchestra.run.smithi189.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-16T20:01:52.538 INFO:teuthology.orchestra.run.smithi189.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-16T20:01:52.606 INFO:teuthology.orchestra.run.smithi189.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-16T20:01:52.606 INFO:teuthology.orchestra.run.smithi189.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-16T20:01:52.626 INFO:teuthology.orchestra.run.smithi189.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-16T20:01:52.685 INFO:teuthology.orchestra.run.smithi189.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-16T20:01:52.685 INFO:teuthology.orchestra.run.smithi189.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-16T20:01:52.686 INFO:teuthology.orchestra.run.smithi189.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-16T20:01:52.689 INFO:teuthology.orchestra.run.smithi189.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-16T20:01:52.696 INFO:teuthology.orchestra.run.smithi189.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-16T20:01:52.701 INFO:teuthology.orchestra.run.smithi189.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-16T20:01:52.703 INFO:teuthology.orchestra.run.smithi189.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-16T20:01:52.766 INFO:teuthology.orchestra.run.smithi189.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-16T20:01:52.848 INFO:teuthology.orchestra.run.smithi189.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-16T20:01:52.848 INFO:teuthology.orchestra.run.smithi189.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-16T20:01:53.051 INFO:teuthology.orchestra.run.smithi189.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7219-g911cc357-1bionic [3,157 kB] 2020-11-16T20:01:53.120 INFO:teuthology.orchestra.run.smithi189.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-16T20:01:53.158 INFO:teuthology.orchestra.run.smithi189.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-16T20:01:53.159 INFO:teuthology.orchestra.run.smithi189.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-16T20:01:53.202 INFO:teuthology.orchestra.run.smithi189.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-16T20:01:53.203 INFO:teuthology.orchestra.run.smithi189.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-16T20:01:53.203 INFO:teuthology.orchestra.run.smithi189.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-16T20:01:53.203 INFO:teuthology.orchestra.run.smithi189.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-16T20:01:53.204 INFO:teuthology.orchestra.run.smithi189.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-16T20:01:53.204 INFO:teuthology.orchestra.run.smithi189.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-16T20:01:53.204 INFO:teuthology.orchestra.run.smithi189.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-16T20:01:53.277 INFO:teuthology.orchestra.run.smithi189.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-16T20:01:53.278 INFO:teuthology.orchestra.run.smithi189.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-16T20:01:53.356 INFO:teuthology.orchestra.run.smithi189.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-16T20:01:53.357 INFO:teuthology.orchestra.run.smithi189.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-16T20:01:53.361 INFO:teuthology.orchestra.run.smithi189.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-16T20:01:53.361 INFO:teuthology.orchestra.run.smithi189.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-16T20:01:53.362 INFO:teuthology.orchestra.run.smithi189.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-16T20:01:53.363 INFO:teuthology.orchestra.run.smithi189.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-16T20:01:53.364 INFO:teuthology.orchestra.run.smithi189.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-16T20:01:53.365 INFO:teuthology.orchestra.run.smithi189.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-16T20:01:53.434 INFO:teuthology.orchestra.run.smithi189.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-16T20:01:53.439 INFO:teuthology.orchestra.run.smithi189.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-16T20:01:53.442 INFO:teuthology.orchestra.run.smithi189.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7219-g911cc357-1bionic [479 kB] 2020-11-16T20:01:53.505 INFO:teuthology.orchestra.run.smithi189.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7219-g911cc357-1bionic [332 kB] 2020-11-16T20:01:53.511 INFO:teuthology.orchestra.run.smithi189.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-16T20:01:53.518 INFO:teuthology.orchestra.run.smithi189.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-16T20:01:53.519 INFO:teuthology.orchestra.run.smithi189.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-16T20:01:53.520 INFO:teuthology.orchestra.run.smithi189.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-16T20:01:53.520 INFO:teuthology.orchestra.run.smithi189.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-16T20:01:53.522 INFO:teuthology.orchestra.run.smithi189.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-16T20:01:53.526 INFO:teuthology.orchestra.run.smithi189.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-16T20:01:53.526 INFO:teuthology.orchestra.run.smithi189.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-16T20:01:53.533 INFO:teuthology.orchestra.run.smithi189.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7219-g911cc357-1bionic [20.6 kB] 2020-11-16T20:01:53.533 INFO:teuthology.orchestra.run.smithi189.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7219-g911cc357-1bionic [168 kB] 2020-11-16T20:01:53.555 INFO:teuthology.orchestra.run.smithi189.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7219-g911cc357-1bionic [27.7 kB] 2020-11-16T20:01:53.556 INFO:teuthology.orchestra.run.smithi189.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7219-g911cc357-1bionic [330 kB] 2020-11-16T20:01:53.590 INFO:teuthology.orchestra.run.smithi189.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-16T20:01:53.590 INFO:teuthology.orchestra.run.smithi189.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-16T20:01:53.599 INFO:teuthology.orchestra.run.smithi189.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7219-g911cc357-1bionic [3,177 kB] 2020-11-16T20:01:53.668 INFO:teuthology.orchestra.run.smithi189.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-16T20:01:53.690 INFO:teuthology.orchestra.run.smithi189.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-16T20:01:53.691 INFO:teuthology.orchestra.run.smithi189.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-16T20:01:53.691 INFO:teuthology.orchestra.run.smithi189.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-16T20:01:53.826 INFO:teuthology.orchestra.run.smithi189.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-16T20:01:53.826 INFO:teuthology.orchestra.run.smithi189.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-16T20:01:53.828 INFO:teuthology.orchestra.run.smithi189.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-16T20:01:53.838 INFO:teuthology.orchestra.run.smithi189.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-16T20:01:53.838 INFO:teuthology.orchestra.run.smithi189.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-16T20:01:53.839 INFO:teuthology.orchestra.run.smithi189.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-16T20:01:53.839 INFO:teuthology.orchestra.run.smithi189.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-16T20:01:53.841 INFO:teuthology.orchestra.run.smithi189.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-16T20:01:53.841 INFO:teuthology.orchestra.run.smithi189.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-16T20:01:53.921 INFO:teuthology.orchestra.run.smithi189.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7219-g911cc357-1bionic [99.1 kB] 2020-11-16T20:01:53.938 INFO:teuthology.orchestra.run.smithi189.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7219-g911cc357-1bionic [414 kB] 2020-11-16T20:01:53.981 INFO:teuthology.orchestra.run.smithi189.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7219-g911cc357-1bionic [20.5 MB] 2020-11-16T20:01:55.623 INFO:teuthology.orchestra.run.smithi189.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7219-g911cc357-1bionic [5,566 kB] 2020-11-16T20:01:56.002 INFO:teuthology.orchestra.run.smithi189.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7219-g911cc357-1bionic [165 kB] 2020-11-16T20:01:56.009 INFO:teuthology.orchestra.run.smithi189.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7219-g911cc357-1bionic [1,362 kB] 2020-11-16T20:01:56.100 INFO:teuthology.orchestra.run.smithi189.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7219-g911cc357-1bionic [6,657 kB] 2020-11-16T20:01:56.534 INFO:teuthology.orchestra.run.smithi189.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7219-g911cc357-1bionic [24.6 MB] 2020-11-16T20:01:58.048 INFO:teuthology.orchestra.run.smithi189.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7219-g911cc357-1bionic [3,804 B] 2020-11-16T20:01:58.048 INFO:teuthology.orchestra.run.smithi189.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7219-g911cc357-1bionic [563 kB] 2020-11-16T20:01:58.084 INFO:teuthology.orchestra.run.smithi189.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7219-g911cc357-1bionic [1,942 kB] 2020-11-16T20:01:58.198 INFO:teuthology.orchestra.run.smithi189.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7219-g911cc357-1bionic [51.7 kB] 2020-11-16T20:01:58.201 INFO:teuthology.orchestra.run.smithi189.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7219-g911cc357-1bionic [56.2 kB] 2020-11-16T20:01:58.203 INFO:teuthology.orchestra.run.smithi189.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7219-g911cc357-1bionic [2,876 kB] 2020-11-16T20:01:58.378 INFO:teuthology.orchestra.run.smithi189.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7219-g911cc357-1bionic [7,891 kB] 2020-11-16T20:01:58.839 INFO:teuthology.orchestra.run.smithi189.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7219-g911cc357-1bionic [3,924 B] 2020-11-16T20:01:58.840 INFO:teuthology.orchestra.run.smithi189.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7219-g911cc357-1bionic [22.3 kB] 2020-11-16T20:01:58.840 INFO:teuthology.orchestra.run.smithi189.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7219-g911cc357-1bionic [49.6 MB] 2020-11-16T20:02:02.583 INFO:teuthology.orchestra.run.smithi189.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7219-g911cc357-1bionic [19.5 kB] 2020-11-16T20:02:02.583 INFO:teuthology.orchestra.run.smithi189.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7219-g911cc357-1bionic [9,855 kB] 2020-11-16T20:02:03.340 INFO:teuthology.orchestra.run.smithi189.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-11-16-1845/911cc35794d871071a8bebfb0919482263b1d18b/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7219-g911cc357-1bionic [77.0 kB] 2020-11-16T20:02:04.049 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 168 MB in 12s (14.5 MB/s) 2020-11-16T20:02:04.223 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-16T20:02:04.271 INFO:teuthology.orchestra.run.smithi189.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137607 files and directories currently installed.) 2020-11-16T20:02:04.274 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-16T20:02:04.322 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-16T20:02:04.591 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-16T20:02:04.607 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-16T20:02:04.703 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-16T20:02:05.028 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../002-librbd1_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:05.154 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librbd1 (16.0.0-7219-g911cc357-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-16T20:02:05.699 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../003-librados2_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:05.816 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librados2 (16.0.0-7219-g911cc357-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-16T20:02:06.628 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libcephfs2. 2020-11-16T20:02:06.642 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:06.687 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libcephfs2 (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:07.023 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rados. 2020-11-16T20:02:07.038 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../005-python3-rados_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:07.081 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rados (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:07.400 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-16T20:02:07.414 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-7219-g911cc357-1bionic_all.deb ... 2020-11-16T20:02:07.459 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-ceph-argparse (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:07.753 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cephfs. 2020-11-16T20:02:07.769 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:07.811 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cephfs (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:08.151 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-16T20:02:08.166 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-7219-g911cc357-1bionic_all.deb ... 2020-11-16T20:02:08.214 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-ceph-common (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:08.559 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-prettytable. 2020-11-16T20:02:08.573 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-16T20:02:08.617 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-16T20:02:08.887 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rbd. 2020-11-16T20:02:08.900 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:08.928 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rbd (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:09.256 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-16T20:02:09.272 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-16T20:02:09.356 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-16T20:02:09.676 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-16T20:02:09.689 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-16T20:02:09.726 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-16T20:02:10.095 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librgw2. 2020-11-16T20:02:10.109 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../013-librgw2_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:10.137 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librgw2 (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:10.698 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rgw. 2020-11-16T20:02:10.713 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:10.756 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rgw (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:11.109 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-16T20:02:11.122 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-16T20:02:11.167 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-16T20:02:11.445 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-16T20:02:11.459 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-16T20:02:11.503 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-16T20:02:11.882 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package liboath0. 2020-11-16T20:02:11.898 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-16T20:02:11.940 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-16T20:02:12.276 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libradosstriper1. 2020-11-16T20:02:12.290 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:12.334 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libradosstriper1 (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:12.804 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-common. 2020-11-16T20:02:12.820 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../019-ceph-common_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:12.854 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-common (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:14.718 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-base. 2020-11-16T20:02:14.731 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../020-ceph-base_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:14.777 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-base (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:15.740 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-dateutil. 2020-11-16T20:02:15.753 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-16T20:02:15.790 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-16T20:02:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-16T20:02:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-16T20:02:16.192 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-16T20:02:16.453 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-16T20:02:16.468 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-16T20:02:16.512 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-16T20:02:16.839 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cryptography. 2020-11-16T20:02:16.855 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-16T20:02:16.898 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-16T20:02:17.335 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-openssl. 2020-11-16T20:02:17.349 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-16T20:02:17.393 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-16T20:02:17.662 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-16T20:02:17.675 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-7219-g911cc357-1bionic_all.deb ... 2020-11-16T20:02:17.704 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:18.050 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-16T20:02:18.058 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-16T20:02:18.092 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-16T20:02:18.428 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-16T20:02:18.442 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-16T20:02:18.486 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-16T20:02:18.840 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jwt. 2020-11-16T20:02:18.854 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-16T20:02:18.898 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-16T20:02:19.251 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-logutils. 2020-11-16T20:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-16T20:02:19.343 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-16T20:02:19.645 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-16T20:02:19.660 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-16T20:02:19.704 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-16T20:02:20.040 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-mako. 2020-11-16T20:02:20.054 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-16T20:02:20.098 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-16T20:02:20.401 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-16T20:02:20.417 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-16T20:02:20.460 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-16T20:02:20.821 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-16T20:02:20.834 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-16T20:02:20.871 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-16T20:02:21.207 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-webob. 2020-11-16T20:02:21.221 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-16T20:02:21.266 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-16T20:02:21.568 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-bs4. 2020-11-16T20:02:21.582 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-16T20:02:21.627 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-16T20:02:22.671 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-waitress. 2020-11-16T20:02:22.686 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-16T20:02:22.732 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-16T20:02:23.093 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-tempita. 2020-11-16T20:02:23.107 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-16T20:02:23.143 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-16T20:02:23.445 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-paste. 2020-11-16T20:02:23.461 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-16T20:02:23.504 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-16T20:02:23.857 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-16T20:02:23.870 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-16T20:02:23.915 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-16T20:02:24.243 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-16T20:02:24.257 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-16T20:02:24.301 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-16T20:02:24.604 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-webtest. 2020-11-16T20:02:24.618 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-16T20:02:24.662 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-16T20:02:25.032 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pecan. 2020-11-16T20:02:25.046 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-16T20:02:25.090 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-16T20:02:25.436 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-16T20:02:25.452 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-16T20:02:25.494 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-16T20:02:25.872 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr. 2020-11-16T20:02:25.886 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:25.964 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:26.401 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mon. 2020-11-16T20:02:26.414 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:26.459 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mon (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:27.745 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-osd. 2020-11-16T20:02:27.759 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:27.803 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-osd (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:29.792 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph. 2020-11-16T20:02:29.808 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../048-ceph_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:29.844 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:30.153 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-fuse. 2020-11-16T20:02:30.169 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:30.211 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-fuse (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:30.656 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mds. 2020-11-16T20:02:30.671 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:30.747 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mds (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:31.236 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package cephadm. 2020-11-16T20:02:31.249 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../051-cephadm_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:31.317 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking cephadm (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:31.553 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jinja2. 2020-11-16T20:02:31.566 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-16T20:02:31.611 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-16T20:02:31.964 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-16T20:02:31.978 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-7219-g911cc357-1bionic_all.deb ... 2020-11-16T20:02:32.022 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:32.342 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-16T20:02:32.356 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-16T20:02:32.392 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-16T20:02:32.720 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-routes. 2020-11-16T20:02:32.734 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-16T20:02:32.778 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-16T20:02:33.139 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-16T20:02:33.153 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-7219-g911cc357-1bionic_all.deb ... 2020-11-16T20:02:33.197 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:34.138 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-numpy. 2020-11-16T20:02:34.154 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-16T20:02:34.196 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-16T20:02:34.835 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-decorator. 2020-11-16T20:02:34.851 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-16T20:02:34.893 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-16T20:02:35.196 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-scipy. 2020-11-16T20:02:35.210 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-16T20:02:35.254 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-16T20:02:36.621 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-16T20:02:36.638 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-16T20:02:36.679 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-16T20:02:37.216 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-joblib. 2020-11-16T20:02:37.230 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-16T20:02:37.274 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-16T20:02:37.636 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-sklearn. 2020-11-16T20:02:37.650 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-16T20:02:37.694 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-16T20:02:38.300 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-16T20:02:38.314 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-7219-g911cc357-1bionic_all.deb ... 2020-11-16T20:02:38.400 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:39.478 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-monotonic. 2020-11-16T20:02:39.486 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-16T20:02:39.553 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-16T20:02:39.864 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-fasteners. 2020-11-16T20:02:39.878 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-16T20:02:39.922 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-16T20:02:40.242 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-httplib2. 2020-11-16T20:02:40.258 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-16T20:02:40.300 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-16T20:02:40.619 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-16T20:02:40.634 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-16T20:02:40.711 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-16T20:02:41.048 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-16T20:02:41.064 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-16T20:02:41.106 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-16T20:02:41.459 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rsa. 2020-11-16T20:02:41.474 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-16T20:02:41.517 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-16T20:02:41.903 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-16T20:02:41.918 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-16T20:02:41.962 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-16T20:02:42.432 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-16T20:02:42.449 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-16T20:02:42.492 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-16T20:02:42.835 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-setuptools. 2020-11-16T20:02:42.850 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-16T20:02:42.894 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-16T20:02:43.332 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-websocket. 2020-11-16T20:02:43.345 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-16T20:02:43.388 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-16T20:02:43.733 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-16T20:02:43.748 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-16T20:02:43.791 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-16T20:02:44.237 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-16T20:02:44.252 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-7219-g911cc357-1bionic_all.deb ... 2020-11-16T20:02:44.295 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:44.614 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-16T20:02:44.629 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-16T20:02:44.672 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-16T20:02:45.051 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-16T20:02:45.065 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-16T20:02:45.112 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-16T20:02:45.462 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-16T20:02:45.477 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-7219-g911cc357-1bionic_all.deb ... 2020-11-16T20:02:45.528 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-rook (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:45.886 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-16T20:02:45.902 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-16T20:02:45.948 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-16T20:02:46.309 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-16T20:02:46.324 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-16T20:02:46.478 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-16T20:02:46.764 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package jq. 2020-11-16T20:02:46.778 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-16T20:02:46.822 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-16T20:02:47.117 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package socat. 2020-11-16T20:02:47.132 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-16T20:02:47.175 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-16T20:02:47.545 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-16T20:02:47.561 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-16T20:02:47.645 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-16T20:02:47.939 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package xmlstarlet. 2020-11-16T20:02:47.956 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-16T20:02:47.997 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-16T20:02:48.377 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-test. 2020-11-16T20:02:48.392 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../085-ceph-test_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:48.435 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-test (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:52.056 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-16T20:02:52.070 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:02:52.114 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libcephfs-dev (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:02:52.400 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package nvme-cli. 2020-11-16T20:02:52.414 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-16T20:02:52.458 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-16T20:02:52.831 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-attr. 2020-11-16T20:02:52.847 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-16T20:02:52.896 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-16T20:02:53.198 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-funcsigs. 2020-11-16T20:02:53.213 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-16T20:02:53.256 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-16T20:02:53.593 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-pluggy. 2020-11-16T20:02:53.606 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-16T20:02:53.651 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-16T20:02:54.020 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-py. 2020-11-16T20:02:54.035 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-16T20:02:54.078 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-16T20:02:54.381 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-pytest. 2020-11-16T20:02:54.399 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-16T20:02:54.440 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-16T20:02:54.828 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-attr. 2020-11-16T20:02:54.841 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-16T20:02:54.885 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-16T20:02:55.221 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cycler. 2020-11-16T20:02:55.238 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-16T20:02:55.279 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-16T20:02:55.598 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-webencodings. 2020-11-16T20:02:55.612 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-16T20:02:55.656 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-16T20:02:56.043 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-html5lib. 2020-11-16T20:02:56.057 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-16T20:02:56.101 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-16T20:02:56.487 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-16T20:02:56.503 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-16T20:02:56.545 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-16T20:02:57.015 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-16T20:02:57.030 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-16T20:02:57.073 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-16T20:02:57.968 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-tz. 2020-11-16T20:02:57.985 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-16T20:02:58.059 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-16T20:02:58.362 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-16T20:02:58.376 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-16T20:02:58.420 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-16T20:02:59.143 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-olefile. 2020-11-16T20:02:59.158 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-16T20:02:59.227 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-16T20:02:59.554 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pastescript. 2020-11-16T20:02:59.569 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-16T20:02:59.613 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-16T20:02:59.991 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-16T20:03:00.005 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-16T20:03:00.049 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-16T20:03:00.428 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pluggy. 2020-11-16T20:03:00.444 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-16T20:03:00.485 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-16T20:03:00.788 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-py. 2020-11-16T20:03:00.804 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-16T20:03:00.846 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-16T20:03:01.191 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-16T20:03:01.205 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-16T20:03:01.249 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-16T20:03:01.602 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pytest. 2020-11-16T20:03:01.617 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-16T20:03:01.661 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-16T20:03:02.014 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-simplejson. 2020-11-16T20:03:02.028 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-16T20:03:02.072 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-16T20:03:02.442 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-16T20:03:02.459 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-16T20:03:02.500 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-16T20:03:02.987 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package radosgw. 2020-11-16T20:03:03.002 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../110-radosgw_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:03:03.045 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking radosgw (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:04.073 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package rbd-fuse. 2020-11-16T20:03:04.089 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-7219-g911cc357-1bionic_amd64.deb ... 2020-11-16T20:03:04.131 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking rbd-fuse (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:04.488 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-16T20:03:04.778 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-16T20:03:05.154 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-16T20:03:05.280 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-16T20:03:05.874 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-16T20:03:06.253 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-16T20:03:06.529 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-16T20:03:06.830 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-16T20:03:07.041 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-16T20:03:07.320 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-16T20:03:07.509 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-16T20:03:07.874 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-16T20:03:08.186 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-16T20:03:08.488 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-ceph-argparse (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:08.822 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-16T20:03:09.156 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-16T20:03:09.282 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-16T20:03:10.210 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-16T20:03:10.561 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-16T20:03:10.876 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-py (1.5.2-1) ... 2020-11-16T20:03:11.114 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-16T20:03:11.482 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-16T20:03:11.571 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-16T20:03:11.817 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-16T20:03:11.951 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-16T20:03:12.185 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-16T20:03:12.478 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-16T20:03:12.796 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-16T20:03:13.105 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-16T20:03:13.507 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-16T20:03:13.850 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-16T20:03:14.159 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up cephadm (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:14.845 INFO:teuthology.orchestra.run.smithi189.stdout:Adding system user cephadm....done 2020-11-16T20:03:15.032 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-16T20:03:15.333 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-16T20:03:15.459 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-16T20:03:15.619 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-16T20:03:15.961 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-16T20:03:16.271 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-16T20:03:16.397 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-16T20:03:16.748 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-ceph-common (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:17.066 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-16T20:03:17.376 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-16T20:03:17.502 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-16T20:03:17.861 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-16T20:03:18.213 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-16T20:03:18.302 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-16T20:03:18.343 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-16T20:03:18.581 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-16T20:03:18.899 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-16T20:03:19.225 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-16T20:03:19.601 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-16T20:03:19.911 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-16T20:03:20.145 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-16T20:03:20.563 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-16T20:03:20.661 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-16T20:03:20.932 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-16T20:03:21.107 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-16T20:03:21.234 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-16T20:03:21.510 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-16T20:03:21.636 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-16T20:03:22.012 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-16T20:03:22.138 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-16T20:03:22.423 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-16T20:03:22.766 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-16T20:03:23.087 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-16T20:03:23.393 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-16T20:03:23.762 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-16T20:03:24.070 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-16T20:03:24.374 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-16T20:03:24.689 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-16T20:03:25.099 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-16T20:03:25.401 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-16T20:03:25.544 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-16T20:03:26.707 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-16T20:03:27.025 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-16T20:03:27.384 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-16T20:03:27.527 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-16T20:03:27.903 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-16T20:03:28.163 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-16T20:03:28.764 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-16T20:03:28.915 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librados2 (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:29.041 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-16T20:03:29.367 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libcephfs2 (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:29.543 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libcephfs-dev (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:29.669 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rados (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:29.795 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-16T20:03:30.104 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-16T20:03:30.405 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-16T20:03:30.732 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-16T20:03:31.653 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-16T20:03:32.482 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-16T20:03:32.783 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libradosstriper1 (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:32.926 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cephfs (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:33.252 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librgw2 (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:33.387 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-fuse (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:33.600 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-16T20:03:33.600 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-16T20:03:33.880 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librbd1 (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:34.006 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-16T20:03:34.137 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-16T20:03:34.169 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-16T20:03:34.459 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-16T20:03:35.145 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-modules-core (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:35.304 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-16T20:03:35.630 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rgw (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:35.756 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up rbd-fuse (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:35.907 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rbd (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:36.033 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-common (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:36.609 INFO:teuthology.orchestra.run.smithi189.stdout:Adding group ceph....done 2020-11-16T20:03:36.987 INFO:teuthology.orchestra.run.smithi189.stdout:Adding system user ceph....done 2020-11-16T20:03:37.213 INFO:teuthology.orchestra.run.smithi189.stdout:Setting system user ceph properties....done 2020-11-16T20:03:37.377 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-16T20:03:37.594 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-16T20:03:37.886 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up radosgw (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:38.145 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-16T20:03:38.145 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-16T20:03:38.596 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-test (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:38.742 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-base (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:39.076 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-16T20:03:39.365 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:39.583 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-16T20:03:39.584 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-16T20:03:39.858 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:40.026 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-osd (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:40.484 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-16T20:03:40.484 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-16T20:03:40.770 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-cephadm (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:40.913 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-rook (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:41.097 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mds (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:41.316 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-16T20:03:41.316 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-16T20:03:41.590 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mon (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:41.840 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-16T20:03:41.840 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-16T20:03:42.125 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:42.284 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-dashboard (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:42.444 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph (16.0.0-7219-g911cc357-1bionic) ... 2020-11-16T20:03:42.570 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-16T20:03:42.771 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-16T20:03:44.815 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-16T20:03:44.858 INFO:teuthology.orchestra.run.smithi189.stdout:ureadahead will be reprofiled on next reboot 2020-11-16T20:03:44.916 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-16T20:03:45.415 INFO:teuthology.orchestra.run.smithi189.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-16T20:03:45.418 DEBUG:teuthology.parallel:result is None 2020-11-16T20:03:45.419 INFO:teuthology.orchestra.run.smithi189:> sudo lsb_release -is 2020-11-16T20:03:45.465 INFO:teuthology.orchestra.run.smithi189.stdout:Ubuntu 2020-11-16T20:03:45.469 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-16T20:03:45.470 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=911cc35794d871071a8bebfb0919482263b1d18b 2020-11-16T20:03:45.633 INFO:teuthology.orchestra.run.smithi189:> dpkg-query -W -f '${Version}' ceph 2020-11-16T20:03:45.657 INFO:teuthology.orchestra.run.smithi189.stdout:16.0.0-7219-g911cc357-1bionic 2020-11-16T20:03:45.657 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7219-g911cc357-1bionic 2020-11-16T20:03:45.658 INFO:teuthology.task.install:The correct ceph version 16.0.0-7219-g911cc357-1bionic is installed. 2020-11-16T20:03:45.658 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-11-16T20:03:45.658 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-16T20:03:45.659 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-16T20:03:45.703 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-16T20:03:45.711 INFO:teuthology.orchestra.run.smithi189.stderr:24+1 records in 2020-11-16T20:03:45.711 INFO:teuthology.orchestra.run.smithi189.stderr:24+1 records out 2020-11-16T20:03:45.712 INFO:teuthology.orchestra.run.smithi189.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000122833 s, 103 MB/s 2020-11-16T20:03:45.713 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-11-16T20:03:45.713 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-16T20:03:45.713 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2020-11-16T20:03:45.759 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-16T20:03:45.767 INFO:teuthology.orchestra.run.smithi189.stderr:5+1 records in 2020-11-16T20:03:45.768 INFO:teuthology.orchestra.run.smithi189.stderr:5+1 records out 2020-11-16T20:03:45.768 INFO:teuthology.orchestra.run.smithi189.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.7624e-05 s, 31.3 MB/s 2020-11-16T20:03:45.769 INFO:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-16T20:03:45.830 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-11-16T20:03:45.831 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-16T20:03:45.831 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-16T20:03:45.875 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-16T20:03:45.884 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-16T20:03:45.884 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-16T20:03:45.884 INFO:teuthology.orchestra.run.smithi189.stderr:299 bytes copied, 6.9047e-05 s, 4.3 MB/s 2020-11-16T20:03:45.885 INFO:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-16T20:03:45.940 INFO:teuthology.run_tasks:Running task workunit... 2020-11-16T20:03:45.958 INFO:tasks.workunit:Pulling workunits from ref 911cc35794d871071a8bebfb0919482263b1d18b 2020-11-16T20:03:45.960 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-11-16T20:03:45.960 INFO:tasks.workunit:timeout=3h 2020-11-16T20:03:45.960 INFO:tasks.workunit:cleanup=True 2020-11-16T20:03:45.961 INFO:teuthology.orchestra.run.smithi189:> stat -- /home/ubuntu/cephtest/mnt.0 2020-11-16T20:03:45.988 INFO:teuthology.orchestra.run.smithi189.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-11-16T20:03:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-16T20:03:45.989 INFO:teuthology.orchestra.run.smithi189:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-11-16T20:03:46.036 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-11-16T20:03:46.037 INFO:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-11-16T20:03:46.084 INFO:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 911cc35794d871071a8bebfb0919482263b1d18b 2020-11-16T20:03:46.194 INFO:tasks.workunit.client.0.smithi189.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-11-16T20:04:42.810 INFO:tasks.workunit.client.0.smithi189.stderr:Note: checking out '911cc35794d871071a8bebfb0919482263b1d18b'. 2020-11-16T20:04:42.810 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T20:04:42.811 INFO:tasks.workunit.client.0.smithi189.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-11-16T20:04:42.811 INFO:tasks.workunit.client.0.smithi189.stderr:changes and commit them, and you can discard any commits you make in this 2020-11-16T20:04:42.811 INFO:tasks.workunit.client.0.smithi189.stderr:state without impacting any branches by performing another checkout. 2020-11-16T20:04:42.811 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T20:04:42.811 INFO:tasks.workunit.client.0.smithi189.stderr:If you want to create a new branch to retain commits you create, you may 2020-11-16T20:04:42.811 INFO:tasks.workunit.client.0.smithi189.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-11-16T20:04:42.811 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T20:04:42.812 INFO:tasks.workunit.client.0.smithi189.stderr: git checkout -b 2020-11-16T20:04:42.812 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T20:04:42.812 INFO:tasks.workunit.client.0.smithi189.stderr:HEAD is now at 911cc35794 Merge branch 'wip-mgr-boost-cleanup' of https://github.com/tchaikov/ceph into wip-kefu-testing-2020-11-16-1845 2020-11-16T20:04:42.814 INFO:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-11-16T20:04:42.822 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-16T20:04:42.822 INFO:teuthology.orchestra.run.smithi189:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2020-11-16T20:04:42.869 INFO:teuthology.orchestra.run.smithi189.stderr:+ dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2020-11-16T20:04:42.870 INFO:teuthology.orchestra.run.smithi189.stdout:scrub/osd-scrub-test.shscrub/osd-scrub-dump.shscrub/osd-recovery-scrub.shscrub/osd-scrub-snaps.shscrub/osd-scrub-repair.shscrub/osd-unexpected-clone.shspecial/test-failure.shspecial/ceph_objectstore_tool.pymisc/ok-to-stop.shmisc/network-ping.shmisc/test-ceph-helpers.shmisc/rados-striper.shmisc/ver-health.shmon/osd-pool-df.shmon/mon-handle-forward.shmon/mkfs.shmon/osd-erasure-code-profile.shmon/mon-seesaw.shmon/osd-crush.shmon/osd-pool-create.shmon/misc.shmon/osd-df.shmon/mon-last-epoch-clean.shmon/mon-bind.shmon/test_pool_quota.shmon/mon-osdmap-prune.shmon/mon-ping.shmon/health-mute.shmon/mon-scrub.shmon/mon-created-time.shceph-helpers.shmgr/balancer.sherasure-code/test-erasure-code.sherasure-code/test-erasure-code-plugins.sherasure-code/test-erasure-eio.shcrush/crush-choose-args.shcrush/crush-classes.shosd/osd-force-create-pg.shosd/ec-error-rollforward.shosd/osd-rep-recov-eio.shosd/repeer-on-acting-back.shosd/divergent-priors.shosd/osd-config.shosd/osd-reuse-id.shosd/bad-inc-map.shosd/osd-recovery-stats.shosd/osd-recovery-space.shosd/osd-backfill-prio.shosd/repro_long_log.shosd/osd-backfill-recovery-log.shosd/osd-fast-mark-down.shosd/pg-split-merge.shosd/osd-bench.shosd/osd-recovery-prio.shosd/osd-copy-from.shosd/osd-backfill-space.shosd/osd-markdown.shosd/osd-bluefs-volume-ops.shosd/osd-dup.shosd/osd-reactivate.shosd/osd-backfill-stats.sh 2020-11-16T20:04:42.871 INFO:teuthology.orchestra.run.smithi189.stderr:2+1 records in 2020-11-16T20:04:42.871 INFO:teuthology.orchestra.run.smithi189.stderr:2+1 records out 2020-11-16T20:04:42.871 INFO:teuthology.orchestra.run.smithi189.stderr:1423 bytes (1.4 kB, 1.4 KiB) copied, 0.000153581 s, 9.3 MB/s 2020-11-16T20:04:42.871 INFO:tasks.workunit:Running workunits matching osd on client.0... 2020-11-16T20:04:42.872 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2020-11-16T20:04:42.872 INFO:teuthology.orchestra.run.smithi189:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2020-11-16T20:04:42.936 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T20:04:42.981 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T20:04:42.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:04:42.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:04:42.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T20:04:42.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T20:04:42.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T20:04:42.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T20:04:42.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T20:04:42.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T20:04:42.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/bad-inc-map 2020-11-16T20:04:42.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2020-11-16T20:04:42.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2020-11-16T20:04:42.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:55461 2020-11-16T20:04:42.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:55461 2020-11-16T20:04:42.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2020-11-16T20:04:42.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2020-11-16T20:04:42.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=429d7f6d-bc80-4329-ac6c-a8eccc57786d --auth-supported=none ' 2020-11-16T20:04:42.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:55461 ' 2020-11-16T20:04:42.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2020-11-16T20:04:42.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2020-11-16T20:04:42.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T20:04:42.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2020-11-16T20:04:42.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2020-11-16T20:04:42.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2020-11-16T20:04:42.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2020-11-16T20:04:42.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2020-11-16T20:04:42.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2020-11-16T20:04:42.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:04:42.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2020-11-16T20:04:42.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:04:42.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:04:42.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:04:42.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:04:42.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:04:43.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:04:43.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:04:43.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:04:43.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:04:43.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:04:43.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:04:43.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:04:43.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:04:43.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:04:43.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:04:43.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:04:43.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:04:43.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:04:43.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2020-11-16T20:04:43.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:04:43.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:04:43.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.9317 2020-11-16T20:04:43.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.9317 2020-11-16T20:04:43.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:04:43.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:04:43.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2020-11-16T20:04:43.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:04:43.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:04:43.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.9317 2020-11-16T20:04:43.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.9317 2020-11-16T20:04:43.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:04:43.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T20:04:43.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T20:04:43.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:04:43.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2020-11-16T20:04:43.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2020-11-16T20:04:43.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2020-11-16T20:04:43.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2020-11-16T20:04:43.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2020-11-16T20:04:43.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:04:43.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:04:43.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:04:43.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2020-11-16T20:04:43.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2020-11-16T20:04:43.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:04:43.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:04:43.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:04:43.433 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:04:43.433 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:04:43.434 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.9317 2020-11-16T20:04:43.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.9317/$cluster-$name.asok' 2020-11-16T20:04:43.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.9317/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:04:43.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:04:43.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:04:43.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:04:43.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:04:43.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:04:43.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:04:43.639 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:04:43.640 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:04:43.640 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:04:43.640 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:04:43.641 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:04:43.641 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.9317 2020-11-16T20:04:43.641 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.9317/ceph-mon.a.asok 2020-11-16T20:04:43.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:04:43.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.9317/ceph-mon.a.asok config get fsid 2020-11-16T20:04:44.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:04:44.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:04:44.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:04:44.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:04:44.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:04:44.094 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:04:44.094 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:04:44.094 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:04:44.094 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:04:44.094 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:04:44.094 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.9317 2020-11-16T20:04:44.095 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.9317/ceph-mon.a.asok 2020-11-16T20:04:44.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:04:44.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.9317/ceph-mon.a.asok config get mon_host 2020-11-16T20:04:44.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2020-11-16T20:04:44.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/bad-inc-map 2020-11-16T20:04:44.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:04:44.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:04:44.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:04:44.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/bad-inc-map/x 2020-11-16T20:04:44.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:04:44.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:04:44.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:04:44.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:04:44.523 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:04:44.523 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:04:44.523 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.9317 2020-11-16T20:04:44.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.9317/$cluster-$name.asok' 2020-11-16T20:04:44.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:04:44.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.9317/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:04:44.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2020-11-16T20:04:44.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2020-11-16T20:04:44.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:04:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:04:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:04:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/0 2020-11-16T20:04:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=429d7f6d-bc80-4329-ac6c-a8eccc57786d --auth-supported=none --mon-host=127.0.0.1:55461 ' 2020-11-16T20:04:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:04:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:04:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:04:44.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2020-11-16T20:04:44.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2020-11-16T20:04:44.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:04:44.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:04:44.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2020-11-16T20:04:44.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:04:44.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:04:44.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:04:44.553 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:04:44.553 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:04:44.553 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.9317 2020-11-16T20:04:44.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.9317/$cluster-$name.asok' 2020-11-16T20:04:44.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.9317/$cluster-$name.asok' 2020-11-16T20:04:44.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:04:44.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:04:44.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:04:44.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2020-11-16T20:04:44.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2020-11-16T20:04:44.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:04:44.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:04:44.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:04:44.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:04:44.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:04:44.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/0 2020-11-16T20:04:44.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:04:44.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ac81eb58-e98d-4575-b1af-d7e152179ec6 2020-11-16T20:04:44.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 ac81eb58-e98d-4575-b1af-d7e152179ec6' 2020-11-16T20:04:44.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:04:44.556 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 ac81eb58-e98d-4575-b1af-d7e152179ec6 2020-11-16T20:04:44.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBc27JfwZ8TIhAAgKO9YdHwLQoyUiwuZyv70g== 2020-11-16T20:04:44.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBc27JfwZ8TIhAAgKO9YdHwLQoyUiwuZyv70g=="}' 2020-11-16T20:04:44.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ac81eb58-e98d-4575-b1af-d7e152179ec6 -i td/bad-inc-map/0/new.json 2020-11-16T20:04:44.936 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:04:44.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/0/new.json 2020-11-16T20:04:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=429d7f6d-bc80-4329-ac6c-a8eccc57786d --auth-supported=none --mon-host=127.0.0.1:55461 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.9317/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBc27JfwZ8TIhAAgKO9YdHwLQoyUiwuZyv70g== --osd-uuid ac81eb58-e98d-4575-b1af-d7e152179ec6 2020-11-16T20:04:45.011 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:45.010+0000 7f94028a0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:45.013 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:45.014+0000 7f94028a0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:45.015 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:45.014+0000 7f94028a0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:45.016 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:45.014+0000 7f94028a0d80 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2020-11-16T20:04:45.016 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:45.014+0000 7f94028a0d80 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2020-11-16T20:04:45.016 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:45.014+0000 7f94028a0d80 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2020-11-16T20:04:45.106 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:45.106+0000 7f94028a0d80 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2020-11-16T20:04:49.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/0/keyring 2020-11-16T20:04:49.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:04:49.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:04:49.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:04:49.746 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:04:50.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:04:50.013 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:04:50.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=429d7f6d-bc80-4329-ac6c-a8eccc57786d --auth-supported=none --mon-host=127.0.0.1:55461 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.9317/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:04:50.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:04:50.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:04:50.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:04:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:50.038+0000 7f8bc6116d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:50.041 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:50.038+0000 7f8bc6116d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:50.041 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:50.042+0000 7f8bc6116d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:50.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:04:50.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:04:50.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:04:50.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:04:50.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:04:50.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:04:50.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:04:50.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:04:50.248 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:04:50.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:04:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:04:50.596 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:50.594+0000 7f8bc6116d80 -1 Falling back to public interface 2020-11-16T20:04:51.506 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:51.506+0000 7f8bc6116d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:04:51.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:04:51.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:04:51.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:04:51.542 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:04:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:04:51.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:04:51.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:04:52.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:04:52.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:04:52.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:04:52.958 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:04:52.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:04:52.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:04:53.341 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/10122,v1:127.0.0.1:6803/10122] [v2:127.0.0.1:6804/10122,v1:127.0.0.1:6805/10122] exists,up ac81eb58-e98d-4575-b1af-d7e152179ec6 2020-11-16T20:04:53.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:04:53.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:04:53.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:04:53.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2020-11-16T20:04:53.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2020-11-16T20:04:53.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:04:53.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:04:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:04:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/1 2020-11-16T20:04:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=429d7f6d-bc80-4329-ac6c-a8eccc57786d --auth-supported=none --mon-host=127.0.0.1:55461 ' 2020-11-16T20:04:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:04:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:04:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:04:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2020-11-16T20:04:53.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2020-11-16T20:04:53.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:04:53.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:04:53.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2020-11-16T20:04:53.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:04:53.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:04:53.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:04:53.346 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:04:53.346 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:04:53.346 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.9317 2020-11-16T20:04:53.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.9317/$cluster-$name.asok' 2020-11-16T20:04:53.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.9317/$cluster-$name.asok' 2020-11-16T20:04:53.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:04:53.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:04:53.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:04:53.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2020-11-16T20:04:53.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2020-11-16T20:04:53.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:04:53.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:04:53.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:04:53.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:04:53.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:04:53.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/1 2020-11-16T20:04:53.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:04:53.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8e5157c6-2e34-487a-b351-fe905960811a 2020-11-16T20:04:53.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 8e5157c6-2e34-487a-b351-fe905960811a' 2020-11-16T20:04:53.349 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 8e5157c6-2e34-487a-b351-fe905960811a 2020-11-16T20:04:53.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:04:53.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBl27Jf95i7FRAAiOojEm2/CeEM7xFKgNU/eg== 2020-11-16T20:04:53.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBl27Jf95i7FRAAiOojEm2/CeEM7xFKgNU/eg=="}' 2020-11-16T20:04:53.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8e5157c6-2e34-487a-b351-fe905960811a -i td/bad-inc-map/1/new.json 2020-11-16T20:04:53.838 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:04:53.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/1/new.json 2020-11-16T20:04:53.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=429d7f6d-bc80-4329-ac6c-a8eccc57786d --auth-supported=none --mon-host=127.0.0.1:55461 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.9317/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBl27Jf95i7FRAAiOojEm2/CeEM7xFKgNU/eg== --osd-uuid 8e5157c6-2e34-487a-b351-fe905960811a 2020-11-16T20:04:53.877 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:53.878+0000 7fde01a01d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:53.880 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:53.878+0000 7fde01a01d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:53.881 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:53.882+0000 7fde01a01d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:53.882 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:53.882+0000 7fde01a01d80 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2020-11-16T20:04:53.882 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:53.882+0000 7fde01a01d80 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2020-11-16T20:04:53.882 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:53.882+0000 7fde01a01d80 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2020-11-16T20:04:53.941 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:53.938+0000 7fde01a01d80 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2020-11-16T20:04:58.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/1/keyring 2020-11-16T20:04:58.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:04:58.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:04:58.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:04:58.258 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:04:58.746 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:04:58.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:04:58.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=429d7f6d-bc80-4329-ac6c-a8eccc57786d --auth-supported=none --mon-host=127.0.0.1:55461 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.9317/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:04:58.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:04:58.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:04:58.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:04:58.771 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:58.770+0000 7f8a313f7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:58.773 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:58.770+0000 7f8a313f7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:58.773 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:58.774+0000 7f8a313f7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:04:59.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:04:59.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:04:59.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:04:59.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:04:59.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:04:59.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:04:59.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:04:59.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:04:59.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:04:59.096 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:04:59.307 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:59.306+0000 7f8a313f7d80 -1 Falling back to public interface 2020-11-16T20:04:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:04:59.899 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:04:59.894+0000 7f8a313f7d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:05:00.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:05:00.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:05:00.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:05:00.484 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:05:00.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:00.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:05:00.847 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/11193,v1:127.0.0.1:6811/11193] [v2:127.0.0.1:6812/11193,v1:127.0.0.1:6813/11193] exists,up 8e5157c6-2e34-487a-b351-fe905960811a 2020-11-16T20:05:00.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:05:00.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:05:00.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:05:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2020-11-16T20:05:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2020-11-16T20:05:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:05:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:05:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:05:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/2 2020-11-16T20:05:00.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=429d7f6d-bc80-4329-ac6c-a8eccc57786d --auth-supported=none --mon-host=127.0.0.1:55461 ' 2020-11-16T20:05:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:05:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:05:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:05:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2020-11-16T20:05:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2020-11-16T20:05:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:05:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:05:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2020-11-16T20:05:00.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:05:00.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:05:00.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:05:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:05:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.9317 2020-11-16T20:05:00.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.9317/$cluster-$name.asok' 2020-11-16T20:05:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.9317/$cluster-$name.asok' 2020-11-16T20:05:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:05:00.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:05:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:05:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2020-11-16T20:05:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2020-11-16T20:05:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:05:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:05:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:05:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:05:00.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:05:00.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/2 2020-11-16T20:05:00.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:05:00.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3f023d1a-e170-4d1e-a7d2-b855746263e6 2020-11-16T20:05:00.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 3f023d1a-e170-4d1e-a7d2-b855746263e6' 2020-11-16T20:05:00.855 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 3f023d1a-e170-4d1e-a7d2-b855746263e6 2020-11-16T20:05:00.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:05:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBs27JfiowtNBAAuxHXyKGmTsai5rNR6T9JnQ== 2020-11-16T20:05:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBs27JfiowtNBAAuxHXyKGmTsai5rNR6T9JnQ=="}' 2020-11-16T20:05:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3f023d1a-e170-4d1e-a7d2-b855746263e6 -i td/bad-inc-map/2/new.json 2020-11-16T20:05:01.391 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:05:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/2/new.json 2020-11-16T20:05:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=429d7f6d-bc80-4329-ac6c-a8eccc57786d --auth-supported=none --mon-host=127.0.0.1:55461 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.9317/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBs27JfiowtNBAAuxHXyKGmTsai5rNR6T9JnQ== --osd-uuid 3f023d1a-e170-4d1e-a7d2-b855746263e6 2020-11-16T20:05:01.425 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:01.426+0000 7ff4686b3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:01.429 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:01.430+0000 7ff4686b3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:01.431 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:01.430+0000 7ff4686b3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:01.430+0000 7ff4686b3d80 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2020-11-16T20:05:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:01.430+0000 7ff4686b3d80 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2020-11-16T20:05:01.432 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:01.430+0000 7ff4686b3d80 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2020-11-16T20:05:01.486 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:01.486+0000 7ff4686b3d80 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2020-11-16T20:05:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/2/keyring 2020-11-16T20:05:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:05:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:05:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:05:05.551 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:05:06.133 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:05:06.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:05:06.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=429d7f6d-bc80-4329-ac6c-a8eccc57786d --auth-supported=none --mon-host=127.0.0.1:55461 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.9317/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:05:06.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:05:06.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:05:06.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:05:06.162 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:06.157+0000 7fa19f6cdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:06.165 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:06.161+0000 7fa19f6cdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:06.167 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:06.165+0000 7fa19f6cdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:06.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:05:06.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:05:06.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:05:06.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:05:06.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:05:06.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:05:06.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:05:06.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:06.543 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:05:06.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:05:06.720 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:06.716+0000 7fa19f6cdd80 -1 Falling back to public interface 2020-11-16T20:05:06.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:05:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:07.580+0000 7fa19f6cdd80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:05:07.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:05:07.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:05:07.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:05:07.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:07.959 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:05:07.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:05:08.339 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/12190,v1:127.0.0.1:6819/12190] [v2:127.0.0.1:6820/12190,v1:127.0.0.1:6821/12190] exists,up 3f023d1a-e170-4d1e-a7d2-b855746263e6 2020-11-16T20:05:08.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:05:08.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:05:08.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:05:08.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2020-11-16T20:05:08.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2020-11-16T20:05:08.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create foo 8 2020-11-16T20:05:09.432 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' already exists 2020-11-16T20:05:09.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:05:10.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2020-11-16T20:05:12.054 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 min_size to 1 2020-11-16T20:05:12.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2020-11-16T20:05:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 min_size to 2 2020-11-16T20:05:13.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2020-11-16T20:05:18.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2020-11-16T20:05:18.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2020-11-16T20:05:18.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:05:18.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:05:18.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:05:18.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:05:18.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2020-11-16T20:05:18.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:05:18.148 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:05:18.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:18.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:05:18.587 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/10122,v1:127.0.0.1:6803/10122] [v2:127.0.0.1:6804/10122,v1:127.0.0.1:6805/10122] exists,up ac81eb58-e98d-4575-b1af-d7e152179ec6 2020-11-16T20:05:18.587 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:05:18.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:05:18.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:05:18.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:05:18.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2020-11-16T20:05:18.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2020-11-16T20:05:18.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:05:18.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:05:18.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:05:18.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:05:18.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2020-11-16T20:05:18.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:05:18.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:05:18.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:18.968 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/11193,v1:127.0.0.1:6811/11193] [v2:127.0.0.1:6812/11193,v1:127.0.0.1:6813/11193] exists,up 8e5157c6-2e34-487a-b351-fe905960811a 2020-11-16T20:05:18.968 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:05:18.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:05:18.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:05:18.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:05:18.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2020-11-16T20:05:18.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2020-11-16T20:05:18.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:05:18.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:05:18.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:05:18.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:05:18.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2020-11-16T20:05:18.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:05:18.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:18.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:05:19.323 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/12190,v1:127.0.0.1:6819/12190] [v2:127.0.0.1:6820/12190,v1:127.0.0.1:6821/12190] exists,up 3f023d1a-e170-4d1e-a7d2-b855746263e6 2020-11-16T20:05:19.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:05:19.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:05:19.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:05:19.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2020-11-16T20:05:19.327 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:09.312+0000 7fa188b6f700 -1 osd.2 15 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.327 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:09.556+0000 7fa188b6f700 -1 osd.2 17 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.328 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:09.828+0000 7fa188b6f700 -1 osd.2 18 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.328 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:10.484+0000 7fa186b6b700 -1 osd.2 19 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.328 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:10.484+0000 7fa186b6b700 -1 osd.2 19 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.328 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:10.528+0000 7fa188b6f700 -1 osd.2 19 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.328 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:11.484+0000 7fa186b6b700 -1 osd.2 21 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.328 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:11.484+0000 7fa186b6b700 -1 osd.2 21 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.328 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:11.844+0000 7fa186b6b700 -1 osd.2 23 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.329 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:12.040+0000 7fa188b6f700 -1 osd.2 23 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.329 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:12.360+0000 7fa188b6f700 -1 osd.2 25 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.329 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:12.364+0000 7fa188b6f700 -1 osd.2 25 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.329 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:12.760+0000 7fa186b6b700 -1 osd.2 26 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.329 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:13.192+0000 7fa186b6b700 -1 osd.2 27 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.329 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:13.408+0000 7fa188b6f700 -1 osd.2 28 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.329 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:13.408+0000 7fa188b6f700 -1 osd.2 28 handle_osd_map injecting map crc failure 2020-11-16T20:05:19.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2020-11-16T20:05:19.332 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:09.312+0000 7fa188b6f700 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2020-11-16T20:05:19.333 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:09.560+0000 7fa188b6f700 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2020-11-16T20:05:19.333 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:09.828+0000 7fa188b6f700 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2020-11-16T20:05:19.333 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:10.484+0000 7fa186b6b700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2020-11-16T20:05:19.333 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:10.484+0000 7fa186b6b700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2020-11-16T20:05:19.333 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:10.528+0000 7fa188b6f700 10 osd.2 19 handle_osd_map bailing because last < start (20<21) 2020-11-16T20:05:19.333 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:11.484+0000 7fa186b6b700 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2020-11-16T20:05:19.334 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:11.484+0000 7fa186b6b700 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2020-11-16T20:05:19.334 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:11.844+0000 7fa186b6b700 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2020-11-16T20:05:19.334 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:12.040+0000 7fa188b6f700 10 osd.2 23 handle_osd_map bailing because last < start (24<25) 2020-11-16T20:05:19.334 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:12.364+0000 7fa188b6f700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2020-11-16T20:05:19.334 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:12.364+0000 7fa188b6f700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2020-11-16T20:05:19.334 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:12.760+0000 7fa186b6b700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2020-11-16T20:05:19.334 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:13.192+0000 7fa186b6b700 10 osd.2 27 handle_osd_map bailing because last < start (27<28) 2020-11-16T20:05:19.335 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:13.408+0000 7fa188b6f700 10 osd.2 28 handle_osd_map bailing because last < start (28<29) 2020-11-16T20:05:19.335 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T20:05:13.408+0000 7fa188b6f700 10 osd.2 28 handle_osd_map bailing because last < start (28<29) 2020-11-16T20:05:19.335 INFO:tasks.workunit.client.0.smithi189.stdout:success 2020-11-16T20:05:19.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2020-11-16T20:05:19.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2020-11-16T20:05:19.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=foo 2020-11-16T20:05:19.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2020-11-16T20:05:19.796 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' does not exist 2020-11-16T20:05:19.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2020-11-16T20:05:19.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:05:19.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:05:19.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:05:19.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:05:19.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:05:19.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:05:19.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2020-11-16T20:05:19.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2020-11-16T20:05:19.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:05:19.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2020-11-16T20:05:19.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:05:19.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:05:19.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:05:19.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:05:19.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:05:19.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:05:19.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:05:19.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:05:19.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:05:19.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:05:19.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:05:19.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:05:19.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:05:19.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:05:19.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:05:19.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:05:19.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:05:19.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:05:19.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2020-11-16T20:05:19.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:05:19.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:19.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.9317 2020-11-16T20:05:19.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.9317 2020-11-16T20:05:19.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:05:19.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:05:19.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T20:05:19.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/bad-inc-map 0 2020-11-16T20:05:19.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2020-11-16T20:05:19.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T20:05:19.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2020-11-16T20:05:19.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:05:19.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:05:19.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:05:19.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:05:19.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:05:19.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:05:19.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:05:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:05:19.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:05:19.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:05:19.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:05:19.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:05:19.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:05:19.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:05:19.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:05:19.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:05:19.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:05:19.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T20:05:19.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2020-11-16T20:05:19.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:05:19.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:19.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.9317 2020-11-16T20:05:19.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.9317 2020-11-16T20:05:19.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:05:19.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:05:19.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T20:05:19.964 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T20:05:19.964 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T20:05:20.020 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2020-11-16T20:05:20.020 INFO:teuthology.orchestra.run.smithi189:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2020-11-16T20:05:20.080 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T20:05:20.096 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T20:05:20.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:05:20.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:05:20.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T20:05:20.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T20:05:20.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T20:05:20.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T20:05:20.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T20:05:20.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T20:05:20.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/divergent-priors 2020-11-16T20:05:20.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2020-11-16T20:05:20.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2020-11-16T20:05:20.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2020-11-16T20:05:20.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2020-11-16T20:05:20.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2020-11-16T20:05:20.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2020-11-16T20:05:20.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2020-11-16T20:05:20.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2020-11-16T20:05:20.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2020-11-16T20:05:20.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2020-11-16T20:05:20.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2020-11-16T20:05:20.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2020-11-16T20:05:20.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2020-11-16T20:05:20.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2020-11-16T20:05:20.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2020-11-16T20:05:20.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2020-11-16T20:05:20.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2020-11-16T20:05:20.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2020-11-16T20:05:20.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2020-11-16T20:05:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2020-11-16T20:05:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2020-11-16T20:05:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2020-11-16T20:05:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2020-11-16T20:05:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2020-11-16T20:05:20.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2020-11-16T20:05:20.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2020-11-16T20:05:20.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none ' 2020-11-16T20:05:20.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2020-11-16T20:05:20.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2020-11-16T20:05:20.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2020-11-16T20:05:20.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:05:20.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2020-11-16T20:05:20.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T20:05:20.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2020-11-16T20:05:20.108 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_divergent_2 2020-11-16T20:05:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_divergent_3 2020-11-16T20:05:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_divergent_ec' 2020-11-16T20:05:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-11-16T20:05:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-11-16T20:05:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-11-16T20:05:20.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-11-16T20:05:20.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-11-16T20:05:20.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:05:20.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-11-16T20:05:20.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:05:20.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:05:20.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:05:20.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:05:20.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:05:20.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:05:20.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:05:20.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:05:20.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:05:20.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:05:20.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:05:20.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:05:20.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:05:20.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:05:20.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:05:20.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:05:20.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:05:20.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:05:20.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-11-16T20:05:20.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:05:20.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:20.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:05:20.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12850 2020-11-16T20:05:20.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:05:20.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:05:20.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-11-16T20:05:20.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:05:20.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:20.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:05:20.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12850 2020-11-16T20:05:20.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:05:20.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T20:05:20.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T20:05:20.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:05:20.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-11-16T20:05:20.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2020-11-16T20:05:20.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2020-11-16T20:05:20.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2020-11-16T20:05:20.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2020-11-16T20:05:20.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2020-11-16T20:05:20.136 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2020-11-16T20:05:20.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2020-11-16T20:05:20.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2020-11-16T20:05:20.150 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:05:20.150 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:05:20.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2020-11-16T20:05:20.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-11-16T20:05:20.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:05:20.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:05:20.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:05:20.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-11-16T20:05:20.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-11-16T20:05:20.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:05:20.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:05:20.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:05:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:05:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:05:20.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:05:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:05:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:05:20.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:05:20.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:05:20.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:05:20.679 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:05:20.679 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:05:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:05:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:05:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:05:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:05:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:05:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-mon.a.asok 2020-11-16T20:05:20.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:05:20.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.12850/ceph-mon.a.asok config get fsid 2020-11-16T20:05:20.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:05:20.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:05:20.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:05:20.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:05:20.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:05:20.791 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:05:20.792 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:05:20.792 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:05:20.792 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:05:20.792 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:20.792 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:05:20.792 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-mon.a.asok 2020-11-16T20:05:20.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:05:20.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.12850/ceph-mon.a.asok config get mon_host 2020-11-16T20:05:20.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2020-11-16T20:05:20.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2020-11-16T20:05:20.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:05:20.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:05:20.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:05:20.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2020-11-16T20:05:20.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:05:21.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:05:21.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:05:21.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:05:21.467 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:05:21.467 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:21.467 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:05:21.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:05:21.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:05:21.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:05:21.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-11-16T20:05:21.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2020-11-16T20:05:21.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:05:21.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:05:21.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:05:21.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:05:21.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2020-11-16T20:05:21.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:05:21.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:05:21.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:05:21.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:05:21.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-11-16T20:05:21.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-11-16T20:05:21.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:05:21.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:05:21.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:05:21.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:05:21.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:05:21.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:05:21.503 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:05:21.503 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:21.504 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:05:21.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:05:21.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:05:21.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:05:21.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:05:21.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:05:21.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:05:21.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:05:21.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:05:21.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:05:21.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:05:21.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:05:21.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:05:21.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2020-11-16T20:05:21.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:05:21.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4abbfb84-75bc-442b-83c0-1cfd2d77e8da 2020-11-16T20:05:21.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 4abbfb84-75bc-442b-83c0-1cfd2d77e8da' 2020-11-16T20:05:21.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:05:21.507 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 4abbfb84-75bc-442b-83c0-1cfd2d77e8da 2020-11-16T20:05:21.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCB27JfOOVYHxAAzUXZpLj8rBpB+iheeYLq+Q== 2020-11-16T20:05:21.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCB27JfOOVYHxAAzUXZpLj8rBpB+iheeYLq+Q=="}' 2020-11-16T20:05:21.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4abbfb84-75bc-442b-83c0-1cfd2d77e8da -i td/divergent-priors/0/new.json 2020-11-16T20:05:21.891 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:05:21.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2020-11-16T20:05:21.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCB27JfOOVYHxAAzUXZpLj8rBpB+iheeYLq+Q== --osd-uuid 4abbfb84-75bc-442b-83c0-1cfd2d77e8da 2020-11-16T20:05:21.947 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:21.944+0000 7f8536f54d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:21.950 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:21.944+0000 7f8536f54d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:21.951 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:21.948+0000 7f8536f54d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:21.952 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:21.948+0000 7f8536f54d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:05:21.952 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:21.948+0000 7f8536f54d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:05:21.952 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:21.948+0000 7f8536f54d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:05:21.995 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:21.992+0000 7f8536f54d80 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-11-16T20:05:26.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-11-16T20:05:26.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:05:26.385 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:05:26.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:05:26.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:05:26.893 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:05:26.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:05:26.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:05:26.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:05:26.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:05:26.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:05:26.929 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:26.924+0000 7f05204aed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:26.931 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:26.928+0000 7f05204aed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:26.932 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:26.928+0000 7f05204aed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:27.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:05:27.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:05:27.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:05:27.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:05:27.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:05:27.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:05:27.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:05:27.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:27.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:05:27.295 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:05:27.464 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:27.460+0000 7f05204aed80 -1 Falling back to public interface 2020-11-16T20:05:27.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:05:28.292 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:28.280+0000 7f05204aed80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:05:28.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:05:28.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:05:28.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:05:28.679 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:05:28.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:28.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:05:29.065 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/13710,v1:127.0.0.1:6803/13710] [v2:127.0.0.1:6804/13710,v1:127.0.0.1:6805/13710] exists,up 4abbfb84-75bc-442b-83c0-1cfd2d77e8da 2020-11-16T20:05:29.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:05:29.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:05:29.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:05:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-11-16T20:05:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2020-11-16T20:05:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:05:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:05:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:05:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:05:29.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:05:29.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:05:29.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:05:29.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:05:29.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:05:29.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:05:29.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:05:29.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:05:29.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:05:29.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:05:29.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:05:29.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:05:29.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:05:29.070 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:05:29.070 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:29.070 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:05:29.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:05:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:05:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:05:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:05:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:05:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:05:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:05:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:05:29.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:05:29.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:05:29.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:05:29.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:05:29.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:05:29.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:05:29.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=db96d31f-edcb-4682-9cca-4d4871efc940 2020-11-16T20:05:29.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 db96d31f-edcb-4682-9cca-4d4871efc940' 2020-11-16T20:05:29.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:05:29.073 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 db96d31f-edcb-4682-9cca-4d4871efc940 2020-11-16T20:05:29.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCJ27Jffg5sBRAAQeNvFFoOVOxrTQprQClfcA== 2020-11-16T20:05:29.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCJ27Jffg5sBRAAQeNvFFoOVOxrTQprQClfcA=="}' 2020-11-16T20:05:29.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new db96d31f-edcb-4682-9cca-4d4871efc940 -i td/divergent-priors/1/new.json 2020-11-16T20:05:29.618 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:05:29.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2020-11-16T20:05:29.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCJ27Jffg5sBRAAQeNvFFoOVOxrTQprQClfcA== --osd-uuid db96d31f-edcb-4682-9cca-4d4871efc940 2020-11-16T20:05:29.673 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:29.668+0000 7fc367361d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:29.676 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:29.672+0000 7fc367361d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:29.677 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:29.672+0000 7fc367361d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:29.677 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:29.672+0000 7fc367361d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:05:29.678 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:29.672+0000 7fc367361d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:05:29.678 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:29.672+0000 7fc367361d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:05:29.726 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:29.720+0000 7fc367361d80 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-11-16T20:05:34.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-11-16T20:05:34.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:05:34.347 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:05:34.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:05:34.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:05:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:05:34.848 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:05:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:05:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:05:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:05:34.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:05:34.882 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:34.876+0000 7f8863f3dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:34.884 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:34.880+0000 7f8863f3dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:34.885 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:34.880+0000 7f8863f3dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:35.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:05:35.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:05:35.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:05:35.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:05:35.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:05:35.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:05:35.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:05:35.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:35.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:05:35.277 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:05:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:35.432+0000 7f8863f3dd80 -1 Falling back to public interface 2020-11-16T20:05:35.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:05:36.292 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:36.288+0000 7f8863f3dd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:05:36.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:05:36.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:05:36.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:05:36.674 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:05:36.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:36.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:05:37.169 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/14704,v1:127.0.0.1:6811/14704] [v2:127.0.0.1:6812/14704,v1:127.0.0.1:6813/14704] exists,up db96d31f-edcb-4682-9cca-4d4871efc940 2020-11-16T20:05:37.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:05:37.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:05:37.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:05:37.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-11-16T20:05:37.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2020-11-16T20:05:37.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:05:37.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:05:37.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:05:37.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:05:37.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2020-11-16T20:05:37.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:05:37.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:05:37.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:05:37.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:05:37.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-11-16T20:05:37.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-11-16T20:05:37.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:05:37.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:05:37.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:05:37.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:05:37.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:05:37.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:05:37.174 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:05:37.174 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:05:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:05:37.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:05:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:05:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:05:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:05:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:05:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:05:37.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:05:37.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:05:37.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:05:37.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:05:37.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:05:37.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:05:37.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2020-11-16T20:05:37.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:05:37.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=abc43c59-78f4-4d31-87b4-2eb042089e2d 2020-11-16T20:05:37.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 abc43c59-78f4-4d31-87b4-2eb042089e2d' 2020-11-16T20:05:37.178 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 abc43c59-78f4-4d31-87b4-2eb042089e2d 2020-11-16T20:05:37.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:05:37.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCR27JfU5oqDBAA7NzpC4PXMNCeeaDwz7965g== 2020-11-16T20:05:37.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCR27JfU5oqDBAA7NzpC4PXMNCeeaDwz7965g=="}' 2020-11-16T20:05:37.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new abc43c59-78f4-4d31-87b4-2eb042089e2d -i td/divergent-priors/2/new.json 2020-11-16T20:05:37.720 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:05:37.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2020-11-16T20:05:37.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCR27JfU5oqDBAA7NzpC4PXMNCeeaDwz7965g== --osd-uuid abc43c59-78f4-4d31-87b4-2eb042089e2d 2020-11-16T20:05:37.771 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:37.768+0000 7ff6630d5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:37.774 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:37.768+0000 7ff6630d5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:37.775 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:37.772+0000 7ff6630d5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:37.776 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:37.772+0000 7ff6630d5d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:05:37.776 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:37.772+0000 7ff6630d5d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:05:37.776 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:37.772+0000 7ff6630d5d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:05:37.815 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:37.812+0000 7ff6630d5d80 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-11-16T20:05:42.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-11-16T20:05:42.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:05:42.677 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:05:42.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:05:42.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:05:43.169 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:05:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:05:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:05:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:05:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:05:43.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:05:43.205 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:43.200+0000 7f31fb637d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:43.208 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:43.204+0000 7f31fb637d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:43.210 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:43.204+0000 7f31fb637d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:05:43.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:05:43.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:05:43.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:05:43.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:05:43.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:05:43.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:05:43.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:05:43.580 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:05:43.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:43.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:05:43.764 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:43.760+0000 7f31fb637d80 -1 Falling back to public interface 2020-11-16T20:05:43.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:05:44.625 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:05:44.620+0000 7f31fb637d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:05:44.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:05:44.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:05:44.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:05:44.999 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:05:45.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:05:45.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:05:45.425 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/15698,v1:127.0.0.1:6819/15698] [v2:127.0.0.1:6820/15698,v1:127.0.0.1:6821/15698] exists,up abc43c59-78f4-4d31-87b4-2eb042089e2d 2020-11-16T20:05:45.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:05:45.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:05:45.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:05:45.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2020-11-16T20:05:46.145 INFO:tasks.workunit.client.0.smithi189.stderr:noout is set 2020-11-16T20:05:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2020-11-16T20:05:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:noin is set 2020-11-16T20:05:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2020-11-16T20:05:47.682 INFO:tasks.workunit.client.0.smithi189.stderr:nodown is set 2020-11-16T20:05:47.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2020-11-16T20:05:47.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T20:05:48.302 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T20:05:48.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:05:49.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2020-11-16T20:05:50.272 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 3 2020-11-16T20:05:50.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2020-11-16T20:05:50.991 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 min_size to 2 2020-11-16T20:05:51.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2020-11-16T20:05:51.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:05:51.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:05:51.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:05:51.417 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:05:51.417 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:05:51.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:05:51.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:05:51.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:05:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836488 2020-11-16T20:05:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836488 2020-11-16T20:05:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488' 2020-11-16T20:05:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:05:51.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:05:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2020-11-16T20:05:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2020-11-16T20:05:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2020-11-16T20:05:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:05:51.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:05:51.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2020-11-16T20:05:51.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2020-11-16T20:05:51.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672965 2-64424509444' 2020-11-16T20:05:51.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:05:51.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836488 2020-11-16T20:05:51.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:05:51.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:05:51.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2020-11-16T20:05:51.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:05:51.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836488 2020-11-16T20:05:51.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2020-11-16T20:05:51.981 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836488 2020-11-16T20:05:51.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:05:52.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836486 -lt 21474836488 2020-11-16T20:05:52.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:05:53.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:05:53.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:05:53.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836488 2020-11-16T20:05:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:05:53.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2020-11-16T20:05:53.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:05:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:05:53.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2020-11-16T20:05:53.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:05:53.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2020-11-16T20:05:53.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-11-16T20:05:53.792 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672965 2020-11-16T20:05:53.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:05:54.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672965 2020-11-16T20:05:54.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:05:54.173 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2020-11-16T20:05:54.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:05:54.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:05:54.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2020-11-16T20:05:54.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:05:54.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2020-11-16T20:05:54.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-11-16T20:05:54.179 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509444 2020-11-16T20:05:54.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:05:54.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2020-11-16T20:05:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2020-11-16T20:05:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:05:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:05:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:05:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:05:54.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:05:54.567 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:05:54.567 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:05:54.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:05:54.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:05:54.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:05:54.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:05:54.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:05:54.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:05:54.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:05:54.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:05:55.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:05:55.167 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:05:55.167 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:05:55.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:05:55.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:05:55.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:05:55.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2020-11-16T20:05:55.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2020-11-16T20:05:55.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2020-11-16T20:05:55.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:05:55.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:05:55.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2020-11-16T20:05:55.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2020-11-16T20:05:55.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672968' 2020-11-16T20:05:55.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:05:55.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:05:55.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509446 2020-11-16T20:05:55.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509446 2020-11-16T20:05:55.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672968 2-64424509446' 2020-11-16T20:05:55.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:05:55.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2020-11-16T20:05:55.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:05:55.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:05:55.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2020-11-16T20:05:55.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:05:55.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2020-11-16T20:05:55.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2020-11-16T20:05:55.685 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836491 2020-11-16T20:05:55.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:05:56.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836489 -lt 21474836491 2020-11-16T20:05:56.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:05:57.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:05:57.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:05:57.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836491 2020-11-16T20:05:57.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:05:57.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2020-11-16T20:05:57.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:05:57.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:05:57.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2020-11-16T20:05:57.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:05:57.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2020-11-16T20:05:57.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-11-16T20:05:57.497 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672968 2020-11-16T20:05:57.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:05:57.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2020-11-16T20:05:57.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:05:57.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:05:57.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2020-11-16T20:05:57.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:05:57.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2020-11-16T20:05:57.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:05:57.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509446 2020-11-16T20:05:57.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-11-16T20:05:57.878 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509446 2020-11-16T20:05:57.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:05:58.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509447 -lt 64424509446 2020-11-16T20:05:58.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:05:58.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:05:58.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:05:58.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:05:58.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:05:58.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:05:58.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:05:58.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:05:58.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:05:58.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:05:58.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:05:59.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:05:59.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:05:59.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:05:59.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:05:59.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:05:59.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:05:59.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:05:59.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2020-11-16T20:05:59.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2020-11-16T20:05:59.997 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:06:00.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2020-11-16T20:06:00.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2020-11-16T20:06:00.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2020-11-16T20:06:00.008 INFO:tasks.workunit.client.0.smithi189.stdout:primary and soon to be divergent is 1 2020-11-16T20:06:00.375 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:06:00.375 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:05:49.741443+0000 0'0 27:13 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:06:00.376 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:06:00.376 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:06:00.377 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:06:00.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2020-11-16T20:06:00.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-11-16T20:06:00.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2020-11-16T20:06:00.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2020-11-16T20:06:00.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-11-16T20:06:00.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2020-11-16T20:06:00.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2020-11-16T20:06:00.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-11-16T20:06:00.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2020-11-16T20:06:00.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2020-11-16T20:06:00.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2020-11-16T20:06:00.393 INFO:tasks.workunit.client.0.smithi189.stdout:writing initial objects 2020-11-16T20:06:00.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2020-11-16T20:06:00.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:00.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2020-11-16T20:06:00.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:00.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2020-11-16T20:06:00.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:00.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2020-11-16T20:06:00.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:00.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2020-11-16T20:06:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:00.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2020-11-16T20:06:01.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:01.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2020-11-16T20:06:01.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:01.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2020-11-16T20:06:01.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:01.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2020-11-16T20:06:01.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:01.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2020-11-16T20:06:01.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:01.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2020-11-16T20:06:01.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:01.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2020-11-16T20:06:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2020-11-16T20:06:02.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:02.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2020-11-16T20:06:02.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:02.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2020-11-16T20:06:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:02.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2020-11-16T20:06:02.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:02.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2020-11-16T20:06:02.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:02.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2020-11-16T20:06:02.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:02.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2020-11-16T20:06:03.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:03.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2020-11-16T20:06:03.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:03.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2020-11-16T20:06:03.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2020-11-16T20:06:03.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:03.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2020-11-16T20:06:03.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:03.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2020-11-16T20:06:03.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2020-11-16T20:06:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2020-11-16T20:06:04.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:04.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2020-11-16T20:06:04.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:04.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2020-11-16T20:06:04.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:04.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2020-11-16T20:06:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:04.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2020-11-16T20:06:04.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:04.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2020-11-16T20:06:04.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:04.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2020-11-16T20:06:04.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:04.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2020-11-16T20:06:05.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:05.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2020-11-16T20:06:05.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:05.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2020-11-16T20:06:05.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2020-11-16T20:06:05.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:05.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2020-11-16T20:06:05.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:05.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2020-11-16T20:06:05.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:05.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2020-11-16T20:06:06.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:06.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2020-11-16T20:06:06.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:06.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2020-11-16T20:06:06.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:06.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2020-11-16T20:06:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2020-11-16T20:06:06.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:06.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2020-11-16T20:06:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2020-11-16T20:06:06.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:06.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2020-11-16T20:06:07.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:07.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2020-11-16T20:06:07.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:07.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2020-11-16T20:06:07.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:07.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2020-11-16T20:06:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2020-11-16T20:06:07.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:07.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2020-11-16T20:06:07.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:07.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2020-11-16T20:06:07.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:07.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2020-11-16T20:06:08.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:08.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2020-11-16T20:06:08.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:08.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2020-11-16T20:06:08.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:08.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2020-11-16T20:06:08.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:08.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2020-11-16T20:06:08.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:08.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2020-11-16T20:06:08.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:08.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2020-11-16T20:06:08.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:08.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2020-11-16T20:06:09.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:09.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2020-11-16T20:06:09.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:09.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2020-11-16T20:06:09.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:09.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2020-11-16T20:06:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2020-11-16T20:06:09.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:09.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2020-11-16T20:06:09.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:09.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2020-11-16T20:06:09.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:09.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2020-11-16T20:06:10.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:10.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2020-11-16T20:06:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2020-11-16T20:06:10.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:10.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2020-11-16T20:06:10.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:10.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2020-11-16T20:06:10.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:10.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2020-11-16T20:06:10.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:10.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2020-11-16T20:06:10.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:10.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2020-11-16T20:06:11.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:11.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2020-11-16T20:06:11.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:11.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2020-11-16T20:06:11.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:11.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2020-11-16T20:06:11.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:11.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2020-11-16T20:06:11.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:11.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2020-11-16T20:06:11.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:11.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2020-11-16T20:06:12.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:12.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2020-11-16T20:06:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:12.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2020-11-16T20:06:12.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:12.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2020-11-16T20:06:12.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:12.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2020-11-16T20:06:12.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:12.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2020-11-16T20:06:12.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:12.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2020-11-16T20:06:12.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:12.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2020-11-16T20:06:13.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:13.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2020-11-16T20:06:13.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:13.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2020-11-16T20:06:13.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:13.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2020-11-16T20:06:13.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:13.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2020-11-16T20:06:13.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:13.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2020-11-16T20:06:13.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:13.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2020-11-16T20:06:14.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:14.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2020-11-16T20:06:14.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:14.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2020-11-16T20:06:14.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:14.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2020-11-16T20:06:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:14.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2020-11-16T20:06:14.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:14.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2020-11-16T20:06:14.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:14.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2020-11-16T20:06:14.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:14.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2020-11-16T20:06:15.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-11-16T20:06:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-11-16T20:06:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-11-16T20:06:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2020-11-16T20:06:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:06:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:06:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:06:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:06:15.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2020-11-16T20:06:15.200 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:06:15.200 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:06:15.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:06:15.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:06:15.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:06:15.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:06:15.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:06:15.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:06:15.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:06:15.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:06:15.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:06:15.727 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:06:15.727 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:06:15.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:06:15.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:06:15.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:06:15.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836497 2020-11-16T20:06:15.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836497 2020-11-16T20:06:15.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497' 2020-11-16T20:06:15.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:06:15.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:06:16.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2020-11-16T20:06:16.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2020-11-16T20:06:16.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672974' 2020-11-16T20:06:16.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:06:16.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:06:16.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509453 2020-11-16T20:06:16.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509453 2020-11-16T20:06:16.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672974 2-64424509453' 2020-11-16T20:06:16.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:06:16.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836497 2020-11-16T20:06:16.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:06:16.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:06:16.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836497 2020-11-16T20:06:16.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:06:16.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836497 2020-11-16T20:06:16.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2020-11-16T20:06:16.290 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836497 2020-11-16T20:06:16.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:06:16.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836495 -lt 21474836497 2020-11-16T20:06:16.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:06:17.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:06:17.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:06:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836497 -lt 21474836497 2020-11-16T20:06:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:06:18.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2020-11-16T20:06:18.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:06:18.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:06:18.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2020-11-16T20:06:18.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:06:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2020-11-16T20:06:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2020-11-16T20:06:18.121 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672974 2020-11-16T20:06:18.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:06:18.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2020-11-16T20:06:18.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:06:18.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509453 2020-11-16T20:06:18.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:06:18.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:06:18.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509453 2020-11-16T20:06:18.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:06:18.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509453 2020-11-16T20:06:18.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2020-11-16T20:06:18.528 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509453 2020-11-16T20:06:18.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:06:18.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509453 -lt 64424509453 2020-11-16T20:06:18.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:06:18.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:06:18.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:06:19.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:06:19.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:06:19.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:06:19.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:06:19.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:06:19.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:06:19.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:06:19.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:06:19.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:06:19.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:06:19.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:06:19.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:06:20.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:06:20.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:06:20.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:06:20.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2020-11-16T20:06:20.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T20:06:20.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2020-11-16T20:06:20.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2020-11-16T20:06:20.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T20:06:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2020-11-16T20:06:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2020-11-16T20:06:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2020-11-16T20:06:20.698 INFO:tasks.workunit.client.0.smithi189.stdout:blackholing osds 0 2 2020-11-16T20:06:21.067 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:06:21.068 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2020-11-16T20:05:49.741443+0000 28'100 28:113 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:06:21.068 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:06:21.068 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:06:21.070 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:06:21.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-11-16T20:06:21.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2020-11-16T20:06:21.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T20:06:21.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T20:06:21.089 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:06:21.089 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:06:21.089 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:06:21.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.0.asok 2020-11-16T20:06:21.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-11-16T20:06:21.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.0.asok config set objectstore_blackhole 1 2020-11-16T20:06:21.180 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:06:21.180 INFO:tasks.workunit.client.0.smithi189.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-11-16T20:06:21.180 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:06:21.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-11-16T20:06:21.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2020-11-16T20:06:21.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-11-16T20:06:21.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-11-16T20:06:21.191 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:06:21.191 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:06:21.191 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:06:21.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.2.asok 2020-11-16T20:06:21.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-11-16T20:06:21.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.2.asok config set objectstore_blackhole 1 2020-11-16T20:06:21.294 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:06:21.294 INFO:tasks.workunit.client.0.smithi189.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-11-16T20:06:21.294 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:06:21.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2020-11-16T20:06:21.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2020-11-16T20:06:21.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2020-11-16T20:06:21.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2020-11-16T20:06:21.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2020-11-16T20:06:21.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2020-11-16T20:06:21.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2020-11-16T20:06:21.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-11-16T20:06:21.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-11-16T20:06:21.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-11-16T20:06:21.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=test 2020-11-16T20:06:21.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-11-16T20:06:21.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=existing_99 2020-11-16T20:06:21.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-11-16T20:06:21.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/divergent-priors 2020-11-16T20:06:21.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-11-16T20:06:21.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2020-11-16T20:06:21.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-11-16T20:06:21.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-11-16T20:06:21.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-11-16T20:06:21.310 INFO:tasks.workunit.client.0.smithi189.stdout:writing divergent object 2020-11-16T20:06:21.311 INFO:tasks.workunit.client.0.smithi189.stdout:create missing divergent object 2020-11-16T20:06:21.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds test existing_99 2020-11-16T20:06:21.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test 2020-11-16T20:06:21.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=existing_99 2020-11-16T20:06:21.312 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-11-16T20:06:21.312 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test existing_99 2020-11-16T20:06:21.708 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=1 2020-11-16T20:06:21.708 INFO:tasks.workunit.client.0.smithi189.stderr:0 2020-11-16T20:06:21.708 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:06:21.708 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 1 0 2 2020-11-16T20:06:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-11-16T20:06:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2020-11-16T20:06:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2020-11-16T20:06:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2020-11-16T20:06:21.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/divergent-priors/1/type 2020-11-16T20:06:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-11-16T20:06:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-11-16T20:06:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-11-16T20:06:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2020-11-16T20:06:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-11-16T20:06:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-11-16T20:06:21.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-11-16T20:06:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2020-11-16T20:06:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:06:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:06:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:06:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:06:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:06:21.715 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.1.asok 2020-11-16T20:06:21.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.12850/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-11-16T20:06:21.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-11-16T20:06:21.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-11-16T20:06:21.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-11-16T20:06:21.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2020-11-16T20:06:21.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:06:21.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:06:21.819 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:06:21.819 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:06:21.819 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:06:21.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.1.asok 2020-11-16T20:06:21.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-11-16T20:06:21.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.1.asok injectdataerr test existing_99 2020-11-16T20:06:21.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2020-11-16T20:06:21.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2020-11-16T20:06:31.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2020-11-16T20:06:31.953 INFO:tasks.workunit.client.0.smithi189.stderr:rados: no process found 2020-11-16T20:06:31.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2020-11-16T20:06:31.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2020-11-16T20:06:31.954 INFO:tasks.workunit.client.0.smithi189.stdout:killing all the osds 2020-11-16T20:06:32.337 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:06:32.338 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 active+clean 2020-11-16T20:06:22.283415+0000 28'101 28:122 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:06:32.338 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:06:32.338 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:06:32.340 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:06:32.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2020-11-16T20:06:32.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:06:32.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:06:32.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:06:32.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:06:32.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:06:32.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:06:32.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-11-16T20:06:32.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2020-11-16T20:06:33.015 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already down. 2020-11-16T20:06:33.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-11-16T20:06:33.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2020-11-16T20:06:33.637 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already down. 2020-11-16T20:06:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-11-16T20:06:33.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2020-11-16T20:06:34.213 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already down. 2020-11-16T20:06:34.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-11-16T20:06:34.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2020-11-16T20:06:34.783 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T20:06:34.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-11-16T20:06:34.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2020-11-16T20:06:35.349 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already out. 2020-11-16T20:06:35.359 INFO:tasks.workunit.client.0.smithi189.stdout:bringing up non_divergent 0 2 2020-11-16T20:06:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2020-11-16T20:06:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2020-11-16T20:06:35.734 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:06:35.734 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 stale+active+clean 2020-11-16T20:06:22.283415+0000 28'101 28:122 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:06:35.735 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:06:35.735 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:06:35.737 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:06:35.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-11-16T20:06:35.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2020-11-16T20:06:35.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:06:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:06:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T20:06:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:06:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2020-11-16T20:06:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:06:35.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:06:35.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:06:35.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:06:35.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-11-16T20:06:35.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-11-16T20:06:35.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:06:35.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:06:35.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:06:35.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:06:35.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:06:35.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:06:35.751 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:06:35.751 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:06:35.751 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:06:35.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:06:35.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:06:35.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:06:35.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:06:35.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:06:35.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:06:35.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:06:35.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:06:35.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:06:35.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:06:35.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2020-11-16T20:06:35.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T20:06:35.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:06:35.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2020-11-16T20:06:35.754 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:06:35.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T20:06:35.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:06:35.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:06:35.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:06:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:06:35.780+0000 7f769f196d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:06:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:06:35.784+0000 7f769f196d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:06:35.788 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:06:35.784+0000 7f769f196d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:06:36.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T20:06:36.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:06:36.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:06:36.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:06:36.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:06:36.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:06:36.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:06:36.144 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:06:36.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:06:36.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:06:36.332 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:06:36.328+0000 7f769f196d80 -1 Falling back to public interface 2020-11-16T20:06:36.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:06:37.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:06:37.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:06:37.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:06:37.541 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:06:37.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:06:37.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:06:37.669 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:06:37.668+0000 7f769f196d80 -1 osd.0 28 log_to_monitors {default=true} 2020-11-16T20:06:37.964 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/19663,v1:127.0.0.1:6803/19663] [v2:127.0.0.1:6804/19663,v1:127.0.0.1:6805/19663] exists,up 4abbfb84-75bc-442b-83c0-1cfd2d77e8da 2020-11-16T20:06:37.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:06:37.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:06:37.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:06:37.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-11-16T20:06:37.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2020-11-16T20:06:37.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:06:37.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:06:37.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-11-16T20:06:37.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:06:37.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2020-11-16T20:06:37.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:06:37.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:06:37.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:06:37.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:06:37.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-11-16T20:06:37.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-11-16T20:06:37.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:06:37.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:06:37.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:06:37.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:06:37.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:06:37.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:06:37.972 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:06:37.972 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:06:37.972 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:06:37.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:06:37.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:06:37.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:06:37.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:06:37.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:06:37.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:06:37.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:06:37.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:06:37.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:06:37.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:06:37.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2020-11-16T20:06:37.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-11-16T20:06:37.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:06:37.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2020-11-16T20:06:37.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-11-16T20:06:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:06:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:06:37.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:06:37.978 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:06:37.998 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:06:37.996+0000 7fc70852cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:06:38.010 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:06:38.008+0000 7fc70852cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:06:38.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:06:38.012+0000 7fc70852cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:06:38.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-11-16T20:06:38.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:06:38.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:06:38.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:06:38.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:06:38.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:06:38.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:06:38.353 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:06:38.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:06:38.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:06:38.559 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:06:38.556+0000 7fc70852cd80 -1 Falling back to public interface 2020-11-16T20:06:38.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:06:39.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:06:39.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:06:39.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:06:39.730 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:06:39.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:06:39.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:06:39.986 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:06:39.984+0000 7fc70852cd80 -1 osd.2 28 log_to_monitors {default=true} 2020-11-16T20:06:40.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:06:41.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:06:41.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:06:41.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:06:41.149 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:06:41.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:06:41.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:06:41.579 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/20062,v1:127.0.0.1:6811/20062] [v2:127.0.0.1:6812/20062,v1:127.0.0.1:6813/20062] exists,up abc43c59-78f4-4d31-87b4-2eb042089e2d 2020-11-16T20:06:41.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:06:41.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:06:41.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:06:41.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-11-16T20:06:41.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2020-11-16T20:06:42.139 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already in. 2020-11-16T20:06:42.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-11-16T20:06:42.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2020-11-16T20:06:43.063 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already in. 2020-11-16T20:06:43.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-11-16T20:06:43.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2020-11-16T20:06:43.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:06:43.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:06:43.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:06:43.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:06:43.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2020-11-16T20:06:43.080 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:06:43.080 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:06:43.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:06:43.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:06:43.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:06:43.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:06:43.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:06:43.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:06:43.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:06:43.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:06:43.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:06:43.740 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:06:43.740 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:06:43.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:06:43.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:06:43.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:06:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=146028888068 2020-11-16T20:06:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 146028888068 2020-11-16T20:06:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-146028888068' 2020-11-16T20:06:43.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:06:43.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:06:44.045 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-11-16T20:06:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T20:06:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T20:06:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T20:06:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:06:44.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:06:44.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855363 2020-11-16T20:06:44.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855363 2020-11-16T20:06:44.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-146028888068 2-150323855363' 2020-11-16T20:06:44.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:06:44.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-146028888068 2020-11-16T20:06:44.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:06:44.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:06:44.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-146028888068 2020-11-16T20:06:44.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:06:44.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=146028888068 2020-11-16T20:06:44.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 146028888068' 2020-11-16T20:06:44.219 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 146028888068 2020-11-16T20:06:44.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:06:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 146028888066 -lt 146028888068 2020-11-16T20:06:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:06:45.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:06:45.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:06:46.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 146028888068 -lt 146028888068 2020-11-16T20:06:50.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:06:50.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-150323855363 2020-11-16T20:06:50.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:06:50.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:06:50.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-150323855363 2020-11-16T20:06:50.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:06:50.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855363 2020-11-16T20:06:50.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-11-16T20:06:50.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:06:50.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855364 -lt 150323855363 2020-11-16T20:06:50.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:06:50.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:06:50.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:06:50.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:06:50.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:06:50.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:06:50.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:06:50.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:06:50.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:06:50.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:06:50.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:06:50.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:06:50.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:06:50.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:06:50.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:06:50.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:06:50.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:06:50.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:06:50.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:06:50.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:06:50.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:06:50.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:06:50.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:06:50.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:06:50.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:06:50.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:06:50.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:06:50.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:06:50.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:06:50.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:06:50.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:06:50.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:06:50.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:06:50.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:06:50.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:06:50.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:06:50.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:06:50.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:06:50.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:06:50.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:06:50.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:06:50.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:06:50.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:06:50.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:06:50.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2020-11-16T20:06:50.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:06:50.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-11-16T20:06:50.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:06:50.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:06:50.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:06:50.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:06:50.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:06:50.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:06:50.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:06:50.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:06:50.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:06:50.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:06:50.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:06:50.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:06:50.120 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 150323855363 2020-11-16T20:06:50.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:06:50.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:06:50.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:06:50.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:06:50.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:06:50.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:06:50.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:06:50.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:06:50.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:06:50.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:06:50.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:06:50.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2020-11-16T20:06:50.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:06:50.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2020-11-16T20:06:51.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:06:51.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:06:51.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:06:51.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:06:51.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:06:51.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:06:51.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:06:51.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:06:51.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:06:51.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:06:51.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:06:51.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:06:52.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:06:52.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:06:52.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:06:52.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:06:52.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:06:52.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:06:52.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:06:52.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:06:52.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:06:52.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:06:52.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:06:52.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2020-11-16T20:06:52.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:06:52.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2020-11-16T20:06:53.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:06:53.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:06:53.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:06:53.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:06:53.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:06:53.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:06:53.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:06:53.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:06:53.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:06:53.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:06:53.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:06:53.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:06:54.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:06:54.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:06:54.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:06:54.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:06:54.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:06:54.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:06:54.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:06:54.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:06:54.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:06:54.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:06:54.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:06:54.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2020-11-16T20:06:54.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:06:54.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2020-11-16T20:06:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:06:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:06:56.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:06:56.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:06:56.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:06:56.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:06:56.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:06:56.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:06:56.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:06:56.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:06:56.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:06:56.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:06:57.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:06:57.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:06:57.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:06:57.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:06:57.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:06:57.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:06:57.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:06:57.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:06:57.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:06:57.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:06:57.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:06:57.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2020-11-16T20:06:57.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:06:57.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2020-11-16T20:07:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:07:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:07:00.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:07:00.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:07:00.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:07:00.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:07:00.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:07:00.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:07:01.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:07:01.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:07:01.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:07:01.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:07:01.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:07:01.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:07:01.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:07:01.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:07:01.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:07:01.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:07:01.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:07:01.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:07:01.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:07:02.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:07:02.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:07:02.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2020-11-16T20:07:02.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:07:02.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2020-11-16T20:07:08.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:07:08.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:07:08.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:07:08.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:07:08.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:07:08.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:07:08.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:07:08.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:07:09.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:07:09.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:07:09.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:07:09.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:07:09.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:07:09.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:07:09.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:07:09.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:07:09.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:07:09.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:07:09.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:07:09.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:07:09.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:07:10.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:07:10.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:07:10.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2020-11-16T20:07:10.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:07:10.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2020-11-16T20:07:17.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:07:17.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:07:17.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:07:17.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:07:17.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:07:17.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:07:17.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:07:17.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:07:17.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:07:17.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:07:17.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:07:17.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:07:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:07:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:07:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:07:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:07:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:07:22.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:07:22.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:07:22.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:07:22.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:07:22.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:07:22.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:07:22.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2020-11-16T20:07:22.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2020-11-16T20:07:22.553 INFO:tasks.workunit.client.0.smithi189.stderr:report 1293202165 2020-11-16T20:07:22.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2020-11-16T20:07:22.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2020-11-16T20:07:22.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2020-11-16T20:07:22.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2020-11-16T20:07:22.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2020-11-16T20:07:22.554 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:07:22.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2020-11-16T20:07:22.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2020-11-16T20:07:22.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2020-11-16T20:07:22.555 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:07:22.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-11-16T20:07:22.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2020-11-16T20:07:22.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T20:07:22.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T20:07:22.555 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:07:22.556 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:07:22.556 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:07:22.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.0.asok 2020-11-16T20:07:22.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-11-16T20:07:22.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.0.asok set_recovery_delay 100000 2020-11-16T20:07:22.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-11-16T20:07:22.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2020-11-16T20:07:22.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-11-16T20:07:22.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-11-16T20:07:22.557 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:07:22.557 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:07:22.557 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:07:22.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.2.asok 2020-11-16T20:07:22.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-11-16T20:07:22.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.2.asok set_recovery_delay 100000 2020-11-16T20:07:22.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2020-11-16T20:07:22.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2020-11-16T20:07:22.558 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:07:22.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2020-11-16T20:07:22.558 INFO:tasks.workunit.client.0.smithi189.stderr:noup is set 2020-11-16T20:07:22.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2020-11-16T20:07:22.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:07:22.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:07:22.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T20:07:22.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:07:22.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:07:22.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:07:22.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:07:22.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:07:22.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:07:22.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:07:22.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:07:22.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:07:22.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:07:22.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:07:22.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:07:22.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:07:22.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:07:22.561 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:07:22.561 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:07:22.561 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:07:22.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:07:22.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:07:22.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:07:22.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:07:22.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:07:22.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:07:22.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:07:22.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:07:22.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:07:22.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:07:22.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:07:22.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T20:07:22.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:07:22.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2020-11-16T20:07:22.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T20:07:22.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:07:22.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:07:22.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:07:22.564 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:07:21.532+0000 7fa1a3958d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:07:22.564 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:07:21.536+0000 7fa1a3958d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:07:22.564 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:07:21.536+0000 7fa1a3958d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:07:22.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2020-11-16T20:07:22.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2020-11-16T20:07:22.565 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:07:22.064+0000 7fa1a3958d80 -1 Falling back to public interface 2020-11-16T20:07:22.565 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:07:22.565 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_fingerprint": "787df021-3f45-499d-965d-322938694ca4", 2020-11-16T20:07:22.565 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "16.0.0-7219-g911cc357", 2020-11-16T20:07:22.566 INFO:tasks.workunit.client.0.smithi189.stdout: "commit": "911cc35794d871071a8bebfb0919482263b1d18b", 2020-11-16T20:07:22.566 INFO:tasks.workunit.client.0.smithi189.stdout: "timestamp": "2020-11-16T20:07:19.321900+0000", 2020-11-16T20:07:22.566 INFO:tasks.workunit.client.0.smithi189.stdout: "tag": "", 2020-11-16T20:07:22.566 INFO:tasks.workunit.client.0.smithi189.stdout: "health": { 2020-11-16T20:07:22.566 INFO:tasks.workunit.client.0.smithi189.stdout: "status": "HEALTH_WARN", 2020-11-16T20:07:22.566 INFO:tasks.workunit.client.0.smithi189.stdout: "checks": { 2020-11-16T20:07:22.566 INFO:tasks.workunit.client.0.smithi189.stdout: "OSDMAP_FLAGS": { 2020-11-16T20:07:22.567 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:07:22.567 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:07:22.567 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "nodown,noout,noin flag(s) set", 2020-11-16T20:07:22.567 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 17 2020-11-16T20:07:22.567 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.567 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [], 2020-11-16T20:07:22.567 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:07:22.568 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.568 INFO:tasks.workunit.client.0.smithi189.stdout: "OSD_DOWN": { 2020-11-16T20:07:22.568 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:07:22.568 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:07:22.568 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 osds down", 2020-11-16T20:07:22.568 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:07:22.568 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.569 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:07:22.569 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.569 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "osd.1 (root=default,host=smithi189) is down" 2020-11-16T20:07:22.569 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.569 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.569 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:07:22.569 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.570 INFO:tasks.workunit.client.0.smithi189.stdout: "PG_DEGRADED": { 2020-11-16T20:07:22.570 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:07:22.570 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:07:22.570 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-11-16T20:07:22.570 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:07:22.570 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.571 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:07:22.571 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.571 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:07:22.571 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.571 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.572 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:07:22.572 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.572 INFO:tasks.workunit.client.0.smithi189.stdout: "POOL_APP_NOT_ENABLED": { 2020-11-16T20:07:22.572 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:07:22.572 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:07:22.572 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 pool(s) do not have an application enabled", 2020-11-16T20:07:22.572 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:07:22.573 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.573 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:07:22.573 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.573 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "application not enabled on pool 'test'" 2020-11-16T20:07:22.573 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.574 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.574 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-11-16T20:07:22.574 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.574 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.574 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:07:22.574 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.574 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.575 INFO:tasks.workunit.client.0.smithi189.stdout: "mutes": [] 2020-11-16T20:07:22.575 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.575 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_first_committed": 1, 2020-11-16T20:07:22.575 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_last_committed": 1, 2020-11-16T20:07:22.575 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap": { 2020-11-16T20:07:22.575 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:07:22.575 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:07:22.576 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:05:20.180392Z", 2020-11-16T20:07:22.576 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:05:20.180392Z", 2020-11-16T20:07:22.576 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release": 16, 2020-11-16T20:07:22.576 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release_name": "pacific", 2020-11-16T20:07:22.576 INFO:tasks.workunit.client.0.smithi189.stdout: "election_strategy": 1, 2020-11-16T20:07:22.576 INFO:tasks.workunit.client.0.smithi189.stdout: "disallowed_leaders: ": "", 2020-11-16T20:07:22.576 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": false, 2020-11-16T20:07:22.577 INFO:tasks.workunit.client.0.smithi189.stdout: "features": { 2020-11-16T20:07:22.577 INFO:tasks.workunit.client.0.smithi189.stdout: "persistent": [ 2020-11-16T20:07:22.577 INFO:tasks.workunit.client.0.smithi189.stdout: "kraken", 2020-11-16T20:07:22.577 INFO:tasks.workunit.client.0.smithi189.stdout: "luminous", 2020-11-16T20:07:22.577 INFO:tasks.workunit.client.0.smithi189.stdout: "mimic", 2020-11-16T20:07:22.577 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap-prune", 2020-11-16T20:07:22.577 INFO:tasks.workunit.client.0.smithi189.stdout: "nautilus", 2020-11-16T20:07:22.578 INFO:tasks.workunit.client.0.smithi189.stdout: "octopus", 2020-11-16T20:07:22.578 INFO:tasks.workunit.client.0.smithi189.stdout: "pacific", 2020-11-16T20:07:22.578 INFO:tasks.workunit.client.0.smithi189.stdout: "elector-pinging" 2020-11-16T20:07:22.578 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.578 INFO:tasks.workunit.client.0.smithi189.stdout: "optional": [] 2020-11-16T20:07:22.578 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.578 INFO:tasks.workunit.client.0.smithi189.stdout: "mons": [ 2020-11-16T20:07:22.579 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.579 INFO:tasks.workunit.client.0.smithi189.stdout: "rank": 0, 2020-11-16T20:07:22.579 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "a", 2020-11-16T20:07:22.579 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:07:22.579 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.579 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.579 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.580 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115", 2020-11-16T20:07:22.580 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 0 2020-11-16T20:07:22.580 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.580 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.580 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.580 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115/0", 2020-11-16T20:07:22.580 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:7115/0", 2020-11-16T20:07:22.581 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 0, 2020-11-16T20:07:22.581 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 0, 2020-11-16T20:07:22.581 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_location": "{}" 2020-11-16T20:07:22.581 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.581 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.581 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.581 INFO:tasks.workunit.client.0.smithi189.stdout: "quorum": [ 2020-11-16T20:07:22.582 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2020-11-16T20:07:22.582 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.582 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap": { 2020-11-16T20:07:22.582 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 40, 2020-11-16T20:07:22.582 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:07:22.582 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:05:20.968409+0000", 2020-11-16T20:07:22.582 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:06:43.131774+0000", 2020-11-16T20:07:22.583 INFO:tasks.workunit.client.0.smithi189.stdout: "last_up_change": "2020-11-16T20:06:40.066455+0000", 2020-11-16T20:07:22.583 INFO:tasks.workunit.client.0.smithi189.stdout: "last_in_change": "2020-11-16T20:06:42.615907+0000", 2020-11-16T20:07:22.583 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-11-16T20:07:22.583 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_num": 5800384, 2020-11-16T20:07:22.583 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_set": [ 2020-11-16T20:07:22.583 INFO:tasks.workunit.client.0.smithi189.stdout: "nodown", 2020-11-16T20:07:22.583 INFO:tasks.workunit.client.0.smithi189.stdout: "noin", 2020-11-16T20:07:22.584 INFO:tasks.workunit.client.0.smithi189.stdout: "noout", 2020-11-16T20:07:22.584 INFO:tasks.workunit.client.0.smithi189.stdout: "pglog_hardlimit", 2020-11-16T20:07:22.584 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snapdirs", 2020-11-16T20:07:22.584 INFO:tasks.workunit.client.0.smithi189.stdout: "recovery_deletes", 2020-11-16T20:07:22.584 INFO:tasks.workunit.client.0.smithi189.stdout: "sortbitwise" 2020-11-16T20:07:22.584 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.584 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_version": 7, 2020-11-16T20:07:22.585 INFO:tasks.workunit.client.0.smithi189.stdout: "full_ratio": 0.99000000953674316, 2020-11-16T20:07:22.585 INFO:tasks.workunit.client.0.smithi189.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-11-16T20:07:22.585 INFO:tasks.workunit.client.0.smithi189.stdout: "nearfull_ratio": 0.85000002384185791, 2020-11-16T20:07:22.585 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_snapshot": "", 2020-11-16T20:07:22.585 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_max": 1, 2020-11-16T20:07:22.585 INFO:tasks.workunit.client.0.smithi189.stdout: "max_osd": 3, 2020-11-16T20:07:22.585 INFO:tasks.workunit.client.0.smithi189.stdout: "require_min_compat_client": "luminous", 2020-11-16T20:07:22.586 INFO:tasks.workunit.client.0.smithi189.stdout: "min_compat_client": "jewel", 2020-11-16T20:07:22.586 INFO:tasks.workunit.client.0.smithi189.stdout: "require_osd_release": "pacific", 2020-11-16T20:07:22.586 INFO:tasks.workunit.client.0.smithi189.stdout: "pools": [ 2020-11-16T20:07:22.586 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.586 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:07:22.586 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_name": "test", 2020-11-16T20:07:22.587 INFO:tasks.workunit.client.0.smithi189.stdout: "create_time": "2020-11-16T20:05:48.055814+0000", 2020-11-16T20:07:22.587 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": 1, 2020-11-16T20:07:22.587 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_names": "hashpspool", 2020-11-16T20:07:22.587 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:07:22.587 INFO:tasks.workunit.client.0.smithi189.stdout: "size": 3, 2020-11-16T20:07:22.587 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 2, 2020-11-16T20:07:22.587 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_rule": 0, 2020-11-16T20:07:22.588 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_count": 0, 2020-11-16T20:07:22.588 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_target": 0, 2020-11-16T20:07:22.588 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_barrier": 0, 2020-11-16T20:07:22.588 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2020-11-16T20:07:22.588 INFO:tasks.workunit.client.0.smithi189.stdout: "object_hash": 2, 2020-11-16T20:07:22.588 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_autoscale_mode": "off", 2020-11-16T20:07:22.588 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num": 1, 2020-11-16T20:07:22.589 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num": 1, 2020-11-16T20:07:22.589 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num_target": 1, 2020-11-16T20:07:22.589 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_target": 1, 2020-11-16T20:07:22.589 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_pending": 1, 2020-11-16T20:07:22.589 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pg_merge_meta": { 2020-11-16T20:07:22.589 INFO:tasks.workunit.client.0.smithi189.stdout: "source_pgid": "0.0", 2020-11-16T20:07:22.589 INFO:tasks.workunit.client.0.smithi189.stdout: "ready_epoch": 0, 2020-11-16T20:07:22.590 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_started": 0, 2020-11-16T20:07:22.590 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": 0, 2020-11-16T20:07:22.590 INFO:tasks.workunit.client.0.smithi189.stdout: "source_version": "0'0", 2020-11-16T20:07:22.590 INFO:tasks.workunit.client.0.smithi189.stdout: "target_version": "0'0" 2020-11-16T20:07:22.590 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.590 INFO:tasks.workunit.client.0.smithi189.stdout: "last_change": "28", 2020-11-16T20:07:22.590 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend": "0", 2020-11-16T20:07:22.591 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_prenautilus": "0", 2020-11-16T20:07:22.591 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_preluminous": "0", 2020-11-16T20:07:22.591 INFO:tasks.workunit.client.0.smithi189.stdout: "auid": 0, 2020-11-16T20:07:22.591 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_mode": "selfmanaged", 2020-11-16T20:07:22.591 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_seq": 0, 2020-11-16T20:07:22.591 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_epoch": 0, 2020-11-16T20:07:22.591 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_snaps": [], 2020-11-16T20:07:22.592 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps": "[]", 2020-11-16T20:07:22.592 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_bytes": 0, 2020-11-16T20:07:22.592 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_objects": 0, 2020-11-16T20:07:22.592 INFO:tasks.workunit.client.0.smithi189.stdout: "tiers": [], 2020-11-16T20:07:22.592 INFO:tasks.workunit.client.0.smithi189.stdout: "tier_of": -1, 2020-11-16T20:07:22.592 INFO:tasks.workunit.client.0.smithi189.stdout: "read_tier": -1, 2020-11-16T20:07:22.592 INFO:tasks.workunit.client.0.smithi189.stdout: "write_tier": -1, 2020-11-16T20:07:22.593 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_mode": "none", 2020-11-16T20:07:22.593 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_bytes": 0, 2020-11-16T20:07:22.593 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_objects": 0, 2020-11-16T20:07:22.593 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-11-16T20:07:22.593 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-11-16T20:07:22.593 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_full_ratio_micro": 800000, 2020-11-16T20:07:22.593 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_flush_age": 0, 2020-11-16T20:07:22.593 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_evict_age": 0, 2020-11-16T20:07:22.594 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profile": "", 2020-11-16T20:07:22.594 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_params": { 2020-11-16T20:07:22.594 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "none" 2020-11-16T20:07:22.594 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.594 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_period": 0, 2020-11-16T20:07:22.594 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_count": 0, 2020-11-16T20:07:22.594 INFO:tasks.workunit.client.0.smithi189.stdout: "use_gmt_hitset": true, 2020-11-16T20:07:22.595 INFO:tasks.workunit.client.0.smithi189.stdout: "min_read_recency_for_promote": 0, 2020-11-16T20:07:22.595 INFO:tasks.workunit.client.0.smithi189.stdout: "min_write_recency_for_promote": 0, 2020-11-16T20:07:22.595 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_grade_decay_rate": 0, 2020-11-16T20:07:22.595 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_search_last_n": 0, 2020-11-16T20:07:22.595 INFO:tasks.workunit.client.0.smithi189.stdout: "grade_table": [], 2020-11-16T20:07:22.595 INFO:tasks.workunit.client.0.smithi189.stdout: "stripe_width": 0, 2020-11-16T20:07:22.595 INFO:tasks.workunit.client.0.smithi189.stdout: "expected_num_objects": 0, 2020-11-16T20:07:22.596 INFO:tasks.workunit.client.0.smithi189.stdout: "fast_read": false, 2020-11-16T20:07:22.596 INFO:tasks.workunit.client.0.smithi189.stdout: "options": {}, 2020-11-16T20:07:22.596 INFO:tasks.workunit.client.0.smithi189.stdout: "application_metadata": {} 2020-11-16T20:07:22.596 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.596 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.596 INFO:tasks.workunit.client.0.smithi189.stdout: "osds": [ 2020-11-16T20:07:22.596 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.597 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:07:22.597 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "4abbfb84-75bc-442b-83c0-1cfd2d77e8da", 2020-11-16T20:07:22.597 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:07:22.597 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:07:22.597 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:07:22.597 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:07:22.597 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 5, 2020-11-16T20:07:22.598 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 28, 2020-11-16T20:07:22.598 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 34, 2020-11-16T20:07:22.598 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 38, 2020-11-16T20:07:22.598 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 29, 2020-11-16T20:07:22.598 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:07:22.598 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:07:22.598 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.599 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.599 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.599 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6802", 2020-11-16T20:07:22.599 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 19663 2020-11-16T20:07:22.599 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.599 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.599 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.600 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6803", 2020-11-16T20:07:22.600 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 19663 2020-11-16T20:07:22.600 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.600 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.600 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.600 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:07:22.600 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.601 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.601 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.601 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6804", 2020-11-16T20:07:22.601 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 19663 2020-11-16T20:07:22.601 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.601 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.601 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.602 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6805", 2020-11-16T20:07:22.602 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 19663 2020-11-16T20:07:22.602 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.602 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.602 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.602 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:07:22.602 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.603 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.603 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.603 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6808", 2020-11-16T20:07:22.603 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 19663 2020-11-16T20:07:22.603 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.603 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.603 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.604 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6809", 2020-11-16T20:07:22.604 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 19663 2020-11-16T20:07:22.604 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.604 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.604 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.604 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:07:22.604 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.604 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.605 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.605 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6806", 2020-11-16T20:07:22.605 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 19663 2020-11-16T20:07:22.605 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.605 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.605 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.605 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6807", 2020-11-16T20:07:22.606 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 19663 2020-11-16T20:07:22.606 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.606 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.606 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.606 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6803/19663", 2020-11-16T20:07:22.606 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6805/19663", 2020-11-16T20:07:22.606 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6809/19663", 2020-11-16T20:07:22.607 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6807/19663", 2020-11-16T20:07:22.607 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:07:22.607 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:07:22.607 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:07:22.607 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.607 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.607 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.608 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:07:22.608 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "db96d31f-edcb-4682-9cca-4d4871efc940", 2020-11-16T20:07:22.608 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 0, 2020-11-16T20:07:22.608 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:07:22.608 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:07:22.608 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:07:22.608 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 0, 2020-11-16T20:07:22.609 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 0, 2020-11-16T20:07:22.609 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 10, 2020-11-16T20:07:22.609 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 22, 2020-11-16T20:07:22.609 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 30, 2020-11-16T20:07:22.609 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:07:22.609 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:07:22.609 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.610 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.610 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.610 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:07:22.610 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 14704 2020-11-16T20:07:22.610 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.610 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.610 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.611 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:07:22.611 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 14704 2020-11-16T20:07:22.611 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.611 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.611 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.611 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:07:22.611 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.612 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.612 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.612 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:07:22.612 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 14704 2020-11-16T20:07:22.612 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.612 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.612 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.613 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:07:22.613 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 14704 2020-11-16T20:07:22.613 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.613 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.613 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.613 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:07:22.613 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.614 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.614 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.614 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:07:22.614 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 14704 2020-11-16T20:07:22.615 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.615 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.615 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.615 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:07:22.615 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 14704 2020-11-16T20:07:22.615 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.616 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.616 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.616 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:07:22.616 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.616 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.616 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.617 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:07:22.617 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 14704 2020-11-16T20:07:22.617 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.617 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.617 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.617 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:07:22.617 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 14704 2020-11-16T20:07:22.618 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.618 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.618 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.618 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/14704", 2020-11-16T20:07:22.618 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/14704", 2020-11-16T20:07:22.618 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/14704", 2020-11-16T20:07:22.618 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/14704", 2020-11-16T20:07:22.619 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:07:22.619 INFO:tasks.workunit.client.0.smithi189.stdout: "exists" 2020-11-16T20:07:22.619 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.619 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.619 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.619 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:07:22.619 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "abc43c59-78f4-4d31-87b4-2eb042089e2d", 2020-11-16T20:07:22.620 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:07:22.620 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:07:22.620 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:07:22.620 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:07:22.620 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 15, 2020-11-16T20:07:22.620 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 28, 2020-11-16T20:07:22.620 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 35, 2020-11-16T20:07:22.621 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 0, 2020-11-16T20:07:22.621 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 31, 2020-11-16T20:07:22.621 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:07:22.621 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:07:22.621 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.621 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.622 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.622 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:07:22.622 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 20062 2020-11-16T20:07:22.622 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.622 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.622 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.622 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:07:22.623 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 20062 2020-11-16T20:07:22.623 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.623 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.623 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.623 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:07:22.623 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.623 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.624 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.624 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:07:22.624 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 20062 2020-11-16T20:07:22.624 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.624 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.624 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.624 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:07:22.625 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 20062 2020-11-16T20:07:22.625 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.625 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.625 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.625 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:07:22.625 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.625 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.626 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.626 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:07:22.626 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 20062 2020-11-16T20:07:22.626 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.626 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.626 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.627 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:07:22.627 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 20062 2020-11-16T20:07:22.627 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.627 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.627 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.627 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:07:22.627 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:07:22.628 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.628 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:07:22.628 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:07:22.628 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 20062 2020-11-16T20:07:22.628 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.628 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.628 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:07:22.629 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:07:22.629 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 20062 2020-11-16T20:07:22.629 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.629 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.629 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.629 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/20062", 2020-11-16T20:07:22.629 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/20062", 2020-11-16T20:07:22.630 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/20062", 2020-11-16T20:07:22.630 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/20062", 2020-11-16T20:07:22.630 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:07:22.630 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:07:22.630 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:07:22.630 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.630 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.631 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.631 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_xinfo": [ 2020-11-16T20:07:22.631 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.631 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:07:22.631 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:06:32.808682+0000", 2020-11-16T20:07:22.631 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:07:22.631 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:07:22.632 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:07:22.632 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:07:22.632 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:05:29.263826+0000", 2020-11-16T20:07:22.632 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:07:22.632 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.632 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.632 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:07:22.633 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:06:33.426896+0000", 2020-11-16T20:07:22.633 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:07:22.633 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:07:22.633 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:07:22.634 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:07:22.634 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:05:37.335703+0000", 2020-11-16T20:07:22.634 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:07:22.634 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.634 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.634 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:07:22.635 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:06:33.997764+0000", 2020-11-16T20:07:22.635 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:07:22.635 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:07:22.635 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:07:22.635 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:07:22.635 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:05:45.677158+0000", 2020-11-16T20:07:22.635 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:07:22.636 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.636 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.636 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap": [], 2020-11-16T20:07:22.636 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap_items": [], 2020-11-16T20:07:22.636 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_temp": [], 2020-11-16T20:07:22.636 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_temp": [], 2020-11-16T20:07:22.636 INFO:tasks.workunit.client.0.smithi189.stdout: "blocklist": {}, 2020-11-16T20:07:22.636 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profiles": { 2020-11-16T20:07:22.637 INFO:tasks.workunit.client.0.smithi189.stdout: "default": { 2020-11-16T20:07:22.637 INFO:tasks.workunit.client.0.smithi189.stdout: "k": "2", 2020-11-16T20:07:22.637 INFO:tasks.workunit.client.0.smithi189.stdout: "m": "2", 2020-11-16T20:07:22.637 INFO:tasks.workunit.client.0.smithi189.stdout: "plugin": "jerasure", 2020-11-16T20:07:22.637 INFO:tasks.workunit.client.0.smithi189.stdout: "technique": "reed_sol_van" 2020-11-16T20:07:22.637 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.637 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.638 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps_queue": [], 2020-11-16T20:07:22.638 INFO:tasks.workunit.client.0.smithi189.stdout: "new_removed_snaps": [], 2020-11-16T20:07:22.638 INFO:tasks.workunit.client.0.smithi189.stdout: "new_purged_snaps": [], 2020-11-16T20:07:22.638 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_node_flags": {}, 2020-11-16T20:07:22.638 INFO:tasks.workunit.client.0.smithi189.stdout: "device_class_flags": {}, 2020-11-16T20:07:22.638 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": { 2020-11-16T20:07:22.638 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_enabled": false, 2020-11-16T20:07:22.639 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_bucket_count": 0, 2020-11-16T20:07:22.639 INFO:tasks.workunit.client.0.smithi189.stdout: "degraded_stretch_mode": 0, 2020-11-16T20:07:22.639 INFO:tasks.workunit.client.0.smithi189.stdout: "recovering_stretch_mode": 0, 2020-11-16T20:07:22.639 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_bucket": 0 2020-11-16T20:07:22.639 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.639 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.639 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_metadata": [ 2020-11-16T20:07:22.639 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.640 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:07:22.640 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:07:22.640 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6804/19663,v1:127.0.0.1:6805/19663]", 2020-11-16T20:07:22.640 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:07:22.640 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:07:22.640 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:07:22.640 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:07:22.641 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:07:22.641 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:07:22.641 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:07:22.641 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:07:22.641 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:07:22.641 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-11-16T20:07:22.641 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:07:22.641 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:07:22.642 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:07:22.642 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:07:22.642 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:07:22.642 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:07:22.642 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:07:22.642 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:07:22.642 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:07:22.643 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:07:22.643 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:07:22.643 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:07:22.643 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:07:22.643 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:07:22.643 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:07:22.643 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6802/19663,v1:127.0.0.1:6803/19663]", 2020-11-16T20:07:22.643 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:07:22.644 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/19663,v1:127.0.0.1:6809/19663]", 2020-11-16T20:07:22.644 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/19663,v1:127.0.0.1:6807/19663]", 2020-11-16T20:07:22.644 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:07:22.644 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:07:22.644 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:07:22.644 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:07:22.644 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:07:22.645 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:07:22.645 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:07:22.645 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:07:22.645 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:07:22.645 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/0", 2020-11-16T20:07:22.645 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:07:22.645 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:07:22.645 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:07:22.646 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.646 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.646 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:07:22.646 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:07:22.646 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/14704,v1:127.0.0.1:6813/14704]", 2020-11-16T20:07:22.647 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:07:22.647 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:07:22.647 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:07:22.647 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:07:22.647 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:07:22.647 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:07:22.647 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:07:22.647 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:07:22.648 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:07:22.648 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-11-16T20:07:22.648 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:07:22.648 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:07:22.648 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:07:22.648 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:07:22.648 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:07:22.649 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:07:22.649 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:07:22.649 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:07:22.649 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:07:22.649 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:07:22.649 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:07:22.649 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:07:22.649 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:07:22.650 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:07:22.650 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:07:22.650 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/14704,v1:127.0.0.1:6811/14704]", 2020-11-16T20:07:22.650 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:07:22.650 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/14704,v1:127.0.0.1:6817/14704]", 2020-11-16T20:07:22.650 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/14704,v1:127.0.0.1:6815/14704]", 2020-11-16T20:07:22.650 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:07:22.651 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:07:22.651 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:07:22.651 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:07:22.651 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:07:22.651 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:07:22.651 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:07:22.651 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:07:22.652 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:07:22.652 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/1", 2020-11-16T20:07:22.652 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:07:22.652 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:07:22.652 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:07:22.652 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.652 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.652 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:07:22.653 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:07:22.653 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/20062,v1:127.0.0.1:6813/20062]", 2020-11-16T20:07:22.653 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:07:22.653 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:07:22.653 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:07:22.653 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:07:22.653 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:07:22.654 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:07:22.654 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:07:22.654 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:07:22.654 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:07:22.654 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-11-16T20:07:22.654 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:07:22.654 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:07:22.655 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:07:22.655 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:07:22.655 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:07:22.655 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:07:22.655 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:07:22.655 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:07:22.655 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:07:22.655 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:07:22.656 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:07:22.656 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:07:22.656 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:07:22.656 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:07:22.656 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:07:22.656 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/20062,v1:127.0.0.1:6811/20062]", 2020-11-16T20:07:22.656 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:07:22.657 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/20062,v1:127.0.0.1:6817/20062]", 2020-11-16T20:07:22.657 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/20062,v1:127.0.0.1:6815/20062]", 2020-11-16T20:07:22.657 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:07:22.657 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:07:22.657 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:07:22.657 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:07:22.657 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:07:22.657 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:07:22.658 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:07:22.658 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:07:22.658 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:07:22.658 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/2", 2020-11-16T20:07:22.658 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:07:22.658 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:07:22.658 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:07:22.659 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.659 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.659 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_clean_epochs": { 2020-11-16T20:07:22.659 INFO:tasks.workunit.client.0.smithi189.stdout: "min_last_epoch_clean": 0, 2020-11-16T20:07:22.659 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": { 2020-11-16T20:07:22.659 INFO:tasks.workunit.client.0.smithi189.stdout: "per_pool": [] 2020-11-16T20:07:22.659 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.660 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_epochs": [ 2020-11-16T20:07:22.660 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.660 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:07:22.660 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 34 2020-11-16T20:07:22.660 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.660 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.660 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:07:22.660 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 10 2020-11-16T20:07:22.661 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.661 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.661 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:07:22.661 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 35 2020-11-16T20:07:22.661 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.661 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.661 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.661 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_first_committed": 1, 2020-11-16T20:07:22.662 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_last_committed": 40, 2020-11-16T20:07:22.662 INFO:tasks.workunit.client.0.smithi189.stdout: "crushmap": { 2020-11-16T20:07:22.662 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": [ 2020-11-16T20:07:22.662 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.662 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:07:22.662 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.0", 2020-11-16T20:07:22.662 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:07:22.663 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.663 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.663 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:07:22.663 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.1", 2020-11-16T20:07:22.663 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:07:22.663 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.663 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.663 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:07:22.664 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.2", 2020-11-16T20:07:22.664 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:07:22.664 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.664 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.664 INFO:tasks.workunit.client.0.smithi189.stdout: "types": [ 2020-11-16T20:07:22.664 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.664 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 0, 2020-11-16T20:07:22.665 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd" 2020-11-16T20:07:22.665 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.665 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.665 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:07:22.665 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "host" 2020-11-16T20:07:22.665 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.665 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.665 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 2, 2020-11-16T20:07:22.666 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "chassis" 2020-11-16T20:07:22.666 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.666 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.666 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 3, 2020-11-16T20:07:22.666 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "rack" 2020-11-16T20:07:22.666 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.666 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.667 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 4, 2020-11-16T20:07:22.667 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "row" 2020-11-16T20:07:22.667 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.667 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.667 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 5, 2020-11-16T20:07:22.667 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pdu" 2020-11-16T20:07:22.667 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.667 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.668 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 6, 2020-11-16T20:07:22.668 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pod" 2020-11-16T20:07:22.668 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.668 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.668 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 7, 2020-11-16T20:07:22.668 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "room" 2020-11-16T20:07:22.668 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.669 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.669 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 8, 2020-11-16T20:07:22.669 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "datacenter" 2020-11-16T20:07:22.669 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.669 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.669 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 9, 2020-11-16T20:07:22.669 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "zone" 2020-11-16T20:07:22.669 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.670 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.670 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 10, 2020-11-16T20:07:22.670 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "region" 2020-11-16T20:07:22.670 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.670 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.670 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:07:22.670 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "root" 2020-11-16T20:07:22.671 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.671 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.671 INFO:tasks.workunit.client.0.smithi189.stdout: "buckets": [ 2020-11-16T20:07:22.671 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.671 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -1, 2020-11-16T20:07:22.671 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default", 2020-11-16T20:07:22.671 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:07:22.672 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:07:22.672 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:07:22.672 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:07:22.672 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:07:22.672 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:07:22.672 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.672 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:07:22.673 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:07:22.673 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:07:22.673 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.673 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.673 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.673 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.673 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -2, 2020-11-16T20:07:22.674 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default~hdd", 2020-11-16T20:07:22.674 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:07:22.674 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:07:22.674 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:07:22.674 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:07:22.674 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:07:22.675 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:07:22.675 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.675 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:07:22.675 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:07:22.675 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:07:22.675 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.675 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.676 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.676 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.676 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:07:22.676 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189", 2020-11-16T20:07:22.676 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:07:22.676 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:07:22.676 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:07:22.676 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:07:22.677 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:07:22.677 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:07:22.677 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.677 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:07:22.677 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:07:22.677 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:07:22.677 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.678 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.678 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:07:22.678 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:07:22.678 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:07:22.678 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.678 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.678 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:07:22.679 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:07:22.679 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:07:22.679 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.679 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.679 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.679 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.679 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:07:22.679 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189~hdd", 2020-11-16T20:07:22.680 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:07:22.680 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:07:22.680 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:07:22.680 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:07:22.680 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:07:22.680 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:07:22.680 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.681 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:07:22.681 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:07:22.681 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:07:22.681 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.681 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.681 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:07:22.681 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:07:22.681 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:07:22.682 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.682 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.682 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:07:22.682 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:07:22.682 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:07:22.682 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.682 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.683 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.683 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.683 INFO:tasks.workunit.client.0.smithi189.stdout: "rules": [ 2020-11-16T20:07:22.683 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.683 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 0, 2020-11-16T20:07:22.683 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "replicated_rule", 2020-11-16T20:07:22.683 INFO:tasks.workunit.client.0.smithi189.stdout: "ruleset": 0, 2020-11-16T20:07:22.683 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:07:22.684 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 1, 2020-11-16T20:07:22.684 INFO:tasks.workunit.client.0.smithi189.stdout: "max_size": 10, 2020-11-16T20:07:22.684 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2020-11-16T20:07:22.684 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.684 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2020-11-16T20:07:22.684 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2020-11-16T20:07:22.684 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2020-11-16T20:07:22.685 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.685 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.685 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_firstn", 2020-11-16T20:07:22.685 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2020-11-16T20:07:22.685 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2020-11-16T20:07:22.685 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.685 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.686 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2020-11-16T20:07:22.686 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.686 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:07:22.686 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.686 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.686 INFO:tasks.workunit.client.0.smithi189.stdout: "tunables": { 2020-11-16T20:07:22.686 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_tries": 0, 2020-11-16T20:07:22.687 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_fallback_tries": 0, 2020-11-16T20:07:22.687 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_total_tries": 50, 2020-11-16T20:07:22.687 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_descend_once": 1, 2020-11-16T20:07:22.687 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_vary_r": 1, 2020-11-16T20:07:22.687 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_stable": 1, 2020-11-16T20:07:22.687 INFO:tasks.workunit.client.0.smithi189.stdout: "straw_calc_version": 1, 2020-11-16T20:07:22.687 INFO:tasks.workunit.client.0.smithi189.stdout: "allowed_bucket_algs": 54, 2020-11-16T20:07:22.687 INFO:tasks.workunit.client.0.smithi189.stdout: "profile": "jewel", 2020-11-16T20:07:22.688 INFO:tasks.workunit.client.0.smithi189.stdout: "optimal_tunables": 1, 2020-11-16T20:07:22.688 INFO:tasks.workunit.client.0.smithi189.stdout: "legacy_tunables": 0, 2020-11-16T20:07:22.688 INFO:tasks.workunit.client.0.smithi189.stdout: "minimum_required_version": "jewel", 2020-11-16T20:07:22.688 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables": 1, 2020-11-16T20:07:22.688 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables2": 1, 2020-11-16T20:07:22.688 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v2_rules": 0, 2020-11-16T20:07:22.688 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables3": 1, 2020-11-16T20:07:22.689 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v3_rules": 0, 2020-11-16T20:07:22.689 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v4_buckets": 1, 2020-11-16T20:07:22.689 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables5": 1, 2020-11-16T20:07:22.689 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v5_rules": 0 2020-11-16T20:07:22.689 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.689 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_args": {} 2020-11-16T20:07:22.689 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.690 INFO:tasks.workunit.client.0.smithi189.stdout: "fsmap": { 2020-11-16T20:07:22.690 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:07:22.690 INFO:tasks.workunit.client.0.smithi189.stdout: "default_fscid": -1, 2020-11-16T20:07:22.690 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": { 2020-11-16T20:07:22.690 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": {}, 2020-11-16T20:07:22.690 INFO:tasks.workunit.client.0.smithi189.stdout: "ro_compat": {}, 2020-11-16T20:07:22.690 INFO:tasks.workunit.client.0.smithi189.stdout: "incompat": { 2020-11-16T20:07:22.691 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_1": "base v0.20", 2020-11-16T20:07:22.691 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_2": "client writeable ranges", 2020-11-16T20:07:22.691 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_3": "default file layouts on dirs", 2020-11-16T20:07:22.691 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_4": "dir inode in separate object", 2020-11-16T20:07:22.691 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_5": "mds uses versioned encoding", 2020-11-16T20:07:22.691 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_6": "dirfrag is stored in omap", 2020-11-16T20:07:22.692 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_8": "no anchor table", 2020-11-16T20:07:22.692 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_9": "file layout v2", 2020-11-16T20:07:22.692 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_10": "snaprealm v2" 2020-11-16T20:07:22.692 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.692 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.692 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_flags": { 2020-11-16T20:07:22.692 INFO:tasks.workunit.client.0.smithi189.stdout: "enable_multiple": false, 2020-11-16T20:07:22.692 INFO:tasks.workunit.client.0.smithi189.stdout: "ever_enabled_multiple": false 2020-11-16T20:07:22.693 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.693 INFO:tasks.workunit.client.0.smithi189.stdout: "standbys": [], 2020-11-16T20:07:22.693 INFO:tasks.workunit.client.0.smithi189.stdout: "filesystems": [] 2020-11-16T20:07:22.693 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.693 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_first_committed": 1, 2020-11-16T20:07:22.693 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_last_committed": 1, 2020-11-16T20:07:22.693 INFO:tasks.workunit.client.0.smithi189.stdout: "auth": { 2020-11-16T20:07:22.694 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:07:22.694 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 5, 2020-11-16T20:07:22.694 INFO:tasks.workunit.client.0.smithi189.stdout: "num_secrets": 3 2020-11-16T20:07:22.694 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.694 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 1, 2020-11-16T20:07:22.694 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_active": 1, 2020-11-16T20:07:22.694 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_unknown": 0, 2020-11-16T20:07:22.695 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osd": 3, 2020-11-16T20:07:22.695 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_sum": { 2020-11-16T20:07:22.695 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:07:22.695 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 61700, 2020-11-16T20:07:22.695 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 100, 2020-11-16T20:07:22.695 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 0, 2020-11-16T20:07:22.695 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 300, 2020-11-16T20:07:22.695 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:07:22.696 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:07:22.696 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 100, 2020-11-16T20:07:22.696 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:07:22.696 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:07:22.696 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 100, 2020-11-16T20:07:22.696 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:07:22.696 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:07:22.697 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:07:22.697 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 100, 2020-11-16T20:07:22.697 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 100, 2020-11-16T20:07:22.697 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:07:22.697 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:07:22.697 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:07:22.697 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:07:22.698 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:07:22.698 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:07:22.698 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:07:22.698 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:07:22.698 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:07:22.698 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:07:22.698 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:07:22.698 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:07:22.699 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:07:22.699 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:07:22.699 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:07:22.699 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:07:22.699 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:07:22.699 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:07:22.699 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:07:22.700 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:07:22.700 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:07:22.700 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:07:22.700 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:07:22.700 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:07:22.700 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:07:22.700 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.701 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:07:22.701 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:07:22.701 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:07:22.701 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:07:22.701 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 0, 2020-11-16T20:07:22.701 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 0, 2020-11-16T20:07:22.701 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:07:22.701 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:07:22.702 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:07:22.702 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:07:22.702 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:07:22.702 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.702 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 16, 2020-11-16T20:07:22.702 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 16, 2020-11-16T20:07:22.702 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 2, 2020-11-16T20:07:22.703 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 2, 2020-11-16T20:07:22.703 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 0 2020-11-16T20:07:22.703 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.703 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum": { 2020-11-16T20:07:22.703 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:07:22.703 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:07:22.703 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 3, 2020-11-16T20:07:22.704 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:07:22.704 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:07:22.704 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:07:22.704 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:07:22.704 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 18244, 2020-11-16T20:07:22.704 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2088, 2020-11-16T20:07:22.704 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:07:22.704 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 16126, 2020-11-16T20:07:22.705 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314554556, 2020-11-16T20:07:22.705 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:07:22.705 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:07:22.705 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322103865344, 2020-11-16T20:07:22.705 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:07:22.705 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2138112, 2020-11-16T20:07:22.705 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 449020, 2020-11-16T20:07:22.706 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:07:22.706 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:07:22.706 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:07:22.706 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:07:22.706 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 16513144 2020-11-16T20:07:22.706 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.706 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:07:22.706 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:07:22.707 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:07:22.707 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 1, 2020-11-16T20:07:22.707 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:07:22.707 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:07:22.707 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:07:22.707 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.708 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:07:22.708 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 0, 2020-11-16T20:07:22.708 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 0, 2020-11-16T20:07:22.708 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 0, 2020-11-16T20:07:22.708 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 0 2020-11-16T20:07:22.708 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.708 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:07:22.709 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:07:22.709 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.709 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum_by_class": { 2020-11-16T20:07:22.709 INFO:tasks.workunit.client.0.smithi189.stdout: "hdd": { 2020-11-16T20:07:22.709 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:07:22.709 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:07:22.709 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 3, 2020-11-16T20:07:22.709 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:07:22.710 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:07:22.710 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:07:22.710 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:07:22.710 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 18244, 2020-11-16T20:07:22.710 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2088, 2020-11-16T20:07:22.710 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:07:22.711 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 16126, 2020-11-16T20:07:22.711 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314554556, 2020-11-16T20:07:22.711 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:07:22.711 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:07:22.711 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322103865344, 2020-11-16T20:07:22.711 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:07:22.711 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2138112, 2020-11-16T20:07:22.711 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 449020, 2020-11-16T20:07:22.712 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:07:22.712 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:07:22.712 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:07:22.712 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:07:22.712 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 16513144 2020-11-16T20:07:22.712 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.712 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:07:22.713 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:07:22.713 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:07:22.713 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 1, 2020-11-16T20:07:22.713 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:07:22.713 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:07:22.713 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:07:22.713 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.714 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:07:22.714 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 0, 2020-11-16T20:07:22.714 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 0, 2020-11-16T20:07:22.714 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 0, 2020-11-16T20:07:22.714 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 0 2020-11-16T20:07:22.714 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.714 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:07:22.715 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:07:22.715 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.715 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.715 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_stats": [ 2020-11-16T20:07:22.715 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.715 INFO:tasks.workunit.client.0.smithi189.stdout: "poolid": 1, 2020-11-16T20:07:22.715 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 1, 2020-11-16T20:07:22.716 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:07:22.716 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 61700, 2020-11-16T20:07:22.716 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 100, 2020-11-16T20:07:22.716 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 0, 2020-11-16T20:07:22.716 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 300, 2020-11-16T20:07:22.716 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:07:22.717 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:07:22.717 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 100, 2020-11-16T20:07:22.717 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:07:22.717 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:07:22.717 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 100, 2020-11-16T20:07:22.717 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:07:22.717 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:07:22.718 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:07:22.718 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 100, 2020-11-16T20:07:22.718 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 100, 2020-11-16T20:07:22.718 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:07:22.718 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:07:22.718 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:07:22.718 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:07:22.719 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:07:22.719 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:07:22.719 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:07:22.719 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:07:22.719 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:07:22.719 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:07:22.720 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:07:22.720 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:07:22.720 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:07:22.720 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:07:22.720 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:07:22.720 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:07:22.720 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:07:22.721 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:07:22.721 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:07:22.721 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:07:22.721 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:07:22.721 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:07:22.721 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:07:22.721 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:07:22.722 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:07:22.722 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.722 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:07:22.722 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:07:22.722 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:07:22.722 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:07:22.722 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 1228800, 2020-11-16T20:07:22.723 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 185100, 2020-11-16T20:07:22.723 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:07:22.723 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:07:22.723 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:07:22.723 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:07:22.723 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:07:22.724 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.724 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 16, 2020-11-16T20:07:22.724 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 16, 2020-11-16T20:07:22.724 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 2, 2020-11-16T20:07:22.724 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 2, 2020-11-16T20:07:22.724 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 3 2020-11-16T20:07:22.724 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.725 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.725 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_stats": [ 2020-11-16T20:07:22.725 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.725 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:07:22.725 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 146028888074 2020-11-16T20:07:22.725 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.726 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.726 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:07:22.726 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 42949672977 2020-11-16T20:07:22.726 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.726 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.726 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:07:22.727 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 150323855370 2020-11-16T20:07:22.727 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.727 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.727 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_state": [ 2020-11-16T20:07:22.727 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.727 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "active+undersized+degraded", 2020-11-16T20:07:22.727 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 1 2020-11-16T20:07:22.728 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.728 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.728 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_osd": [ 2020-11-16T20:07:22.728 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.728 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:07:22.728 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:07:22.729 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:07:22.729 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:07:22.729 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.729 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.729 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:07:22.729 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:07:22.729 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 0, 2020-11-16T20:07:22.730 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:07:22.730 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.730 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.730 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:07:22.730 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 1, 2020-11-16T20:07:22.730 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:07:22.731 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:07:22.731 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.731 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.731 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": [ 2020-11-16T20:07:22.731 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:07:22.731 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:07:22.732 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": {} 2020-11-16T20:07:22.732 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.732 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:07:22.732 INFO:tasks.workunit.client.0.smithi189.stdout: "servicemap": { 2020-11-16T20:07:22.732 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 4, 2020-11-16T20:07:22.732 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:06:39.190215+0000", 2020-11-16T20:07:22.732 INFO:tasks.workunit.client.0.smithi189.stdout: "services": { 2020-11-16T20:07:22.733 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": { 2020-11-16T20:07:22.733 INFO:tasks.workunit.client.0.smithi189.stdout: "daemons": { 2020-11-16T20:07:22.733 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": "", 2020-11-16T20:07:22.733 INFO:tasks.workunit.client.0.smithi189.stdout: "0": { 2020-11-16T20:07:22.733 INFO:tasks.workunit.client.0.smithi189.stdout: "start_epoch": 0, 2020-11-16T20:07:22.733 INFO:tasks.workunit.client.0.smithi189.stdout: "start_stamp": "0.000000", 2020-11-16T20:07:22.733 INFO:tasks.workunit.client.0.smithi189.stdout: "gid": 0, 2020-11-16T20:07:22.734 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "(unrecognized address family 0)/0", 2020-11-16T20:07:22.734 INFO:tasks.workunit.client.0.smithi189.stdout: "metadata": {}, 2020-11-16T20:07:22.734 INFO:tasks.workunit.client.0.smithi189.stdout: "task_status": {} 2020-11-16T20:07:22.734 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.734 INFO:tasks.workunit.client.0.smithi189.stdout: "2": { 2020-11-16T20:07:22.734 INFO:tasks.workunit.client.0.smithi189.stdout: "start_epoch": 0, 2020-11-16T20:07:22.734 INFO:tasks.workunit.client.0.smithi189.stdout: "start_stamp": "0.000000", 2020-11-16T20:07:22.735 INFO:tasks.workunit.client.0.smithi189.stdout: "gid": 0, 2020-11-16T20:07:22.735 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "(unrecognized address family 0)/0", 2020-11-16T20:07:22.735 INFO:tasks.workunit.client.0.smithi189.stdout: "metadata": {}, 2020-11-16T20:07:22.735 INFO:tasks.workunit.client.0.smithi189.stdout: "task_status": {} 2020-11-16T20:07:22.735 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.735 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.735 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.736 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.736 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:07:22.736 INFO:tasks.workunit.client.0.smithi189.stdout: "paxos": { 2020-11-16T20:07:22.736 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:07:22.736 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 359, 2020-11-16T20:07:22.736 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pn": 0, 2020-11-16T20:07:22.736 INFO:tasks.workunit.client.0.smithi189.stdout: "accepted_pn": 0 2020-11-16T20:07:22.737 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:07:22.737 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:07:22.737 INFO:tasks.workunit.client.0.smithi189.stdout:writing non-divergent object existing_4 2020-11-16T20:07:22.737 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:07:22.737 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2020-11-16T20:06:43.592859+0000 28'100 40:125 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:07:22.737 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:07:22.737 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:07:22.738 INFO:tasks.workunit.client.0.smithi189.stdout:delay recovery 2020-11-16T20:07:22.738 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:07:22.738 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2020-11-16T20:06:43.592859+0000 28'100 40:125 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:07:22.738 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:07:22.738 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:07:22.738 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:07:22.739 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:07:22.739 INFO:tasks.workunit.client.0.smithi189.stdout:revive divergent 1 2020-11-16T20:07:22.739 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:07:22.739 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2020-11-16T20:06:43.592859+0000 28'100 40:125 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:07:22.739 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:07:22.739 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:07:22.739 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:07:23.348 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:07:23.344+0000 7fa1a3958d80 -1 osd.1 28 log_to_monitors {default=true} 2020-11-16T20:07:26.892 INFO:tasks.workunit.client.0.smithi189.stdout:delay recovery divergent 2020-11-16T20:07:26.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2020-11-16T20:07:26.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2020-11-16T20:07:27.269 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:07:27.269 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 100 0 0 61854 0 0 17 17 active+undersized+degraded 2020-11-16T20:06:43.592859+0000 40'101 41:128 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:07:27.269 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:07:27.270 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:07:27.271 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:07:27.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2020-11-16T20:07:27.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:07:27.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:07:27.289 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:07:27.289 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:07:27.290 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:07:27.290 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.1.asok 2020-11-16T20:07:27.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2020-11-16T20:07:27.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.1.asok set_recovery_delay 100000 2020-11-16T20:07:27.392 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:07:27.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2020-11-16T20:07:28.113 INFO:tasks.workunit.client.0.smithi189.stderr:noup is unset 2020-11-16T20:07:28.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2020-11-16T20:07:28.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:07:28.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:07:28.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:07:28.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:07:28.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:07:28.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:07:28.124 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:07:28.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:07:28.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:07:28.593 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 34 up_thru 38 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/19663,v1:127.0.0.1:6803/19663] [v2:127.0.0.1:6804/19663,v1:127.0.0.1:6805/19663] exists,up 4abbfb84-75bc-442b-83c0-1cfd2d77e8da 2020-11-16T20:07:28.594 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:07:28.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:07:28.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:07:28.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:07:28.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2020-11-16T20:07:28.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:07:28.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:07:28.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:07:28.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:07:28.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:07:28.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:07:28.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:07:28.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:07:28.986 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/21636,v1:127.0.0.1:6819/21636] [v2:127.0.0.1:6820/21636,v1:127.0.0.1:6821/21636] exists,up db96d31f-edcb-4682-9cca-4d4871efc940 2020-11-16T20:07:28.987 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:07:28.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:07:28.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:07:28.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:07:28.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2020-11-16T20:07:28.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:07:28.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:07:28.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:07:28.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:07:28.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:07:28.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:07:28.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:07:28.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:07:29.405 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/20062,v1:127.0.0.1:6811/20062] [v2:127.0.0.1:6812/20062,v1:127.0.0.1:6813/20062] exists,up abc43c59-78f4-4d31-87b4-2eb042089e2d 2020-11-16T20:07:29.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:07:29.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:07:29.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:07:29.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2020-11-16T20:07:29.754 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:07:29.755 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 peering 2020-11-16T20:07:28.205834+0000 28'101 44:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:07:29.755 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:07:29.755 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:07:29.757 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:07:29.765 INFO:tasks.workunit.client.0.smithi189.stdout:wait for peering 2020-11-16T20:07:29.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2020-11-16T20:07:29.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2020-11-16T20:07:30.119 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:07:30.119 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 peering 2020-11-16T20:07:28.205834+0000 28'101 44:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:07:30.120 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:07:30.120 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:07:30.121 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:07:30.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2020-11-16T20:07:39.691 INFO:tasks.workunit.client.0.smithi189.stdout:killing divergent 1 2020-11-16T20:07:39.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2020-11-16T20:07:39.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2020-11-16T20:07:40.069 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:07:40.069 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 2 2 0 0 61854 0 0 17 17 active+recovering+degraded+wait 2020-11-16T20:07:28.732526+0000 40'101 45:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:07:40.069 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:07:40.069 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:07:40.071 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:07:40.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2020-11-16T20:07:40.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:07:40.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:07:40.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:07:40.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:07:40.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:07:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:07:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2020-11-16T20:07:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2020-11-16T20:07:40.193 INFO:tasks.workunit.client.0.smithi189.stdout:reviving divergent 1 2020-11-16T20:07:40.577 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:07:40.578 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 2 2 0 0 61854 0 0 17 17 active+recovering+degraded+wait 2020-11-16T20:07:28.732526+0000 40'101 45:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:07:40.578 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:07:40.578 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:07:40.580 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:07:40.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2020-11-16T20:07:40.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:07:40.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:07:40.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T20:07:40.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:07:40.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:07:40.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:07:40.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:07:40.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:07:40.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:07:40.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:07:40.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:07:40.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:07:40.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:07:40.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:07:40.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:07:40.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:07:40.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:07:40.594 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:07:40.594 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:07:40.594 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:07:40.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:07:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:07:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:07:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:07:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:07:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:07:40.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:07:40.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:07:40.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:07:40.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:07:40.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:07:40.597 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:07:40.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T20:07:40.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:07:40.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2020-11-16T20:07:40.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T20:07:40.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:07:40.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:07:40.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:07:40.625 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:07:40.624+0000 7f3a18f73d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:07:40.635 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:07:40.632+0000 7f3a18f73d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:07:40.636 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:07:40.636+0000 7f3a18f73d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:07:40.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T20:07:40.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:07:40.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:07:40.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:07:40.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:07:40.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:07:40.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:07:40.987 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:07:40.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:07:40.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:07:41.174 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:07:41.172+0000 7f3a18f73d80 -1 Falling back to public interface 2020-11-16T20:07:41.489 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/21636,v1:127.0.0.1:6819/21636] [v2:127.0.0.1:6820/21636,v1:127.0.0.1:6821/21636] exists,up db96d31f-edcb-4682-9cca-4d4871efc940 2020-11-16T20:07:41.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:07:41.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:07:41.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:07:41.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2020-11-16T20:07:42.072 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:07:42.068+0000 7f3a18f73d80 -1 osd.1 45 log_to_monitors {default=true} 2020-11-16T20:08:01.492 INFO:tasks.workunit.client.0.smithi189.stdout:allowing recovery 2020-11-16T20:08:01.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2020-11-16T20:08:01.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2020-11-16T20:08:01.880 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:08:01.880 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 0 0 0 0 62471 0 0 18 18 active+clean 2020-11-16T20:07:43.336216+0000 45'102 48:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:08:01.880 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:08:01.880 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:08:01.883 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:08:01.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-11-16T20:08:01.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2020-11-16T20:08:02.053 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:08:02.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-11-16T20:08:02.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T20:08:02.226 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:08:02.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-11-16T20:08:02.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2020-11-16T20:08:02.408 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:08:02.418 INFO:tasks.workunit.client.0.smithi189.stdout:reading divergent objects 2020-11-16T20:08:02.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2020-11-16T20:08:02.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2020-11-16T20:08:02.782 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:08:02.782 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 0 0 0 0 62471 0 0 18 18 active+clean 2020-11-16T20:07:43.336216+0000 45'102 48:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:05:48.106043+0000 0'0 2020-11-16T20:05:48.106043+0000 0 2020-11-16T20:08:02.782 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:08:02.782 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:08:02.784 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:08:02.796 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2020-11-16T20:08:02.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2020-11-16T20:08:02.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:08:02.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2020-11-16T20:08:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:08:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2020-11-16T20:08:02.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:08:02.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2020-11-16T20:08:02.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:08:02.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2020-11-16T20:08:02.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2020-11-16T20:08:02.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-11-16T20:08:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2020-11-16T20:08:02.988 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:07:28.524+0000 7fa181de4700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-11-16T20:06:21.342378+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-11-16T20:08:02.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:07:28.524+0000 7fa181de4700 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-11-16T20:06:21.342378+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-11-16T20:08:02.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:07:28.524+0000 7fa181de4700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:08:02.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:07:28.524+0000 7fa181de4700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-11-16T20:08:02.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:07:28.524+0000 7fa181de4700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-11-16T20:08:02.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:07:28.524+0000 7fa181de4700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-11-16T20:08:02.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:07:28.524+0000 7fa181de4700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'101 2020-11-16T20:08:02.990 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:07:28.524+0000 7fa181de4700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-11-16T20:06:21.342378+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-11-16T20:08:02.990 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:07:28.524+0000 7fa181de4700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-11-16T20:08:02.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-11-16T20:08:02.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2020-11-16T20:08:03.001 INFO:tasks.workunit.client.0.smithi189.stdout:success 2020-11-16T20:08:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2020-11-16T20:08:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2020-11-16T20:08:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T20:08:03.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T20:08:03.502 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T20:08:03.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2020-11-16T20:08:03.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:08:03.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:08:03.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:08:03.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:08:03.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:08:03.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:08:03.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-11-16T20:08:03.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-11-16T20:08:03.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:08:03.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-11-16T20:08:03.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:08:03.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:08:03.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:08:03.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:08:03.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:08:03.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:08:03.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:08:03.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:08:03.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:08:03.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:08:03.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:08:03.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:08:03.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:08:03.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:08:03.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:08:03.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:08:03.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:08:03.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:08:03.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-11-16T20:08:03.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:08:03.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:08:03.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:08:03.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12850 2020-11-16T20:08:03.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:08:03.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:08:03.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-11-16T20:08:03.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-11-16T20:08:03.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-11-16T20:08:03.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-11-16T20:08:03.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-11-16T20:08:03.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:08:03.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-11-16T20:08:03.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:08:03.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:08:03.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:08:03.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:08:03.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:08:03.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:08:03.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:08:03.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:08:03.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:08:03.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:08:03.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:08:03.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:08:03.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:08:03.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:08:03.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:08:03.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:08:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:08:03.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:08:03.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-11-16T20:08:03.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:08:03.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:08:03.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:08:03.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12850 2020-11-16T20:08:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:08:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:08:03.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-11-16T20:08:03.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:08:03.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:08:03.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:08:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12850 2020-11-16T20:08:03.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:08:03.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T20:08:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:08:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-11-16T20:08:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2020-11-16T20:08:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2020-11-16T20:08:03.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2020-11-16T20:08:03.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2020-11-16T20:08:03.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2020-11-16T20:08:03.701 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2020-11-16T20:08:03.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2020-11-16T20:08:03.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2020-11-16T20:08:03.705 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:08:03.705 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:08:03.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2020-11-16T20:08:03.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-11-16T20:08:03.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:08:03.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:08:03.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:08:03.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-11-16T20:08:03.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-11-16T20:08:04.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:08:04.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:08:04.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:08:04.015 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:08:04.015 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:08:04.015 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:08:04.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:08:04.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:08:04.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:08:04.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:08:04.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:08:04.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:08:04.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:08:04.221 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:08:04.221 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:08:04.222 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:08:04.222 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:08:04.222 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:08:04.223 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:08:04.223 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:08:04.223 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-mon.a.asok 2020-11-16T20:08:04.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:08:04.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.12850/ceph-mon.a.asok config get fsid 2020-11-16T20:08:04.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:08:04.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:08:04.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:08:04.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:08:04.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:08:04.330 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:08:04.331 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:08:04.331 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:08:04.331 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:08:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:08:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:08:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-mon.a.asok 2020-11-16T20:08:08.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:08:08.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.12850/ceph-mon.a.asok config get mon_host 2020-11-16T20:08:08.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2020-11-16T20:08:08.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2020-11-16T20:08:08.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:08:08.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:08:08.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:08:08.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2020-11-16T20:08:08.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:08:08.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:08:08.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:08:08.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:08:08.608 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:08:08.608 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:08:08.608 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:08:08.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:08:08.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:08:08.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:08:08.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-11-16T20:08:08.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2020-11-16T20:08:08.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:08:08.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:08:08.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:08:08.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:08:08.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2020-11-16T20:08:08.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:08:08.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:08:08.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:08:08.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:08:08.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-11-16T20:08:08.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-11-16T20:08:08.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:08:08.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:08:08.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:08:08.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:08:08.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:08:08.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:08:08.612 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:08:08.612 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:08:08.612 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:08:08.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:08:08.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:08:08.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:08:08.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:08:08.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:08:08.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:08:08.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:08:08.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:08:08.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:08:08.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:08:08.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:08:08.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:08:08.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2020-11-16T20:08:08.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:08:08.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6f8f3073-dcec-495d-8b27-191eea5f2c57 2020-11-16T20:08:08.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 6f8f3073-dcec-495d-8b27-191eea5f2c57' 2020-11-16T20:08:08.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:08:08.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAl3LJfwohgBBAAIoy+1C09hicfFa3ISFo2NQ== 2020-11-16T20:08:08.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAl3LJfwohgBBAAIoy+1C09hicfFa3ISFo2NQ=="}' 2020-11-16T20:08:08.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6f8f3073-dcec-495d-8b27-191eea5f2c57 -i td/divergent-priors/0/new.json 2020-11-16T20:08:08.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2020-11-16T20:08:08.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAl3LJfwohgBBAAIoy+1C09hicfFa3ISFo2NQ== --osd-uuid 6f8f3073-dcec-495d-8b27-191eea5f2c57 2020-11-16T20:08:08.615 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:05.468+0000 7f8dd8969d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:08.615 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:05.472+0000 7f8dd8969d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:08.615 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:05.472+0000 7f8dd8969d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:08.616 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:05.472+0000 7f8dd8969d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:08:08.616 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:05.472+0000 7f8dd8969d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:08:08.616 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:05.472+0000 7f8dd8969d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:08:08.616 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:05.564+0000 7f8dd8969d80 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-11-16T20:08:08.616 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 6f8f3073-dcec-495d-8b27-191eea5f2c57 2020-11-16T20:08:08.616 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:08:09.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-11-16T20:08:09.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:08:09.908 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:08:09.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:08:09.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:08:10.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:08:10.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:08:10.396 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:08:10.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:08:10.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:08:10.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:08:10.431 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:10.428+0000 7f662e0a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:10.433 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:10.432+0000 7f662e0a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:10.434 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:10.432+0000 7f662e0a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:10.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:08:10.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:08:10.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:08:10.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:08:10.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:08:10.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:08:10.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:08:10.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:08:10.778 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:08:10.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:08:10.969 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:10.968+0000 7f662e0a6d80 -1 Falling back to public interface 2020-11-16T20:08:11.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:08:11.824 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:11.816+0000 7f662e0a6d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:08:12.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:08:12.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:08:12.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:08:12.181 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:08:12.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:08:12.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:08:12.610 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/23730,v1:127.0.0.1:6803/23730] [v2:127.0.0.1:6804/23730,v1:127.0.0.1:6805/23730] exists,up 6f8f3073-dcec-495d-8b27-191eea5f2c57 2020-11-16T20:08:12.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:08:12.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:08:12.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:08:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-11-16T20:08:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2020-11-16T20:08:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:08:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:08:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:08:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:08:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:08:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:08:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:08:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:08:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:08:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:08:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:08:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:08:12.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:08:12.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:08:12.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:08:12.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:08:12.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:08:12.617 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:08:12.618 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:08:12.618 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:08:12.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:08:12.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:08:12.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:08:12.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:08:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:08:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:08:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:08:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:08:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:08:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:08:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:08:12.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:08:12.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:08:12.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:08:12.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ea764750-7154-4c9e-af61-617257854e05 2020-11-16T20:08:12.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 ea764750-7154-4c9e-af61-617257854e05' 2020-11-16T20:08:12.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:08:12.620 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 ea764750-7154-4c9e-af61-617257854e05 2020-11-16T20:08:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAs3LJfajtCJhAA1BqVu6UX2YPF9duVehVOTA== 2020-11-16T20:08:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAs3LJfajtCJhAA1BqVu6UX2YPF9duVehVOTA=="}' 2020-11-16T20:08:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ea764750-7154-4c9e-af61-617257854e05 -i td/divergent-priors/1/new.json 2020-11-16T20:08:13.086 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:08:13.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2020-11-16T20:08:13.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAs3LJfajtCJhAA1BqVu6UX2YPF9duVehVOTA== --osd-uuid ea764750-7154-4c9e-af61-617257854e05 2020-11-16T20:08:13.136 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:13.136+0000 7ffa5d8cdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:13.140 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:13.140+0000 7ffa5d8cdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:13.141 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:13.140+0000 7ffa5d8cdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:13.142 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:13.140+0000 7ffa5d8cdd80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:08:13.142 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:13.140+0000 7ffa5d8cdd80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:08:13.142 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:13.140+0000 7ffa5d8cdd80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:08:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:13.180+0000 7ffa5d8cdd80 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-11-16T20:08:17.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-11-16T20:08:17.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:08:17.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:08:17.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:08:17.904 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:08:18.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:08:18.395 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:08:18.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:08:18.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:08:18.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:08:18.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:08:18.429 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:18.428+0000 7fcb5215fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:18.431 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:18.428+0000 7fcb5215fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:18.432 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:18.432+0000 7fcb5215fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:18.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:08:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:08:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:08:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:08:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:08:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:08:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:08:18.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:08:18.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:08:18.786 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:08:18.998 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:18.996+0000 7fcb5215fd80 -1 Falling back to public interface 2020-11-16T20:08:19.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:08:19.827 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:19.828+0000 7fcb5215fd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:08:20.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:08:20.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:08:20.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:08:20.215 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:08:20.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:08:20.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:08:20.638 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/24725,v1:127.0.0.1:6811/24725] [v2:127.0.0.1:6812/24725,v1:127.0.0.1:6813/24725] exists,up ea764750-7154-4c9e-af61-617257854e05 2020-11-16T20:08:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:08:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:08:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:08:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-11-16T20:08:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2020-11-16T20:08:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:08:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:08:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:08:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:08:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2020-11-16T20:08:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:08:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:08:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:08:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:08:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-11-16T20:08:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-11-16T20:08:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:08:20.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:08:20.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:08:20.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:08:20.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:08:20.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:08:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:08:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:08:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:08:20.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:08:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:08:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:08:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:08:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:08:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:08:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:08:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:08:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:08:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:08:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:08:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:08:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2020-11-16T20:08:20.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:08:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=83927725-0d3c-4d38-9733-6d8054f1fa2b 2020-11-16T20:08:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 83927725-0d3c-4d38-9733-6d8054f1fa2b' 2020-11-16T20:08:20.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:08:20.647 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 83927725-0d3c-4d38-9733-6d8054f1fa2b 2020-11-16T20:08:20.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA03LJfDingJxAAuql9NS7csYBhZOqTmSrabQ== 2020-11-16T20:08:20.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA03LJfDingJxAAuql9NS7csYBhZOqTmSrabQ=="}' 2020-11-16T20:08:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 83927725-0d3c-4d38-9733-6d8054f1fa2b -i td/divergent-priors/2/new.json 2020-11-16T20:08:21.164 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:08:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2020-11-16T20:08:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA03LJfDingJxAAuql9NS7csYBhZOqTmSrabQ== --osd-uuid 83927725-0d3c-4d38-9733-6d8054f1fa2b 2020-11-16T20:08:21.213 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:21.212+0000 7ff6f5479d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:21.216 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:21.216+0000 7ff6f5479d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:21.217 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:21.216+0000 7ff6f5479d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:21.218 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:21.216+0000 7ff6f5479d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:08:21.218 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:21.216+0000 7ff6f5479d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:08:21.218 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:21.216+0000 7ff6f5479d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:08:21.293 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:21.296+0000 7ff6f5479d80 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-11-16T20:08:25.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-11-16T20:08:25.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:08:25.630 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:08:25.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:08:25.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:08:26.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:08:26.143 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:08:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:08:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:08:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:08:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:08:26.178 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:26.176+0000 7f54bbcc3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:26.180 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:26.180+0000 7f54bbcc3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:26.181 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:26.180+0000 7f54bbcc3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:08:26.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:08:26.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:08:26.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:08:26.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:08:26.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:08:26.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:08:26.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:08:26.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:08:26.534 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:08:26.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:08:26.725 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:26.724+0000 7f54bbcc3d80 -1 Falling back to public interface 2020-11-16T20:08:26.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:08:27.568 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:08:27.568+0000 7f54bbcc3d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:08:27.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:08:27.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:08:27.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:08:27.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:08:27.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:08:27.918 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:08:28.338 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/25719,v1:127.0.0.1:6819/25719] [v2:127.0.0.1:6820/25719,v1:127.0.0.1:6821/25719] exists,up 83927725-0d3c-4d38-9733-6d8054f1fa2b 2020-11-16T20:08:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:08:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:08:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:08:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2020-11-16T20:08:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:noout is set 2020-11-16T20:08:29.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2020-11-16T20:08:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:noin is set 2020-11-16T20:08:30.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2020-11-16T20:08:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:nodown is set 2020-11-16T20:08:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2020-11-16T20:08:30.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T20:08:31.351 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T20:08:31.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:08:32.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2020-11-16T20:08:33.096 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 3 2020-11-16T20:08:33.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2020-11-16T20:08:33.771 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 min_size to 2 2020-11-16T20:08:33.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2020-11-16T20:08:33.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:08:33.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:08:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:08:34.182 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:08:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:08:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:08:34.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:08:34.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:08:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2020-11-16T20:08:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2020-11-16T20:08:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2020-11-16T20:08:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:08:34.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:08:34.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2020-11-16T20:08:34.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2020-11-16T20:08:34.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2020-11-16T20:08:34.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:08:34.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:08:34.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2020-11-16T20:08:34.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2020-11-16T20:08:34.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509444' 2020-11-16T20:08:34.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:08:34.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2020-11-16T20:08:34.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:08:34.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:08:34.724 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2020-11-16T20:08:34.724 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:08:34.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2020-11-16T20:08:34.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-11-16T20:08:34.726 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836487 2020-11-16T20:08:34.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:08:35.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2020-11-16T20:08:35.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:08:35.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2020-11-16T20:08:35.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:08:35.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:08:35.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2020-11-16T20:08:35.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:08:35.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2020-11-16T20:08:35.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-11-16T20:08:35.131 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672965 2020-11-16T20:08:35.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:08:35.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672965 -lt 42949672965 2020-11-16T20:08:35.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:08:35.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2020-11-16T20:08:35.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:08:35.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:08:35.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2020-11-16T20:08:35.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:08:35.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2020-11-16T20:08:35.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-11-16T20:08:35.527 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509444 2020-11-16T20:08:35.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:08:36.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509442 -lt 64424509444 2020-11-16T20:08:36.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:08:37.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:08:37.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:08:37.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2020-11-16T20:08:37.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2020-11-16T20:08:37.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:08:37.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:08:37.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:08:37.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:08:37.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:08:37.458 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:08:37.458 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:08:37.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:08:37.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:08:37.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:08:37.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:08:37.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:08:37.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:08:37.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:08:37.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:08:38.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:08:38.034 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:08:38.035 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:08:38.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:08:38.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:08:38.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:08:38.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2020-11-16T20:08:38.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2020-11-16T20:08:38.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2020-11-16T20:08:38.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:08:38.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:08:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2020-11-16T20:08:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2020-11-16T20:08:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2020-11-16T20:08:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:08:38.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:08:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509446 2020-11-16T20:08:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509446 2020-11-16T20:08:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509446' 2020-11-16T20:08:38.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:08:38.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2020-11-16T20:08:38.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:08:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:08:38.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2020-11-16T20:08:38.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:08:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2020-11-16T20:08:38.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-11-16T20:08:38.572 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836490 2020-11-16T20:08:38.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:08:38.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2020-11-16T20:08:38.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:08:38.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2020-11-16T20:08:38.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:08:38.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:08:38.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:08:38.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2020-11-16T20:08:38.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2020-11-16T20:08:38.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-11-16T20:08:38.971 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672968 2020-11-16T20:08:38.972 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:08:39.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2020-11-16T20:08:39.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:08:39.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2020-11-16T20:08:39.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:08:39.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:08:39.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:08:39.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2020-11-16T20:08:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509446 2020-11-16T20:08:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-11-16T20:08:39.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:08:39.374 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509446 2020-11-16T20:08:39.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509446 -lt 64424509446 2020-11-16T20:08:39.757 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:08:39.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:08:39.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:08:40.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:08:40.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:08:40.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:08:40.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:08:40.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:08:40.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:08:40.263 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:08:40.263 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:08:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:08:40.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:08:40.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:08:40.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:08:41.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:08:41.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:08:41.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:08:41.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2020-11-16T20:08:41.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2020-11-16T20:08:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:08:41.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2020-11-16T20:08:41.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2020-11-16T20:08:41.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:08:41.487 INFO:tasks.workunit.client.0.smithi189.stdout:primary and soon to be divergent is 1 2020-11-16T20:08:41.835 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:08:41.836 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:08:32.065865+0000 0'0 28:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:08:41.836 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:08:41.836 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:08:41.838 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:08:41.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2020-11-16T20:08:41.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-11-16T20:08:41.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2020-11-16T20:08:41.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2020-11-16T20:08:41.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-11-16T20:08:41.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2020-11-16T20:08:41.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2020-11-16T20:08:41.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-11-16T20:08:41.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2020-11-16T20:08:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2020-11-16T20:08:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2020-11-16T20:08:41.849 INFO:tasks.workunit.client.0.smithi189.stdout:writing initial objects 2020-11-16T20:08:41.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2020-11-16T20:08:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2020-11-16T20:08:41.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:41.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2020-11-16T20:08:42.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:42.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2020-11-16T20:08:42.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:42.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2020-11-16T20:08:42.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:42.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2020-11-16T20:08:42.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:42.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2020-11-16T20:08:42.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:42.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2020-11-16T20:08:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2020-11-16T20:08:42.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:42.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2020-11-16T20:08:43.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:43.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2020-11-16T20:08:43.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:43.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2020-11-16T20:08:43.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:43.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2020-11-16T20:08:43.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:43.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2020-11-16T20:08:43.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:43.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2020-11-16T20:08:43.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:43.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2020-11-16T20:08:44.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:44.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2020-11-16T20:08:44.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:44.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2020-11-16T20:08:44.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:44.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2020-11-16T20:08:44.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:44.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2020-11-16T20:08:44.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:44.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2020-11-16T20:08:44.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:44.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2020-11-16T20:08:44.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:44.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2020-11-16T20:08:45.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:45.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2020-11-16T20:08:45.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:45.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2020-11-16T20:08:45.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:45.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2020-11-16T20:08:45.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:45.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2020-11-16T20:08:45.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:45.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2020-11-16T20:08:45.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:45.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2020-11-16T20:08:45.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:45.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2020-11-16T20:08:46.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:46.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2020-11-16T20:08:46.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:46.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2020-11-16T20:08:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2020-11-16T20:08:46.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:46.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2020-11-16T20:08:46.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:46.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2020-11-16T20:08:46.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:46.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2020-11-16T20:08:46.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:46.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2020-11-16T20:08:47.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:47.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2020-11-16T20:08:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2020-11-16T20:08:47.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:47.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2020-11-16T20:08:47.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:47.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2020-11-16T20:08:47.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:47.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2020-11-16T20:08:47.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:47.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2020-11-16T20:08:47.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:47.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2020-11-16T20:08:48.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:48.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2020-11-16T20:08:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:48.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2020-11-16T20:08:48.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:48.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2020-11-16T20:08:48.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:48.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2020-11-16T20:08:48.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:48.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2020-11-16T20:08:48.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:48.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2020-11-16T20:08:48.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:48.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2020-11-16T20:08:49.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:49.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2020-11-16T20:08:49.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:49.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2020-11-16T20:08:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:49.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2020-11-16T20:08:49.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:49.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2020-11-16T20:08:49.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:49.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2020-11-16T20:08:49.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:49.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2020-11-16T20:08:50.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:50.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2020-11-16T20:08:50.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:50.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2020-11-16T20:08:50.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:50.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2020-11-16T20:08:50.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:50.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2020-11-16T20:08:50.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:50.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2020-11-16T20:08:50.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:50.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2020-11-16T20:08:50.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:50.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2020-11-16T20:08:51.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:51.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2020-11-16T20:08:51.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:51.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2020-11-16T20:08:51.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:51.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2020-11-16T20:08:51.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:51.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2020-11-16T20:08:51.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:51.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2020-11-16T20:08:51.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:51.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2020-11-16T20:08:51.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:51.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2020-11-16T20:08:52.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:52.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2020-11-16T20:08:52.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:52.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2020-11-16T20:08:52.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:52.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2020-11-16T20:08:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2020-11-16T20:08:52.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:52.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2020-11-16T20:08:52.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:52.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2020-11-16T20:08:53.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:53.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2020-11-16T20:08:53.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:53.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2020-11-16T20:08:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2020-11-16T20:08:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2020-11-16T20:08:53.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:53.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2020-11-16T20:08:53.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:53.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2020-11-16T20:08:53.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:53.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2020-11-16T20:08:54.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:54.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2020-11-16T20:08:54.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:54.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2020-11-16T20:08:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2020-11-16T20:08:54.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:54.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2020-11-16T20:08:54.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:54.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2020-11-16T20:08:54.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:54.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2020-11-16T20:08:54.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:54.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2020-11-16T20:08:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2020-11-16T20:08:55.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:55.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2020-11-16T20:08:55.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:55.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2020-11-16T20:08:55.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:55.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2020-11-16T20:08:55.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:55.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2020-11-16T20:08:55.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:55.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2020-11-16T20:08:55.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:55.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2020-11-16T20:08:56.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:56.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2020-11-16T20:08:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2020-11-16T20:08:56.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-11-16T20:08:56.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2020-11-16T20:08:56.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-11-16T20:08:56.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2020-11-16T20:08:56.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:08:56.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:08:56.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:08:56.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:08:56.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2020-11-16T20:08:56.635 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:08:56.636 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:08:56.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:08:56.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:08:56.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:08:56.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:08:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:08:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:08:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:08:56.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:08:57.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:08:57.162 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:08:57.162 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:08:57.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:08:57.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:08:57.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:08:57.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836495 2020-11-16T20:08:57.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836495 2020-11-16T20:08:57.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495' 2020-11-16T20:08:57.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:08:57.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:08:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2020-11-16T20:08:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2020-11-16T20:08:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2020-11-16T20:08:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:08:57.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:08:57.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2020-11-16T20:08:57.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2020-11-16T20:08:57.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452' 2020-11-16T20:08:57.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:08:57.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2020-11-16T20:08:57.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:08:57.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:08:57.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2020-11-16T20:08:57.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:08:57.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836495 2020-11-16T20:08:57.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2020-11-16T20:08:57.696 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836495 2020-11-16T20:08:57.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:08:58.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836493 -lt 21474836495 2020-11-16T20:08:58.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:08:59.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:08:59.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:08:59.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836495 2020-11-16T20:08:59.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:08:59.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2020-11-16T20:08:59.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:08:59.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:08:59.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2020-11-16T20:08:59.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:08:59.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2020-11-16T20:08:59.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2020-11-16T20:08:59.507 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672974 2020-11-16T20:08:59.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:08:59.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2020-11-16T20:08:59.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:08:59.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2020-11-16T20:08:59.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:08:59.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:08:59.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:08:59.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2020-11-16T20:08:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2020-11-16T20:08:59.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2020-11-16T20:08:59.902 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509452 2020-11-16T20:08:59.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:09:00.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509452 -lt 64424509452 2020-11-16T20:09:00.307 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:09:00.308 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:00.308 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:00.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:09:00.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:09:00.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:09:00.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:09:00.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:09:00.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:09:00.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:09:00.782 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:09:01.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:09:01.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:09:01.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:01.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:01.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:09:01.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:09:01.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:09:01.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2020-11-16T20:09:01.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T20:09:01.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2020-11-16T20:09:01.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2020-11-16T20:09:01.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T20:09:01.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2020-11-16T20:09:01.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2020-11-16T20:09:01.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:09:02.000 INFO:tasks.workunit.client.0.smithi189.stdout:blackholing osds 0 2 2020-11-16T20:09:02.381 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:09:02.382 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2020-11-16T20:08:32.065865+0000 29'100 29:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:09:02.382 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:09:02.382 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:09:02.384 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:09:02.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-11-16T20:09:02.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2020-11-16T20:09:02.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T20:09:02.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T20:09:02.394 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:09:02.395 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:09:02.395 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:09:02.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.0.asok 2020-11-16T20:09:02.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-11-16T20:09:02.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.0.asok config set objectstore_blackhole 1 2020-11-16T20:09:02.492 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:09:02.493 INFO:tasks.workunit.client.0.smithi189.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-11-16T20:09:02.493 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:09:02.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-11-16T20:09:02.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2020-11-16T20:09:02.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-11-16T20:09:02.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-11-16T20:09:02.504 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:09:02.504 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:09:02.504 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:09:02.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.2.asok 2020-11-16T20:09:02.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-11-16T20:09:02.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.2.asok config set objectstore_blackhole 1 2020-11-16T20:09:02.602 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:09:02.603 INFO:tasks.workunit.client.0.smithi189.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-11-16T20:09:02.603 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:09:02.611 INFO:tasks.workunit.client.0.smithi189.stdout:create new divergent objects 2020-11-16T20:09:02.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2020-11-16T20:09:02.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2020-11-16T20:09:02.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-11-16T20:09:02.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-11-16T20:09:02.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2020-11-16T20:09:02.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2020-11-16T20:09:02.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2020-11-16T20:09:02.615 INFO:tasks.workunit.client.0.smithi189.stdout:writing divergent objects 2020-11-16T20:09:02.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2020-11-16T20:09:02.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-11-16T20:09:02.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2020-11-16T20:09:02.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-11-16T20:09:02.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2020-11-16T20:09:02.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2020-11-16T20:09:02.618 INFO:tasks.workunit.client.0.smithi189.stdout:remove divergent objects 2020-11-16T20:09:02.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2020-11-16T20:09:02.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-11-16T20:09:02.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2020-11-16T20:09:02.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2020-11-16T20:09:02.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-11-16T20:09:02.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2020-11-16T20:09:02.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2020-11-16T20:09:02.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2020-11-16T20:09:02.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2020-11-16T20:09:02.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2020-11-16T20:09:12.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2020-11-16T20:09:12.628 INFO:tasks.workunit.client.0.smithi189.stderr:rados: no process found 2020-11-16T20:09:12.628 INFO:tasks.workunit.client.0.smithi189.stdout:killing all the osds 2020-11-16T20:09:12.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2020-11-16T20:09:12.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:09:13.008 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:09:13.008 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 60774 0 0 16 16 active+clean 2020-11-16T20:08:32.065865+0000 29'106 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:09:13.008 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:09:13.008 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:09:13.010 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:09:13.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2020-11-16T20:09:13.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:09:13.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:09:13.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:09:13.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:09:13.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:09:13.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:09:13.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-11-16T20:09:13.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2020-11-16T20:09:13.683 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already down. 2020-11-16T20:09:13.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-11-16T20:09:13.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2020-11-16T20:09:14.231 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already down. 2020-11-16T20:09:14.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-11-16T20:09:14.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2020-11-16T20:09:14.780 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already down. 2020-11-16T20:09:14.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-11-16T20:09:14.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2020-11-16T20:09:15.352 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T20:09:15.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-11-16T20:09:15.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2020-11-16T20:09:15.910 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already out. 2020-11-16T20:09:15.920 INFO:tasks.workunit.client.0.smithi189.stdout:bringing up non_divergent 0 2 2020-11-16T20:09:15.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2020-11-16T20:09:15.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:09:16.287 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:09:16.287 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 60774 0 0 16 16 stale+active+clean 2020-11-16T20:08:32.065865+0000 29'106 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:09:16.287 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:09:16.287 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:09:16.289 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:09:16.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-11-16T20:09:16.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2020-11-16T20:09:16.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:09:16.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:09:16.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T20:09:16.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:09:16.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2020-11-16T20:09:16.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:09:16.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:09:16.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:09:16.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:09:16.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-11-16T20:09:16.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-11-16T20:09:16.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:09:16.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:09:16.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:09:16.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:09:16.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:09:16.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:09:16.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:09:16.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:09:16.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:09:16.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:09:16.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:09:16.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:09:16.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:09:16.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:09:16.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:09:16.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:09:16.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:09:16.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:09:16.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:09:16.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2020-11-16T20:09:16.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T20:09:16.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:09:16.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2020-11-16T20:09:16.307 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:09:16.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T20:09:16.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:09:16.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:09:16.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:09:16.335 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:09:16.336+0000 7ffa5b5cdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:09:16.337 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:09:16.336+0000 7ffa5b5cdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:09:16.338 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:09:16.340+0000 7ffa5b5cdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:09:16.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T20:09:16.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:09:16.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:09:16.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:09:16.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:09:16.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:09:16.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:09:16.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:09:16.698 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:09:16.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:09:16.896 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:09:16.896+0000 7ffa5b5cdd80 -1 Falling back to public interface 2020-11-16T20:09:17.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:09:18.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:09:18.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:09:18.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:09:18.090 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:09:18.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:09:18.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:09:18.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:09:18.521 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:09:18.520+0000 7ffa5b5cdd80 -1 osd.0 29 log_to_monitors {default=true} 2020-11-16T20:09:19.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:09:19.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:09:19.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:09:19.494 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:09:19.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:09:19.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:09:19.917 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/29620,v1:127.0.0.1:6803/29620] [v2:127.0.0.1:6804/29620,v1:127.0.0.1:6805/29620] exists,up 6f8f3073-dcec-495d-8b27-191eea5f2c57 2020-11-16T20:09:19.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:09:19.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:09:19.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:09:19.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-11-16T20:09:19.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2020-11-16T20:09:19.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:09:19.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:09:19.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-11-16T20:09:19.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:09:19.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2020-11-16T20:09:19.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:09:19.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:09:19.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:09:19.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:09:19.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-11-16T20:09:19.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-11-16T20:09:19.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:09:19.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:09:19.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:09:19.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:09:19.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:09:19.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:09:19.923 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:09:19.923 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:09:19.923 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:09:19.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:09:19.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:09:19.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:09:19.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:09:19.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:09:19.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:09:19.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:09:19.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:09:19.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:09:19.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:09:19.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2020-11-16T20:09:19.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-11-16T20:09:19.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:09:19.926 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:09:19.926 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2020-11-16T20:09:19.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-11-16T20:09:19.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:09:19.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:09:19.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:09:19.951 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:09:19.952+0000 7f1a4d255d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:09:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:09:19.952+0000 7f1a4d255d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:09:19.954 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:09:19.952+0000 7f1a4d255d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:09:20.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-11-16T20:09:20.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:09:20.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:09:20.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:09:20.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:09:20.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:09:20.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:09:20.300 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:09:20.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:09:20.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:09:20.508 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:09:20.508+0000 7f1a4d255d80 -1 Falling back to public interface 2020-11-16T20:09:20.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:09:21.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:09:21.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:09:21.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:09:21.705 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:09:21.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:09:21.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:09:21.869 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:09:21.868+0000 7f1a4d255d80 -1 osd.2 29 log_to_monitors {default=true} 2020-11-16T20:09:22.122 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/30046,v1:127.0.0.1:6811/30046] [v2:127.0.0.1:6812/30046,v1:127.0.0.1:6813/30046] exists,up 83927725-0d3c-4d38-9733-6d8054f1fa2b 2020-11-16T20:09:22.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:09:22.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:09:22.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:09:22.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-11-16T20:09:22.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2020-11-16T20:09:22.652 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already in. 2020-11-16T20:09:22.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-11-16T20:09:22.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2020-11-16T20:09:23.212 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already in. 2020-11-16T20:09:23.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-11-16T20:09:23.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2020-11-16T20:09:23.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:09:23.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:09:23.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:09:23.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:09:23.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2020-11-16T20:09:23.223 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:09:23.223 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:09:23.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:09:23.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:09:23.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:09:23.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:09:23.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:09:23.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:09:23.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:09:23.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:09:23.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:09:23.789 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:09:23.789 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:09:23.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:09:23.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:09:23.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:09:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855364 2020-11-16T20:09:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855364 2020-11-16T20:09:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-150323855364' 2020-11-16T20:09:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:09:23.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:09:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-11-16T20:09:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T20:09:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T20:09:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T20:09:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:09:24.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:09:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822659 2020-11-16T20:09:24.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822659 2020-11-16T20:09:24.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-150323855364 2-154618822659' 2020-11-16T20:09:24.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:09:24.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-150323855364 2020-11-16T20:09:24.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:09:24.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:09:24.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-150323855364 2020-11-16T20:09:24.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:09:24.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855364 2020-11-16T20:09:24.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2020-11-16T20:09:24.282 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 150323855364 2020-11-16T20:09:24.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:09:24.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 150323855364 2020-11-16T20:09:24.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:09:25.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:09:25.707 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:09:26.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855364 -lt 150323855364 2020-11-16T20:09:26.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:09:26.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-154618822659 2020-11-16T20:09:26.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:09:26.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:09:26.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-154618822659 2020-11-16T20:09:26.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:09:26.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822659 2020-11-16T20:09:26.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2020-11-16T20:09:26.140 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 154618822659 2020-11-16T20:09:26.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:09:26.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822659 -lt 154618822659 2020-11-16T20:09:26.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:09:26.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:26.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:09:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:09:27.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:09:27.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:09:27.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:09:27.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:09:27.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:09:27.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:09:27.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:09:27.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:09:27.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:27.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:09:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:09:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:09:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:09:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:09:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:09:28.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:09:28.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:09:28.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:09:28.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:09:28.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:09:28.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:09:28.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:09:28.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:09:28.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:09:28.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:09:28.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:28.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:28.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:09:28.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:09:28.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:09:28.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:09:28.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:09:28.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:09:28.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:09:28.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:09:28.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:09:29.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:09:29.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:09:29.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2020-11-16T20:09:29.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:09:29.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-11-16T20:09:29.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:09:29.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:09:29.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:09:29.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:09:29.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:09:29.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:09:29.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:09:29.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:09:30.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:09:30.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:09:30.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:30.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:30.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:09:30.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:09:30.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:09:30.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:09:30.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:09:30.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:09:30.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:09:30.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:09:30.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:09:30.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:09:30.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:09:30.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2020-11-16T20:09:30.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:09:30.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2020-11-16T20:09:31.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:09:31.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:09:31.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:09:31.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:09:31.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:09:31.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:09:31.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:09:31.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:09:31.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:09:31.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:09:31.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:31.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:32.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:09:32.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:09:32.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:09:32.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:09:32.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:09:32.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:09:32.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:09:32.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:09:32.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:09:32.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:09:32.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:09:32.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2020-11-16T20:09:32.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:09:32.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2020-11-16T20:09:33.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:09:33.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:09:33.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:09:33.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:09:33.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:09:33.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:09:33.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:09:33.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:09:33.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:09:33.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:09:33.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:33.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:34.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:09:34.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:09:34.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:09:34.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:09:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:09:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:09:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:09:34.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:09:34.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:09:34.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:09:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:09:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2020-11-16T20:09:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:09:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2020-11-16T20:09:36.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:09:36.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:09:36.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:09:36.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:09:36.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:09:36.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:09:36.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:09:36.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:09:36.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:09:36.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:09:36.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:36.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:37.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:09:37.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:09:37.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:09:37.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:09:37.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:09:37.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:09:37.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:09:37.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:09:37.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:09:37.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:09:37.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:09:37.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2020-11-16T20:09:37.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:09:37.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2020-11-16T20:09:41.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:09:41.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:09:41.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:09:41.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:09:41.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:09:41.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:09:41.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:09:41.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:09:41.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:09:41.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:09:41.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:41.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:41.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:09:41.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:09:41.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:09:41.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:09:41.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:09:41.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:09:41.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:09:41.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:09:41.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:09:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:09:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:09:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2020-11-16T20:09:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:09:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2020-11-16T20:09:48.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:09:48.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:09:48.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:09:48.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:09:48.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:09:48.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:09:48.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:09:48.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:09:49.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:09:49.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:09:49.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:49.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:49.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:09:49.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:09:49.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:09:49.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:09:49.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:09:49.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:09:49.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:09:49.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:09:49.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:09:50.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:09:50.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:09:50.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2020-11-16T20:09:50.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:09:50.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2020-11-16T20:09:57.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:09:57.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:09:57.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:09:57.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:09:57.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:09:57.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:09:57.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:09:57.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:09:57.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:09:57.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:09:57.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:09:57.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:09:58.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:09:58.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:09:58.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:09:58.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:09:58.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:09:58.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:09:58.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:09:58.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:09:58.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:09:58.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:09:58.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:09:58.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2020-11-16T20:09:58.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2020-11-16T20:09:59.312 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:09:59.312 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_fingerprint": "3c742d5f-ee9b-495b-a3f8-1f0879950819", 2020-11-16T20:09:59.312 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "16.0.0-7219-g911cc357", 2020-11-16T20:09:59.313 INFO:tasks.workunit.client.0.smithi189.stdout: "commit": "911cc35794d871071a8bebfb0919482263b1d18b", 2020-11-16T20:09:59.313 INFO:tasks.workunit.client.0.smithi189.stdout: "timestamp": "2020-11-16T20:09:59.316701+0000", 2020-11-16T20:09:59.313 INFO:tasks.workunit.client.0.smithi189.stdout: "tag": "", 2020-11-16T20:09:59.313 INFO:tasks.workunit.client.0.smithi189.stdout: "health": { 2020-11-16T20:09:59.313 INFO:tasks.workunit.client.0.smithi189.stdout: "status": "HEALTH_WARN", 2020-11-16T20:09:59.313 INFO:tasks.workunit.client.0.smithi189.stdout: "checks": { 2020-11-16T20:09:59.313 INFO:tasks.workunit.client.0.smithi189.stdout: "OSDMAP_FLAGS": { 2020-11-16T20:09:59.314 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:09:59.314 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:09:59.314 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "nodown,noout,noin flag(s) set", 2020-11-16T20:09:59.314 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 17 2020-11-16T20:09:59.314 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.314 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [], 2020-11-16T20:09:59.314 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:09:59.315 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.315 INFO:tasks.workunit.client.0.smithi189.stdout: "OSD_DOWN": { 2020-11-16T20:09:59.315 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:09:59.315 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:09:59.315 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 osds down", 2020-11-16T20:09:59.315 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:09:59.315 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.316 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:09:59.316 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.316 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "osd.1 (root=default,host=smithi189) is down" 2020-11-16T20:09:59.316 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.316 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.316 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:09:59.316 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.317 INFO:tasks.workunit.client.0.smithi189.stdout: "PG_DEGRADED": { 2020-11-16T20:09:59.317 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:09:59.317 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:09:59.317 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-11-16T20:09:59.317 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:09:59.317 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.317 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:09:59.317 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.318 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:09:59.318 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.318 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.318 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:09:59.318 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.318 INFO:tasks.workunit.client.0.smithi189.stdout: "POOL_APP_NOT_ENABLED": { 2020-11-16T20:09:59.318 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:09:59.319 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:09:59.319 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 pool(s) do not have an application enabled", 2020-11-16T20:09:59.319 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:09:59.319 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.319 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:09:59.319 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.319 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "application not enabled on pool 'test'" 2020-11-16T20:09:59.319 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.320 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.320 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-11-16T20:09:59.320 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.320 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.320 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:09:59.320 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.320 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.321 INFO:tasks.workunit.client.0.smithi189.stdout: "mutes": [] 2020-11-16T20:09:59.321 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.321 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_first_committed": 1, 2020-11-16T20:09:59.321 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_last_committed": 1, 2020-11-16T20:09:59.321 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap": { 2020-11-16T20:09:59.321 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:09:59.321 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:09:59.321 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:08:03.739653Z", 2020-11-16T20:09:59.322 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:08:03.739653Z", 2020-11-16T20:09:59.322 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release": 16, 2020-11-16T20:09:59.322 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release_name": "pacific", 2020-11-16T20:09:59.322 INFO:tasks.workunit.client.0.smithi189.stdout: "election_strategy": 1, 2020-11-16T20:09:59.322 INFO:tasks.workunit.client.0.smithi189.stdout: "disallowed_leaders: ": "", 2020-11-16T20:09:59.322 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": false, 2020-11-16T20:09:59.322 INFO:tasks.workunit.client.0.smithi189.stdout: "features": { 2020-11-16T20:09:59.323 INFO:tasks.workunit.client.0.smithi189.stdout: "persistent": [ 2020-11-16T20:09:59.323 INFO:tasks.workunit.client.0.smithi189.stdout: "kraken", 2020-11-16T20:09:59.323 INFO:tasks.workunit.client.0.smithi189.stdout: "luminous", 2020-11-16T20:09:59.323 INFO:tasks.workunit.client.0.smithi189.stdout: "mimic", 2020-11-16T20:09:59.323 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap-prune", 2020-11-16T20:09:59.323 INFO:tasks.workunit.client.0.smithi189.stdout: "nautilus", 2020-11-16T20:09:59.323 INFO:tasks.workunit.client.0.smithi189.stdout: "octopus", 2020-11-16T20:09:59.324 INFO:tasks.workunit.client.0.smithi189.stdout: "pacific", 2020-11-16T20:09:59.324 INFO:tasks.workunit.client.0.smithi189.stdout: "elector-pinging" 2020-11-16T20:09:59.324 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.324 INFO:tasks.workunit.client.0.smithi189.stdout: "optional": [] 2020-11-16T20:09:59.324 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.324 INFO:tasks.workunit.client.0.smithi189.stdout: "mons": [ 2020-11-16T20:09:59.324 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.324 INFO:tasks.workunit.client.0.smithi189.stdout: "rank": 0, 2020-11-16T20:09:59.325 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "a", 2020-11-16T20:09:59.325 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:09:59.325 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.325 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.325 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.325 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115", 2020-11-16T20:09:59.325 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 0 2020-11-16T20:09:59.326 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.326 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.326 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.326 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115/0", 2020-11-16T20:09:59.326 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:7115/0", 2020-11-16T20:09:59.326 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 0, 2020-11-16T20:09:59.326 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 0, 2020-11-16T20:09:59.326 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_location": "{}" 2020-11-16T20:09:59.327 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.327 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.327 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.327 INFO:tasks.workunit.client.0.smithi189.stdout: "quorum": [ 2020-11-16T20:09:59.327 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2020-11-16T20:09:59.327 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.327 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap": { 2020-11-16T20:09:59.328 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 39, 2020-11-16T20:09:59.328 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:09:59.328 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:08:04.527996+0000", 2020-11-16T20:09:59.328 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:09:23.302876+0000", 2020-11-16T20:09:59.328 INFO:tasks.workunit.client.0.smithi189.stdout: "last_up_change": "2020-11-16T20:09:21.954974+0000", 2020-11-16T20:09:59.328 INFO:tasks.workunit.client.0.smithi189.stdout: "last_in_change": "2020-11-16T20:09:22.991314+0000", 2020-11-16T20:09:59.328 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-11-16T20:09:59.328 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_num": 5800384, 2020-11-16T20:09:59.329 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_set": [ 2020-11-16T20:09:59.329 INFO:tasks.workunit.client.0.smithi189.stdout: "nodown", 2020-11-16T20:09:59.329 INFO:tasks.workunit.client.0.smithi189.stdout: "noin", 2020-11-16T20:09:59.329 INFO:tasks.workunit.client.0.smithi189.stdout: "noout", 2020-11-16T20:09:59.329 INFO:tasks.workunit.client.0.smithi189.stdout: "pglog_hardlimit", 2020-11-16T20:09:59.329 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snapdirs", 2020-11-16T20:09:59.329 INFO:tasks.workunit.client.0.smithi189.stdout: "recovery_deletes", 2020-11-16T20:09:59.330 INFO:tasks.workunit.client.0.smithi189.stdout: "sortbitwise" 2020-11-16T20:09:59.330 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.330 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_version": 7, 2020-11-16T20:09:59.330 INFO:tasks.workunit.client.0.smithi189.stdout: "full_ratio": 0.99000000953674316, 2020-11-16T20:09:59.330 INFO:tasks.workunit.client.0.smithi189.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-11-16T20:09:59.330 INFO:tasks.workunit.client.0.smithi189.stdout: "nearfull_ratio": 0.85000002384185791, 2020-11-16T20:09:59.330 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_snapshot": "", 2020-11-16T20:09:59.330 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_max": 1, 2020-11-16T20:09:59.331 INFO:tasks.workunit.client.0.smithi189.stdout: "max_osd": 3, 2020-11-16T20:09:59.331 INFO:tasks.workunit.client.0.smithi189.stdout: "require_min_compat_client": "luminous", 2020-11-16T20:09:59.331 INFO:tasks.workunit.client.0.smithi189.stdout: "min_compat_client": "jewel", 2020-11-16T20:09:59.331 INFO:tasks.workunit.client.0.smithi189.stdout: "require_osd_release": "pacific", 2020-11-16T20:09:59.331 INFO:tasks.workunit.client.0.smithi189.stdout: "pools": [ 2020-11-16T20:09:59.331 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.331 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:09:59.332 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_name": "test", 2020-11-16T20:09:59.332 INFO:tasks.workunit.client.0.smithi189.stdout: "create_time": "2020-11-16T20:08:31.085395+0000", 2020-11-16T20:09:59.332 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": 1, 2020-11-16T20:09:59.332 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_names": "hashpspool", 2020-11-16T20:09:59.332 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:09:59.332 INFO:tasks.workunit.client.0.smithi189.stdout: "size": 3, 2020-11-16T20:09:59.332 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 2, 2020-11-16T20:09:59.332 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_rule": 0, 2020-11-16T20:09:59.333 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_count": 0, 2020-11-16T20:09:59.333 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_target": 0, 2020-11-16T20:09:59.333 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_barrier": 0, 2020-11-16T20:09:59.333 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2020-11-16T20:09:59.333 INFO:tasks.workunit.client.0.smithi189.stdout: "object_hash": 2, 2020-11-16T20:09:59.333 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_autoscale_mode": "off", 2020-11-16T20:09:59.333 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num": 1, 2020-11-16T20:09:59.334 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num": 1, 2020-11-16T20:09:59.334 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num_target": 1, 2020-11-16T20:09:59.334 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_target": 1, 2020-11-16T20:09:59.334 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_pending": 1, 2020-11-16T20:09:59.334 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pg_merge_meta": { 2020-11-16T20:09:59.334 INFO:tasks.workunit.client.0.smithi189.stdout: "source_pgid": "0.0", 2020-11-16T20:09:59.334 INFO:tasks.workunit.client.0.smithi189.stdout: "ready_epoch": 0, 2020-11-16T20:09:59.334 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_started": 0, 2020-11-16T20:09:59.335 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": 0, 2020-11-16T20:09:59.335 INFO:tasks.workunit.client.0.smithi189.stdout: "source_version": "0'0", 2020-11-16T20:09:59.335 INFO:tasks.workunit.client.0.smithi189.stdout: "target_version": "0'0" 2020-11-16T20:09:59.335 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.335 INFO:tasks.workunit.client.0.smithi189.stdout: "last_change": "29", 2020-11-16T20:09:59.335 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend": "0", 2020-11-16T20:09:59.335 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_prenautilus": "0", 2020-11-16T20:09:59.336 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_preluminous": "0", 2020-11-16T20:09:59.336 INFO:tasks.workunit.client.0.smithi189.stdout: "auid": 0, 2020-11-16T20:09:59.336 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_mode": "selfmanaged", 2020-11-16T20:09:59.336 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_seq": 0, 2020-11-16T20:09:59.336 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_epoch": 0, 2020-11-16T20:09:59.336 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_snaps": [], 2020-11-16T20:09:59.336 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps": "[]", 2020-11-16T20:09:59.336 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_bytes": 0, 2020-11-16T20:09:59.337 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_objects": 0, 2020-11-16T20:09:59.337 INFO:tasks.workunit.client.0.smithi189.stdout: "tiers": [], 2020-11-16T20:09:59.337 INFO:tasks.workunit.client.0.smithi189.stdout: "tier_of": -1, 2020-11-16T20:09:59.337 INFO:tasks.workunit.client.0.smithi189.stdout: "read_tier": -1, 2020-11-16T20:09:59.337 INFO:tasks.workunit.client.0.smithi189.stdout: "write_tier": -1, 2020-11-16T20:09:59.337 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_mode": "none", 2020-11-16T20:09:59.337 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_bytes": 0, 2020-11-16T20:09:59.337 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_objects": 0, 2020-11-16T20:09:59.338 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-11-16T20:09:59.338 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-11-16T20:09:59.338 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_full_ratio_micro": 800000, 2020-11-16T20:09:59.338 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_flush_age": 0, 2020-11-16T20:09:59.338 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_evict_age": 0, 2020-11-16T20:09:59.338 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profile": "", 2020-11-16T20:09:59.338 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_params": { 2020-11-16T20:09:59.339 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "none" 2020-11-16T20:09:59.339 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.339 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_period": 0, 2020-11-16T20:09:59.339 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_count": 0, 2020-11-16T20:09:59.339 INFO:tasks.workunit.client.0.smithi189.stdout: "use_gmt_hitset": true, 2020-11-16T20:09:59.339 INFO:tasks.workunit.client.0.smithi189.stdout: "min_read_recency_for_promote": 0, 2020-11-16T20:09:59.339 INFO:tasks.workunit.client.0.smithi189.stdout: "min_write_recency_for_promote": 0, 2020-11-16T20:09:59.339 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_grade_decay_rate": 0, 2020-11-16T20:09:59.340 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_search_last_n": 0, 2020-11-16T20:09:59.340 INFO:tasks.workunit.client.0.smithi189.stdout: "grade_table": [], 2020-11-16T20:09:59.340 INFO:tasks.workunit.client.0.smithi189.stdout: "stripe_width": 0, 2020-11-16T20:09:59.340 INFO:tasks.workunit.client.0.smithi189.stdout: "expected_num_objects": 0, 2020-11-16T20:09:59.340 INFO:tasks.workunit.client.0.smithi189.stdout: "fast_read": false, 2020-11-16T20:09:59.340 INFO:tasks.workunit.client.0.smithi189.stdout: "options": {}, 2020-11-16T20:09:59.340 INFO:tasks.workunit.client.0.smithi189.stdout: "application_metadata": {} 2020-11-16T20:09:59.340 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.341 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.341 INFO:tasks.workunit.client.0.smithi189.stdout: "osds": [ 2020-11-16T20:09:59.341 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.341 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:09:59.341 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "6f8f3073-dcec-495d-8b27-191eea5f2c57", 2020-11-16T20:09:59.341 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:09:59.341 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:09:59.342 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:09:59.342 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:09:59.342 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 5, 2020-11-16T20:09:59.342 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 29, 2020-11-16T20:09:59.342 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 35, 2020-11-16T20:09:59.342 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 38, 2020-11-16T20:09:59.342 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 30, 2020-11-16T20:09:59.342 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:09:59.343 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:09:59.343 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.343 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.343 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.343 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6802", 2020-11-16T20:09:59.343 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:09:59.343 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.344 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.344 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.344 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6803", 2020-11-16T20:09:59.344 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:09:59.344 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.344 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.344 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.344 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:09:59.345 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.345 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.345 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.345 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6804", 2020-11-16T20:09:59.345 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:09:59.345 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.345 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.345 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.346 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6805", 2020-11-16T20:09:59.346 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:09:59.346 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.346 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.346 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.346 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:09:59.346 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.347 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.347 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.347 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6808", 2020-11-16T20:09:59.347 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:09:59.347 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.347 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.347 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.347 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6809", 2020-11-16T20:09:59.348 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:09:59.348 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.348 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.348 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.348 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:09:59.348 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.348 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.349 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.349 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6806", 2020-11-16T20:09:59.349 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:09:59.349 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.349 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.349 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.349 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6807", 2020-11-16T20:09:59.349 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:09:59.350 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.350 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.350 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.350 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6803/29620", 2020-11-16T20:09:59.350 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6805/29620", 2020-11-16T20:09:59.350 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6809/29620", 2020-11-16T20:09:59.350 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6807/29620", 2020-11-16T20:09:59.351 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:09:59.351 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:09:59.351 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:09:59.351 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.351 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.351 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.351 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:09:59.351 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "ea764750-7154-4c9e-af61-617257854e05", 2020-11-16T20:09:59.352 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 0, 2020-11-16T20:09:59.352 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:09:59.352 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:09:59.352 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:09:59.352 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 0, 2020-11-16T20:09:59.352 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 0, 2020-11-16T20:09:59.352 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 10, 2020-11-16T20:09:59.352 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 23, 2020-11-16T20:09:59.353 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 31, 2020-11-16T20:09:59.353 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:09:59.353 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:09:59.353 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.353 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.353 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.354 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:09:59.354 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:09:59.354 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.354 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.354 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.354 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:09:59.354 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:09:59.354 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.355 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.355 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.355 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:09:59.355 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.355 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.355 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.355 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:09:59.356 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:09:59.356 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.356 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.356 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.356 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:09:59.356 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:09:59.356 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.356 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.357 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.357 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:09:59.357 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.357 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.357 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.357 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:09:59.357 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:09:59.357 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.358 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.358 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.358 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:09:59.358 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:09:59.358 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.358 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.358 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.359 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:09:59.359 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.359 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.359 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.359 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:09:59.359 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:09:59.359 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.359 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.360 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.360 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:09:59.360 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:09:59.360 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.360 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.360 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.360 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/24725", 2020-11-16T20:09:59.360 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/24725", 2020-11-16T20:09:59.361 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/24725", 2020-11-16T20:09:59.361 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/24725", 2020-11-16T20:09:59.361 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:09:59.361 INFO:tasks.workunit.client.0.smithi189.stdout: "exists" 2020-11-16T20:09:59.361 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.361 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.361 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.362 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:09:59.362 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "83927725-0d3c-4d38-9733-6d8054f1fa2b", 2020-11-16T20:09:59.362 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:09:59.362 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:09:59.362 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:09:59.362 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:09:59.362 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 15, 2020-11-16T20:09:59.362 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 29, 2020-11-16T20:09:59.363 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 36, 2020-11-16T20:09:59.363 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 0, 2020-11-16T20:09:59.363 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 32, 2020-11-16T20:09:59.363 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:09:59.363 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:09:59.363 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.363 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.364 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.364 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:09:59.364 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:09:59.364 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.364 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.364 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.364 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:09:59.364 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:09:59.365 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.365 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.365 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.365 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:09:59.365 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.365 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.365 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.365 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:09:59.366 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:09:59.366 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.366 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.366 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.366 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:09:59.366 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:09:59.366 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.367 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.367 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.367 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:09:59.367 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.367 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.367 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.367 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:09:59.367 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:09:59.368 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.368 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.368 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.368 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:09:59.368 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:09:59.368 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.368 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.369 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.369 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:09:59.369 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:09:59.369 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.369 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:09:59.369 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:09:59.369 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:09:59.369 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.370 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.370 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:09:59.370 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:09:59.370 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:09:59.370 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.370 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.370 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.371 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/30046", 2020-11-16T20:09:59.371 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/30046", 2020-11-16T20:09:59.371 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30046", 2020-11-16T20:09:59.371 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30046", 2020-11-16T20:09:59.371 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:09:59.371 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:09:59.372 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:09:59.372 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.372 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.372 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.372 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_xinfo": [ 2020-11-16T20:09:59.372 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.372 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:09:59.373 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:09:13.490423+0000", 2020-11-16T20:09:59.373 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:09:59.373 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:09:59.373 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:09:59.373 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:09:59.373 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:08:12.796468+0000", 2020-11-16T20:09:59.373 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:09:59.374 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.374 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.374 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:09:59.374 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:09:14.042030+0000", 2020-11-16T20:09:59.374 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:09:59.374 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:09:59.374 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:09:59.374 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:09:59.375 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:08:20.791257+0000", 2020-11-16T20:09:59.375 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:09:59.375 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.375 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.375 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:09:59.375 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:09:14.584643+0000", 2020-11-16T20:09:59.375 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:09:59.376 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:09:59.376 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:09:59.376 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:09:59.376 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:08:28.548492+0000", 2020-11-16T20:09:59.376 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:09:59.376 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.376 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.376 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap": [], 2020-11-16T20:09:59.377 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap_items": [], 2020-11-16T20:09:59.377 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_temp": [], 2020-11-16T20:09:59.377 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_temp": [], 2020-11-16T20:09:59.377 INFO:tasks.workunit.client.0.smithi189.stdout: "blocklist": {}, 2020-11-16T20:09:59.377 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profiles": { 2020-11-16T20:09:59.377 INFO:tasks.workunit.client.0.smithi189.stdout: "default": { 2020-11-16T20:09:59.377 INFO:tasks.workunit.client.0.smithi189.stdout: "k": "2", 2020-11-16T20:09:59.377 INFO:tasks.workunit.client.0.smithi189.stdout: "m": "2", 2020-11-16T20:09:59.378 INFO:tasks.workunit.client.0.smithi189.stdout: "plugin": "jerasure", 2020-11-16T20:09:59.378 INFO:tasks.workunit.client.0.smithi189.stdout: "technique": "reed_sol_van" 2020-11-16T20:09:59.378 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.378 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.378 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps_queue": [], 2020-11-16T20:09:59.378 INFO:tasks.workunit.client.0.smithi189.stdout: "new_removed_snaps": [], 2020-11-16T20:09:59.378 INFO:tasks.workunit.client.0.smithi189.stdout: "new_purged_snaps": [], 2020-11-16T20:09:59.379 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_node_flags": {}, 2020-11-16T20:09:59.379 INFO:tasks.workunit.client.0.smithi189.stdout: "device_class_flags": {}, 2020-11-16T20:09:59.379 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": { 2020-11-16T20:09:59.379 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_enabled": false, 2020-11-16T20:09:59.379 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_bucket_count": 0, 2020-11-16T20:09:59.379 INFO:tasks.workunit.client.0.smithi189.stdout: "degraded_stretch_mode": 0, 2020-11-16T20:09:59.379 INFO:tasks.workunit.client.0.smithi189.stdout: "recovering_stretch_mode": 0, 2020-11-16T20:09:59.379 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_bucket": 0 2020-11-16T20:09:59.380 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.380 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.380 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_metadata": [ 2020-11-16T20:09:59.380 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.380 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:09:59.380 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:09:59.380 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6804/29620,v1:127.0.0.1:6805/29620]", 2020-11-16T20:09:59.381 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:09:59.381 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:09:59.381 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:09:59.381 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:09:59.381 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:09:59.381 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:09:59.381 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:09:59.381 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:09:59.382 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:09:59.382 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-11-16T20:09:59.382 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:09:59.382 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:09:59.382 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:09:59.382 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:09:59.382 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:09:59.382 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:09:59.383 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:09:59.383 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:09:59.383 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:09:59.383 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:09:59.383 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:09:59.383 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:09:59.383 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:09:59.384 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:09:59.384 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:09:59.384 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6802/29620,v1:127.0.0.1:6803/29620]", 2020-11-16T20:09:59.384 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:09:59.384 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/29620,v1:127.0.0.1:6809/29620]", 2020-11-16T20:09:59.384 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/29620,v1:127.0.0.1:6807/29620]", 2020-11-16T20:09:59.384 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:09:59.384 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:09:59.385 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:09:59.385 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:09:59.385 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:09:59.385 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:09:59.385 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:09:59.385 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:09:59.385 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:09:59.386 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/0", 2020-11-16T20:09:59.386 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:09:59.386 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:09:59.386 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:09:59.386 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.386 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.386 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:09:59.386 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:09:59.387 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/24725,v1:127.0.0.1:6813/24725]", 2020-11-16T20:09:59.387 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:09:59.387 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:09:59.387 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:09:59.387 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:09:59.387 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:09:59.387 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:09:59.387 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:09:59.388 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:09:59.388 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:09:59.388 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-11-16T20:09:59.388 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:09:59.388 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:09:59.388 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:09:59.388 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:09:59.389 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:09:59.389 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:09:59.389 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:09:59.389 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:09:59.389 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:09:59.389 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:09:59.389 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:09:59.389 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:09:59.390 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:09:59.390 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:09:59.390 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:09:59.390 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/24725,v1:127.0.0.1:6811/24725]", 2020-11-16T20:09:59.390 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:09:59.390 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/24725,v1:127.0.0.1:6817/24725]", 2020-11-16T20:09:59.390 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/24725,v1:127.0.0.1:6815/24725]", 2020-11-16T20:09:59.391 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:09:59.391 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:09:59.391 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:09:59.391 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:09:59.391 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:09:59.391 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:09:59.391 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:09:59.391 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:09:59.392 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:09:59.392 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/1", 2020-11-16T20:09:59.392 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:09:59.392 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:09:59.392 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:09:59.392 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.392 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.393 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:09:59.393 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:09:59.393 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/30046,v1:127.0.0.1:6813/30046]", 2020-11-16T20:09:59.393 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:09:59.393 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:09:59.393 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:09:59.393 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:09:59.393 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:09:59.394 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:09:59.394 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:09:59.394 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:09:59.394 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:09:59.394 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-11-16T20:09:59.394 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:09:59.394 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:09:59.394 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:09:59.395 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:09:59.395 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:09:59.395 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:09:59.395 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:09:59.395 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:09:59.395 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:09:59.395 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:09:59.396 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:09:59.396 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:09:59.396 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:09:59.396 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:09:59.396 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:09:59.396 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/30046,v1:127.0.0.1:6811/30046]", 2020-11-16T20:09:59.396 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:09:59.397 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30046,v1:127.0.0.1:6817/30046]", 2020-11-16T20:09:59.397 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30046,v1:127.0.0.1:6815/30046]", 2020-11-16T20:09:59.397 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:09:59.397 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:09:59.397 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:09:59.397 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:09:59.397 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:09:59.397 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:09:59.398 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:09:59.398 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:09:59.398 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:09:59.398 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/2", 2020-11-16T20:09:59.398 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:09:59.398 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:09:59.398 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:09:59.399 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.399 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.399 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_clean_epochs": { 2020-11-16T20:09:59.399 INFO:tasks.workunit.client.0.smithi189.stdout: "min_last_epoch_clean": 0, 2020-11-16T20:09:59.399 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": { 2020-11-16T20:09:59.399 INFO:tasks.workunit.client.0.smithi189.stdout: "per_pool": [] 2020-11-16T20:09:59.399 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.399 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_epochs": [ 2020-11-16T20:09:59.400 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.400 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:09:59.400 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 35 2020-11-16T20:09:59.400 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.400 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.400 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:09:59.400 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 10 2020-11-16T20:09:59.401 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.401 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.401 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:09:59.401 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 37 2020-11-16T20:09:59.401 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.401 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.401 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.401 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_first_committed": 1, 2020-11-16T20:09:59.402 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_last_committed": 39, 2020-11-16T20:09:59.402 INFO:tasks.workunit.client.0.smithi189.stdout: "crushmap": { 2020-11-16T20:09:59.402 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": [ 2020-11-16T20:09:59.402 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.402 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:09:59.402 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.0", 2020-11-16T20:09:59.402 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:09:59.402 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.403 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.403 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:09:59.403 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.1", 2020-11-16T20:09:59.403 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:09:59.403 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.403 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.403 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:09:59.404 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.2", 2020-11-16T20:09:59.404 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:09:59.404 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.404 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.404 INFO:tasks.workunit.client.0.smithi189.stdout: "types": [ 2020-11-16T20:09:59.404 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.404 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 0, 2020-11-16T20:09:59.404 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd" 2020-11-16T20:09:59.405 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.405 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.405 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:09:59.405 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "host" 2020-11-16T20:09:59.405 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.405 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.405 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 2, 2020-11-16T20:09:59.405 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "chassis" 2020-11-16T20:09:59.406 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.406 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.406 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 3, 2020-11-16T20:09:59.406 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "rack" 2020-11-16T20:09:59.406 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.406 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.406 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 4, 2020-11-16T20:09:59.407 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "row" 2020-11-16T20:09:59.407 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.407 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.407 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 5, 2020-11-16T20:09:59.407 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pdu" 2020-11-16T20:09:59.407 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.407 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.407 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 6, 2020-11-16T20:09:59.408 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pod" 2020-11-16T20:09:59.408 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.408 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.408 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 7, 2020-11-16T20:09:59.408 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "room" 2020-11-16T20:09:59.408 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.408 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.408 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 8, 2020-11-16T20:09:59.409 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "datacenter" 2020-11-16T20:09:59.409 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.409 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.409 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 9, 2020-11-16T20:09:59.409 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "zone" 2020-11-16T20:09:59.409 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.409 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.410 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 10, 2020-11-16T20:09:59.410 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "region" 2020-11-16T20:09:59.410 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.410 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.410 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:09:59.410 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "root" 2020-11-16T20:09:59.410 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.411 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.411 INFO:tasks.workunit.client.0.smithi189.stdout: "buckets": [ 2020-11-16T20:09:59.411 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.411 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -1, 2020-11-16T20:09:59.411 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default", 2020-11-16T20:09:59.411 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:09:59.411 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:09:59.411 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:09:59.412 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:09:59.412 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:09:59.412 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:09:59.412 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.412 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:09:59.412 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:09:59.412 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:09:59.412 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.413 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.413 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.413 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.413 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -2, 2020-11-16T20:09:59.413 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default~hdd", 2020-11-16T20:09:59.413 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:09:59.413 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:09:59.414 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:09:59.414 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:09:59.414 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:09:59.414 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:09:59.414 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.414 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:09:59.414 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:09:59.414 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:09:59.415 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.415 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.415 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.415 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.415 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:09:59.415 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189", 2020-11-16T20:09:59.415 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:09:59.416 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:09:59.416 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:09:59.416 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:09:59.416 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:09:59.416 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:09:59.416 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.416 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:09:59.416 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:09:59.417 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:09:59.417 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.417 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.417 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:09:59.417 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:09:59.417 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:09:59.417 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.417 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.418 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:09:59.418 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:09:59.418 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:09:59.418 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.418 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.418 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.418 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.419 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:09:59.419 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189~hdd", 2020-11-16T20:09:59.419 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:09:59.419 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:09:59.419 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:09:59.419 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:09:59.419 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:09:59.419 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:09:59.420 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.420 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:09:59.420 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:09:59.420 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:09:59.420 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.420 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.420 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:09:59.420 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:09:59.421 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:09:59.421 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.421 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.421 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:09:59.421 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:09:59.421 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:09:59.421 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.422 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.422 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.422 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.422 INFO:tasks.workunit.client.0.smithi189.stdout: "rules": [ 2020-11-16T20:09:59.422 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.422 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 0, 2020-11-16T20:09:59.422 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "replicated_rule", 2020-11-16T20:09:59.422 INFO:tasks.workunit.client.0.smithi189.stdout: "ruleset": 0, 2020-11-16T20:09:59.423 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:09:59.423 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 1, 2020-11-16T20:09:59.423 INFO:tasks.workunit.client.0.smithi189.stdout: "max_size": 10, 2020-11-16T20:09:59.423 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2020-11-16T20:09:59.423 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.423 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2020-11-16T20:09:59.423 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2020-11-16T20:09:59.423 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2020-11-16T20:09:59.424 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.424 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.424 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_firstn", 2020-11-16T20:09:59.424 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2020-11-16T20:09:59.424 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2020-11-16T20:09:59.424 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.424 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.425 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2020-11-16T20:09:59.425 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.425 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:09:59.425 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.425 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.425 INFO:tasks.workunit.client.0.smithi189.stdout: "tunables": { 2020-11-16T20:09:59.425 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_tries": 0, 2020-11-16T20:09:59.425 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_fallback_tries": 0, 2020-11-16T20:09:59.426 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_total_tries": 50, 2020-11-16T20:09:59.426 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_descend_once": 1, 2020-11-16T20:09:59.426 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_vary_r": 1, 2020-11-16T20:09:59.426 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_stable": 1, 2020-11-16T20:09:59.426 INFO:tasks.workunit.client.0.smithi189.stdout: "straw_calc_version": 1, 2020-11-16T20:09:59.426 INFO:tasks.workunit.client.0.smithi189.stdout: "allowed_bucket_algs": 54, 2020-11-16T20:09:59.426 INFO:tasks.workunit.client.0.smithi189.stdout: "profile": "jewel", 2020-11-16T20:09:59.426 INFO:tasks.workunit.client.0.smithi189.stdout: "optimal_tunables": 1, 2020-11-16T20:09:59.427 INFO:tasks.workunit.client.0.smithi189.stdout: "legacy_tunables": 0, 2020-11-16T20:09:59.427 INFO:tasks.workunit.client.0.smithi189.stdout: "minimum_required_version": "jewel", 2020-11-16T20:09:59.427 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables": 1, 2020-11-16T20:09:59.427 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables2": 1, 2020-11-16T20:09:59.427 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v2_rules": 0, 2020-11-16T20:09:59.427 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables3": 1, 2020-11-16T20:09:59.427 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v3_rules": 0, 2020-11-16T20:09:59.428 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v4_buckets": 1, 2020-11-16T20:09:59.428 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables5": 1, 2020-11-16T20:09:59.428 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v5_rules": 0 2020-11-16T20:09:59.428 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.428 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_args": {} 2020-11-16T20:09:59.428 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.428 INFO:tasks.workunit.client.0.smithi189.stdout: "fsmap": { 2020-11-16T20:09:59.428 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:09:59.429 INFO:tasks.workunit.client.0.smithi189.stdout: "default_fscid": -1, 2020-11-16T20:09:59.429 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": { 2020-11-16T20:09:59.429 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": {}, 2020-11-16T20:09:59.429 INFO:tasks.workunit.client.0.smithi189.stdout: "ro_compat": {}, 2020-11-16T20:09:59.429 INFO:tasks.workunit.client.0.smithi189.stdout: "incompat": { 2020-11-16T20:09:59.429 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_1": "base v0.20", 2020-11-16T20:09:59.429 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_2": "client writeable ranges", 2020-11-16T20:09:59.429 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_3": "default file layouts on dirs", 2020-11-16T20:09:59.430 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_4": "dir inode in separate object", 2020-11-16T20:09:59.430 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_5": "mds uses versioned encoding", 2020-11-16T20:09:59.430 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_6": "dirfrag is stored in omap", 2020-11-16T20:09:59.430 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_8": "no anchor table", 2020-11-16T20:09:59.430 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_9": "file layout v2", 2020-11-16T20:09:59.430 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_10": "snaprealm v2" 2020-11-16T20:09:59.430 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.431 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.431 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_flags": { 2020-11-16T20:09:59.431 INFO:tasks.workunit.client.0.smithi189.stdout: "enable_multiple": false, 2020-11-16T20:09:59.431 INFO:tasks.workunit.client.0.smithi189.stdout: "ever_enabled_multiple": false 2020-11-16T20:09:59.431 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.431 INFO:tasks.workunit.client.0.smithi189.stdout: "standbys": [], 2020-11-16T20:09:59.431 INFO:tasks.workunit.client.0.smithi189.stdout: "filesystems": [] 2020-11-16T20:09:59.431 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.432 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_first_committed": 1, 2020-11-16T20:09:59.432 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_last_committed": 1, 2020-11-16T20:09:59.432 INFO:tasks.workunit.client.0.smithi189.stdout: "auth": { 2020-11-16T20:09:59.432 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:09:59.432 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 5, 2020-11-16T20:09:59.432 INFO:tasks.workunit.client.0.smithi189.stdout: "num_secrets": 3 2020-11-16T20:09:59.432 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.432 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 1, 2020-11-16T20:09:59.433 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_active": 1, 2020-11-16T20:09:59.433 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_unknown": 0, 2020-11-16T20:09:59.433 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osd": 3, 2020-11-16T20:09:59.433 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_sum": { 2020-11-16T20:09:59.433 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:09:59.433 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 61700, 2020-11-16T20:09:59.433 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 100, 2020-11-16T20:09:59.434 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 0, 2020-11-16T20:09:59.434 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 300, 2020-11-16T20:09:59.434 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:09:59.434 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:09:59.434 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 100, 2020-11-16T20:09:59.434 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:09:59.434 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:09:59.434 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 100, 2020-11-16T20:09:59.435 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:09:59.435 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:09:59.435 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:09:59.435 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 100, 2020-11-16T20:09:59.435 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 100, 2020-11-16T20:09:59.435 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:09:59.435 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:09:59.435 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:09:59.436 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:09:59.436 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:09:59.436 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:09:59.436 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:09:59.436 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:09:59.436 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:09:59.436 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:09:59.437 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:09:59.437 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:09:59.437 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:09:59.437 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:09:59.437 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:09:59.437 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:09:59.437 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:09:59.437 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:09:59.438 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:09:59.438 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:09:59.438 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:09:59.438 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:09:59.438 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:09:59.438 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:09:59.438 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:09:59.438 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.439 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:09:59.439 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:09:59.439 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:09:59.439 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:09:59.439 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 0, 2020-11-16T20:09:59.439 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 0, 2020-11-16T20:09:59.439 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:09:59.440 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:09:59.440 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:09:59.440 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:09:59.440 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:09:59.440 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.440 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 16, 2020-11-16T20:09:59.440 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 16, 2020-11-16T20:09:59.440 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 2, 2020-11-16T20:09:59.441 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 2, 2020-11-16T20:09:59.441 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 0 2020-11-16T20:09:59.441 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.441 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum": { 2020-11-16T20:09:59.441 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:09:59.441 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:09:59.441 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 3, 2020-11-16T20:09:59.441 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:09:59.442 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:09:59.442 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:09:59.442 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:09:59.442 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 18168, 2020-11-16T20:09:59.442 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2072, 2020-11-16T20:09:59.442 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:09:59.442 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 16062, 2020-11-16T20:09:59.443 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314554632, 2020-11-16T20:09:59.443 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:09:59.443 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:09:59.443 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322103943168, 2020-11-16T20:09:59.443 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:09:59.443 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2121728, 2020-11-16T20:09:59.443 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 444174, 2020-11-16T20:09:59.443 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:09:59.444 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:09:59.444 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:09:59.444 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:09:59.444 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 16447608 2020-11-16T20:09:59.444 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.444 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:09:59.444 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:09:59.444 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:09:59.445 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:09:59.445 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:09:59.445 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:09:59.445 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:09:59.445 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.445 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:09:59.445 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 0, 2020-11-16T20:09:59.446 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 0, 2020-11-16T20:09:59.446 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 0, 2020-11-16T20:09:59.446 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 0 2020-11-16T20:09:59.446 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.446 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:09:59.446 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:09:59.446 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.446 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum_by_class": { 2020-11-16T20:09:59.447 INFO:tasks.workunit.client.0.smithi189.stdout: "hdd": { 2020-11-16T20:09:59.447 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:09:59.447 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:09:59.447 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 3, 2020-11-16T20:09:59.447 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:09:59.447 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:09:59.447 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:09:59.448 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:09:59.448 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 18168, 2020-11-16T20:09:59.448 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2072, 2020-11-16T20:09:59.448 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:09:59.448 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 16062, 2020-11-16T20:09:59.448 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314554632, 2020-11-16T20:09:59.448 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:09:59.448 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:09:59.449 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322103943168, 2020-11-16T20:09:59.449 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:09:59.449 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2121728, 2020-11-16T20:09:59.449 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 444174, 2020-11-16T20:09:59.449 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:09:59.449 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:09:59.449 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:09:59.449 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:09:59.450 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 16447608 2020-11-16T20:09:59.450 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.450 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:09:59.450 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:09:59.450 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:09:59.450 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:09:59.450 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:09:59.451 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:09:59.451 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:09:59.451 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.451 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:09:59.451 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 0, 2020-11-16T20:09:59.451 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 0, 2020-11-16T20:09:59.451 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 0, 2020-11-16T20:09:59.451 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 0 2020-11-16T20:09:59.452 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.452 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:09:59.452 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:09:59.452 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.452 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.452 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_stats": [ 2020-11-16T20:09:59.452 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.453 INFO:tasks.workunit.client.0.smithi189.stdout: "poolid": 1, 2020-11-16T20:09:59.453 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 1, 2020-11-16T20:09:59.453 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:09:59.453 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 61700, 2020-11-16T20:09:59.453 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 100, 2020-11-16T20:09:59.453 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 0, 2020-11-16T20:09:59.453 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 300, 2020-11-16T20:09:59.453 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:09:59.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:09:59.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 100, 2020-11-16T20:09:59.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:09:59.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:09:59.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 100, 2020-11-16T20:09:59.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:09:59.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:09:59.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:09:59.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 100, 2020-11-16T20:09:59.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 100, 2020-11-16T20:09:59.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:09:59.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:09:59.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:09:59.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:09:59.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:09:59.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:09:59.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:09:59.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:09:59.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:09:59.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:09:59.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:09:59.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:09:59.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:09:59.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:09:59.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:09:59.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:09:59.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:09:59.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:09:59.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:09:59.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:09:59.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:09:59.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:09:59.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:09:59.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:09:59.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:09:59.458 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.458 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:09:59.458 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:09:59.459 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:09:59.459 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:09:59.459 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 1220608, 2020-11-16T20:09:59.459 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 184174, 2020-11-16T20:09:59.459 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:09:59.459 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:09:59.459 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:09:59.460 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:09:59.460 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:09:59.460 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.460 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 16, 2020-11-16T20:09:59.460 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 16, 2020-11-16T20:09:59.460 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 2, 2020-11-16T20:09:59.460 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 2, 2020-11-16T20:09:59.460 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 3 2020-11-16T20:09:59.461 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.461 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.461 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_stats": [ 2020-11-16T20:09:59.461 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.461 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:09:59.461 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 150323855371 2020-11-16T20:09:59.461 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.461 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.462 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:09:59.462 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 42949672977 2020-11-16T20:09:59.462 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.462 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.462 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:09:59.462 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 154618822666 2020-11-16T20:09:59.462 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.463 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.463 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_state": [ 2020-11-16T20:09:59.463 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.463 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "active+undersized+degraded", 2020-11-16T20:09:59.463 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 1 2020-11-16T20:09:59.463 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.463 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.463 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_osd": [ 2020-11-16T20:09:59.464 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.464 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:09:59.464 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:09:59.464 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:09:59.464 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:09:59.464 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.464 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.464 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:09:59.465 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:09:59.465 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 0, 2020-11-16T20:09:59.465 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:09:59.465 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.465 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.465 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:09:59.465 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 1, 2020-11-16T20:09:59.465 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:09:59.466 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:09:59.466 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.466 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.466 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": [ 2020-11-16T20:09:59.466 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:09:59.466 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:09:59.466 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": {} 2020-11-16T20:09:59.467 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.467 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:09:59.467 INFO:tasks.workunit.client.0.smithi189.stdout: "servicemap": { 2020-11-16T20:09:59.467 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 4, 2020-11-16T20:09:59.467 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:09:22.640633+0000", 2020-11-16T20:09:59.467 INFO:tasks.workunit.client.0.smithi189.stdout: "services": { 2020-11-16T20:09:59.467 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": { 2020-11-16T20:09:59.467 INFO:tasks.workunit.client.0.smithi189.stdout: "daemons": { 2020-11-16T20:09:59.468 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": "", 2020-11-16T20:09:59.468 INFO:tasks.workunit.client.0.smithi189.stdout: "0": { 2020-11-16T20:09:59.468 INFO:tasks.workunit.client.0.smithi189.stdout: "start_epoch": 0, 2020-11-16T20:09:59.468 INFO:tasks.workunit.client.0.smithi189.stdout: "start_stamp": "0.000000", 2020-11-16T20:09:59.468 INFO:tasks.workunit.client.0.smithi189.stdout: "gid": 0, 2020-11-16T20:09:59.468 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "(unrecognized address family 0)/0", 2020-11-16T20:09:59.468 INFO:tasks.workunit.client.0.smithi189.stdout: "metadata": {}, 2020-11-16T20:09:59.469 INFO:tasks.workunit.client.0.smithi189.stdout: "task_status": {} 2020-11-16T20:09:59.469 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.469 INFO:tasks.workunit.client.0.smithi189.stdout: "2": { 2020-11-16T20:09:59.469 INFO:tasks.workunit.client.0.smithi189.stdout: "start_epoch": 0, 2020-11-16T20:09:59.469 INFO:tasks.workunit.client.0.smithi189.stdout: "start_stamp": "0.000000", 2020-11-16T20:09:59.469 INFO:tasks.workunit.client.0.smithi189.stdout: "gid": 0, 2020-11-16T20:09:59.469 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "(unrecognized address family 0)/0", 2020-11-16T20:09:59.469 INFO:tasks.workunit.client.0.smithi189.stdout: "metadata": {}, 2020-11-16T20:09:59.470 INFO:tasks.workunit.client.0.smithi189.stdout: "task_status": {} 2020-11-16T20:09:59.470 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.470 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.470 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.470 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.470 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:09:59.470 INFO:tasks.workunit.client.0.smithi189.stdout: "paxos": { 2020-11-16T20:09:59.470 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:09:59.471 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 352, 2020-11-16T20:09:59.471 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pn": 0, 2020-11-16T20:09:59.471 INFO:tasks.workunit.client.0.smithi189.stdout: "accepted_pn": 0 2020-11-16T20:09:59.471 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:09:59.471 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:09:59.472 INFO:tasks.workunit.client.0.smithi189.stderr:report 1907271009 2020-11-16T20:09:59.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2020-11-16T20:09:59.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2020-11-16T20:09:59.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2020-11-16T20:09:59.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2020-11-16T20:09:59.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:09:59.473 INFO:tasks.workunit.client.0.smithi189.stdout:writing non-divergent object existing_4 2020-11-16T20:09:59.690 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:09:59.691 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2020-11-16T20:09:23.568948+0000 29'100 39:120 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:09:59.691 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:09:59.691 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:09:59.693 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:09:59.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2020-11-16T20:09:59.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-11-16T20:09:59.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2020-11-16T20:09:59.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:09:59.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:09:59.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:09:59.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:09:59.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2020-11-16T20:09:59.850 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:09:59.850 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:09:59.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:09:59.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:09:59.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:09:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:09:59.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:09:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:09:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:09:59.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:10:00.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:10:00.387 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:10:00.387 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:10:00.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:10:00.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:10:00.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:10:00.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855373 2020-11-16T20:10:00.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855373 2020-11-16T20:10:00.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-150323855373' 2020-11-16T20:10:00.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:10:00.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:10:00.682 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-11-16T20:10:00.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T20:10:00.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T20:10:00.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T20:10:00.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:10:00.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:10:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822668 2020-11-16T20:10:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822668 2020-11-16T20:10:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-150323855373 2-154618822668' 2020-11-16T20:10:00.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:10:00.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-150323855373 2020-11-16T20:10:00.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:10:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:10:00.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-150323855373 2020-11-16T20:10:00.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:10:00.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855373 2020-11-16T20:10:00.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 150323855373' 2020-11-16T20:10:00.855 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 150323855373 2020-11-16T20:10:00.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:10:01.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855373 -lt 150323855373 2020-11-16T20:10:01.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:10:01.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-154618822668 2020-11-16T20:10:01.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:10:01.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:10:01.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-154618822668 2020-11-16T20:10:01.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:10:01.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822668 2020-11-16T20:10:01.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 154618822668' 2020-11-16T20:10:01.262 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 154618822668 2020-11-16T20:10:01.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:10:01.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822666 -lt 154618822668 2020-11-16T20:10:01.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:10:02.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:10:02.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:10:03.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822669 -lt 154618822668 2020-11-16T20:10:03.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:10:03.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:10:03.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:10:03.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:10:03.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:10:03.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:10:03.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:10:03.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:10:03.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:10:03.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:10:03.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:10:03.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:10:03.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:10:03.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:10:03.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:10:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:10:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:10:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:10:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:10:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:10:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:10:04.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:10:04.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:10:04.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:10:04.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:10:04.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:10:04.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:10:04.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:10:04.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:10:04.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:10:04.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:10:04.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:10:04.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:10:05.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:10:05.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:10:05.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:10:05.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:10:05.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:10:05.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:10:05.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:10:05.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:10:05.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:10:05.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:10:05.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:10:05.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2020-11-16T20:10:05.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:10:05.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-11-16T20:10:06.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:10:06.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:10:06.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:10:06.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:10:06.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:10:06.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:10:06.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:10:06.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:10:06.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:10:06.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:10:06.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:10:06.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:10:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:10:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:10:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:10:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:10:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:10:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:10:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:10:06.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:10:06.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:10:07.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:10:07.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:10:07.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2020-11-16T20:10:07.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:10:07.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2020-11-16T20:10:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:10:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:10:07.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:10:07.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:10:07.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:10:07.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:10:07.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:10:07.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:10:08.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:10:08.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:10:08.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:10:08.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:10:08.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:10:08.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:10:08.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:10:08.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:10:08.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:10:08.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:10:08.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:10:08.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:10:08.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:10:09.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:10:09.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:10:09.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2020-11-16T20:10:09.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:10:09.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2020-11-16T20:10:09.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:10:09.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:10:09.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:10:09.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:10:09.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:10:09.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:10:09.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:10:09.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:10:10.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:10:10.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:10:10.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:10:10.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:10:10.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:10:10.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:10:10.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:10:10.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:10:10.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:10:10.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:10:10.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:10:10.789 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:10:10.789 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:10:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:10:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:10:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2020-11-16T20:10:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:10:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2020-11-16T20:10:12.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:10:12.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:10:12.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:10:12.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:10:12.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:10:12.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:10:12.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:10:12.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:10:13.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:10:13.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:10:13.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:10:13.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:10:13.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:10:13.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:10:13.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:10:13.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:10:13.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:10:13.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:10:13.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:10:13.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:10:13.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:10:14.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:10:14.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:10:14.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2020-11-16T20:10:14.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:10:14.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2020-11-16T20:10:17.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:10:17.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:10:17.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:10:17.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:10:17.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:10:17.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:10:17.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:10:17.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:10:17.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:10:17.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:10:17.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:10:17.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:10:18.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:10:18.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:10:18.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:10:18.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:10:18.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:10:18.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:10:18.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:10:18.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:10:18.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:10:18.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:10:18.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:10:18.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2020-11-16T20:10:18.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:10:18.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2020-11-16T20:10:25.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:10:25.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:10:25.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:10:25.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:10:25.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:10:25.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:10:25.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:10:25.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:10:25.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:10:25.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:10:25.707 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:10:25.707 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:10:26.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:10:26.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:10:26.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:10:26.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:10:26.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:10:26.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:10:26.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:10:26.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:10:26.196 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:10:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:10:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:10:26.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2020-11-16T20:10:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:10:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2020-11-16T20:10:33.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:10:33.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:10:33.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:10:33.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:10:33.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:10:33.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:10:33.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:10:33.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:10:34.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:10:34.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:10:34.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:10:34.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:10:34.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:10:34.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:10:34.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:10:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:10:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:10:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:10:34.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:10:34.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:10:34.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:10:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:10:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:10:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2020-11-16T20:10:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2020-11-16T20:10:35.774 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:10:35.774 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_fingerprint": "3c742d5f-ee9b-495b-a3f8-1f0879950819", 2020-11-16T20:10:35.774 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "16.0.0-7219-g911cc357", 2020-11-16T20:10:35.775 INFO:tasks.workunit.client.0.smithi189.stdout: "commit": "911cc35794d871071a8bebfb0919482263b1d18b", 2020-11-16T20:10:35.775 INFO:tasks.workunit.client.0.smithi189.stdout: "timestamp": "2020-11-16T20:10:35.778694+0000", 2020-11-16T20:10:35.775 INFO:tasks.workunit.client.0.smithi189.stdout: "tag": "", 2020-11-16T20:10:35.775 INFO:tasks.workunit.client.0.smithi189.stdout: "health": { 2020-11-16T20:10:35.775 INFO:tasks.workunit.client.0.smithi189.stdout: "status": "HEALTH_WARN", 2020-11-16T20:10:35.775 INFO:tasks.workunit.client.0.smithi189.stdout: "checks": { 2020-11-16T20:10:35.775 INFO:tasks.workunit.client.0.smithi189.stdout: "OSDMAP_FLAGS": { 2020-11-16T20:10:35.776 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:10:35.776 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:10:35.776 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "nodown,noout,noin flag(s) set", 2020-11-16T20:10:35.776 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 17 2020-11-16T20:10:35.776 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.776 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [], 2020-11-16T20:10:35.776 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:10:35.776 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.777 INFO:tasks.workunit.client.0.smithi189.stdout: "OSD_DOWN": { 2020-11-16T20:10:35.777 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:10:35.777 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:10:35.777 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 osds down", 2020-11-16T20:10:35.777 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:10:35.777 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.777 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:10:35.778 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.778 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "osd.1 (root=default,host=smithi189) is down" 2020-11-16T20:10:35.778 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.778 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.778 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:10:35.778 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.778 INFO:tasks.workunit.client.0.smithi189.stdout: "PG_DEGRADED": { 2020-11-16T20:10:35.779 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:10:35.779 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:10:35.779 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2020-11-16T20:10:35.779 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 2 2020-11-16T20:10:35.779 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.779 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:10:35.779 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.779 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.0 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:10:35.780 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.780 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.780 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:10:35.780 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.780 INFO:tasks.workunit.client.0.smithi189.stdout: "POOL_APP_NOT_ENABLED": { 2020-11-16T20:10:35.780 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:10:35.780 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:10:35.781 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 pool(s) do not have an application enabled", 2020-11-16T20:10:35.781 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:10:35.781 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.781 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:10:35.781 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.781 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "application not enabled on pool 'test'" 2020-11-16T20:10:35.781 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.781 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.782 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-11-16T20:10:35.782 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.782 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.782 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:10:35.782 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.782 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.782 INFO:tasks.workunit.client.0.smithi189.stdout: "mutes": [] 2020-11-16T20:10:35.783 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.783 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_first_committed": 1, 2020-11-16T20:10:35.783 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_last_committed": 1, 2020-11-16T20:10:35.783 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap": { 2020-11-16T20:10:35.783 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:10:35.783 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:10:35.783 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:08:03.739653Z", 2020-11-16T20:10:35.783 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:08:03.739653Z", 2020-11-16T20:10:35.784 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release": 16, 2020-11-16T20:10:35.784 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release_name": "pacific", 2020-11-16T20:10:35.784 INFO:tasks.workunit.client.0.smithi189.stdout: "election_strategy": 1, 2020-11-16T20:10:35.784 INFO:tasks.workunit.client.0.smithi189.stdout: "disallowed_leaders: ": "", 2020-11-16T20:10:35.784 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": false, 2020-11-16T20:10:35.784 INFO:tasks.workunit.client.0.smithi189.stdout: "features": { 2020-11-16T20:10:35.784 INFO:tasks.workunit.client.0.smithi189.stdout: "persistent": [ 2020-11-16T20:10:35.785 INFO:tasks.workunit.client.0.smithi189.stdout: "kraken", 2020-11-16T20:10:35.785 INFO:tasks.workunit.client.0.smithi189.stdout: "luminous", 2020-11-16T20:10:35.785 INFO:tasks.workunit.client.0.smithi189.stdout: "mimic", 2020-11-16T20:10:35.785 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap-prune", 2020-11-16T20:10:35.785 INFO:tasks.workunit.client.0.smithi189.stdout: "nautilus", 2020-11-16T20:10:35.785 INFO:tasks.workunit.client.0.smithi189.stdout: "octopus", 2020-11-16T20:10:35.785 INFO:tasks.workunit.client.0.smithi189.stdout: "pacific", 2020-11-16T20:10:35.785 INFO:tasks.workunit.client.0.smithi189.stdout: "elector-pinging" 2020-11-16T20:10:35.786 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.786 INFO:tasks.workunit.client.0.smithi189.stdout: "optional": [] 2020-11-16T20:10:35.786 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.786 INFO:tasks.workunit.client.0.smithi189.stdout: "mons": [ 2020-11-16T20:10:35.786 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.786 INFO:tasks.workunit.client.0.smithi189.stdout: "rank": 0, 2020-11-16T20:10:35.786 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "a", 2020-11-16T20:10:35.787 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:10:35.787 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.787 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.787 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.787 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115", 2020-11-16T20:10:35.787 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 0 2020-11-16T20:10:35.787 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.787 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.788 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.788 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115/0", 2020-11-16T20:10:35.788 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:7115/0", 2020-11-16T20:10:35.788 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 0, 2020-11-16T20:10:35.788 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 0, 2020-11-16T20:10:35.788 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_location": "{}" 2020-11-16T20:10:35.788 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.789 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.789 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.789 INFO:tasks.workunit.client.0.smithi189.stdout: "quorum": [ 2020-11-16T20:10:35.789 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2020-11-16T20:10:35.789 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.789 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap": { 2020-11-16T20:10:35.789 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 39, 2020-11-16T20:10:35.789 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:10:35.790 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:08:04.527996+0000", 2020-11-16T20:10:35.790 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:09:23.302876+0000", 2020-11-16T20:10:35.790 INFO:tasks.workunit.client.0.smithi189.stdout: "last_up_change": "2020-11-16T20:09:21.954974+0000", 2020-11-16T20:10:35.790 INFO:tasks.workunit.client.0.smithi189.stdout: "last_in_change": "2020-11-16T20:09:22.991314+0000", 2020-11-16T20:10:35.790 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-11-16T20:10:35.790 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_num": 5800384, 2020-11-16T20:10:35.790 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_set": [ 2020-11-16T20:10:35.791 INFO:tasks.workunit.client.0.smithi189.stdout: "nodown", 2020-11-16T20:10:35.791 INFO:tasks.workunit.client.0.smithi189.stdout: "noin", 2020-11-16T20:10:35.791 INFO:tasks.workunit.client.0.smithi189.stdout: "noout", 2020-11-16T20:10:35.791 INFO:tasks.workunit.client.0.smithi189.stdout: "pglog_hardlimit", 2020-11-16T20:10:35.791 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snapdirs", 2020-11-16T20:10:35.791 INFO:tasks.workunit.client.0.smithi189.stdout: "recovery_deletes", 2020-11-16T20:10:35.791 INFO:tasks.workunit.client.0.smithi189.stdout: "sortbitwise" 2020-11-16T20:10:35.791 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.792 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_version": 7, 2020-11-16T20:10:35.792 INFO:tasks.workunit.client.0.smithi189.stdout: "full_ratio": 0.99000000953674316, 2020-11-16T20:10:35.792 INFO:tasks.workunit.client.0.smithi189.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-11-16T20:10:35.792 INFO:tasks.workunit.client.0.smithi189.stdout: "nearfull_ratio": 0.85000002384185791, 2020-11-16T20:10:35.792 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_snapshot": "", 2020-11-16T20:10:35.792 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_max": 1, 2020-11-16T20:10:35.792 INFO:tasks.workunit.client.0.smithi189.stdout: "max_osd": 3, 2020-11-16T20:10:35.792 INFO:tasks.workunit.client.0.smithi189.stdout: "require_min_compat_client": "luminous", 2020-11-16T20:10:35.793 INFO:tasks.workunit.client.0.smithi189.stdout: "min_compat_client": "jewel", 2020-11-16T20:10:35.793 INFO:tasks.workunit.client.0.smithi189.stdout: "require_osd_release": "pacific", 2020-11-16T20:10:35.793 INFO:tasks.workunit.client.0.smithi189.stdout: "pools": [ 2020-11-16T20:10:35.793 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.793 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:10:35.793 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_name": "test", 2020-11-16T20:10:35.793 INFO:tasks.workunit.client.0.smithi189.stdout: "create_time": "2020-11-16T20:08:31.085395+0000", 2020-11-16T20:10:35.794 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": 1, 2020-11-16T20:10:35.794 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_names": "hashpspool", 2020-11-16T20:10:35.794 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:10:35.794 INFO:tasks.workunit.client.0.smithi189.stdout: "size": 3, 2020-11-16T20:10:35.794 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 2, 2020-11-16T20:10:35.794 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_rule": 0, 2020-11-16T20:10:35.794 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_count": 0, 2020-11-16T20:10:35.794 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_target": 0, 2020-11-16T20:10:35.795 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_barrier": 0, 2020-11-16T20:10:35.795 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2020-11-16T20:10:35.795 INFO:tasks.workunit.client.0.smithi189.stdout: "object_hash": 2, 2020-11-16T20:10:35.795 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_autoscale_mode": "off", 2020-11-16T20:10:35.795 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num": 1, 2020-11-16T20:10:35.795 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num": 1, 2020-11-16T20:10:35.795 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num_target": 1, 2020-11-16T20:10:35.796 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_target": 1, 2020-11-16T20:10:35.796 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_pending": 1, 2020-11-16T20:10:35.796 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pg_merge_meta": { 2020-11-16T20:10:35.796 INFO:tasks.workunit.client.0.smithi189.stdout: "source_pgid": "0.0", 2020-11-16T20:10:35.796 INFO:tasks.workunit.client.0.smithi189.stdout: "ready_epoch": 0, 2020-11-16T20:10:35.796 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_started": 0, 2020-11-16T20:10:35.796 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": 0, 2020-11-16T20:10:35.796 INFO:tasks.workunit.client.0.smithi189.stdout: "source_version": "0'0", 2020-11-16T20:10:35.797 INFO:tasks.workunit.client.0.smithi189.stdout: "target_version": "0'0" 2020-11-16T20:10:35.797 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.797 INFO:tasks.workunit.client.0.smithi189.stdout: "last_change": "29", 2020-11-16T20:10:35.797 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend": "0", 2020-11-16T20:10:35.797 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_prenautilus": "0", 2020-11-16T20:10:35.797 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_preluminous": "0", 2020-11-16T20:10:35.797 INFO:tasks.workunit.client.0.smithi189.stdout: "auid": 0, 2020-11-16T20:10:35.798 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_mode": "selfmanaged", 2020-11-16T20:10:35.798 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_seq": 0, 2020-11-16T20:10:35.798 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_epoch": 0, 2020-11-16T20:10:35.798 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_snaps": [], 2020-11-16T20:10:35.798 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps": "[]", 2020-11-16T20:10:35.798 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_bytes": 0, 2020-11-16T20:10:35.798 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_objects": 0, 2020-11-16T20:10:35.798 INFO:tasks.workunit.client.0.smithi189.stdout: "tiers": [], 2020-11-16T20:10:35.799 INFO:tasks.workunit.client.0.smithi189.stdout: "tier_of": -1, 2020-11-16T20:10:35.799 INFO:tasks.workunit.client.0.smithi189.stdout: "read_tier": -1, 2020-11-16T20:10:35.799 INFO:tasks.workunit.client.0.smithi189.stdout: "write_tier": -1, 2020-11-16T20:10:35.799 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_mode": "none", 2020-11-16T20:10:35.799 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_bytes": 0, 2020-11-16T20:10:35.799 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_objects": 0, 2020-11-16T20:10:35.799 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-11-16T20:10:35.800 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-11-16T20:10:35.800 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_full_ratio_micro": 800000, 2020-11-16T20:10:35.800 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_flush_age": 0, 2020-11-16T20:10:35.800 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_evict_age": 0, 2020-11-16T20:10:35.800 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profile": "", 2020-11-16T20:10:35.800 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_params": { 2020-11-16T20:10:35.800 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "none" 2020-11-16T20:10:35.800 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.801 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_period": 0, 2020-11-16T20:10:35.801 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_count": 0, 2020-11-16T20:10:35.801 INFO:tasks.workunit.client.0.smithi189.stdout: "use_gmt_hitset": true, 2020-11-16T20:10:35.801 INFO:tasks.workunit.client.0.smithi189.stdout: "min_read_recency_for_promote": 0, 2020-11-16T20:10:35.801 INFO:tasks.workunit.client.0.smithi189.stdout: "min_write_recency_for_promote": 0, 2020-11-16T20:10:35.801 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_grade_decay_rate": 0, 2020-11-16T20:10:35.801 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_search_last_n": 0, 2020-11-16T20:10:35.801 INFO:tasks.workunit.client.0.smithi189.stdout: "grade_table": [], 2020-11-16T20:10:35.802 INFO:tasks.workunit.client.0.smithi189.stdout: "stripe_width": 0, 2020-11-16T20:10:35.802 INFO:tasks.workunit.client.0.smithi189.stdout: "expected_num_objects": 0, 2020-11-16T20:10:35.802 INFO:tasks.workunit.client.0.smithi189.stdout: "fast_read": false, 2020-11-16T20:10:35.802 INFO:tasks.workunit.client.0.smithi189.stdout: "options": {}, 2020-11-16T20:10:35.802 INFO:tasks.workunit.client.0.smithi189.stdout: "application_metadata": {} 2020-11-16T20:10:35.802 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.802 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.803 INFO:tasks.workunit.client.0.smithi189.stdout: "osds": [ 2020-11-16T20:10:35.803 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.803 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:10:35.803 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "6f8f3073-dcec-495d-8b27-191eea5f2c57", 2020-11-16T20:10:35.803 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:10:35.803 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:10:35.803 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:10:35.803 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:10:35.804 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 5, 2020-11-16T20:10:35.804 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 29, 2020-11-16T20:10:35.804 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 35, 2020-11-16T20:10:35.804 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 38, 2020-11-16T20:10:35.804 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 30, 2020-11-16T20:10:35.804 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:10:35.804 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:10:35.804 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.805 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.805 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.805 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6802", 2020-11-16T20:10:35.805 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:10:35.805 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.805 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.805 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.806 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6803", 2020-11-16T20:10:35.806 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:10:35.806 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.806 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.806 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.806 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:10:35.806 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.806 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.807 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.807 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6804", 2020-11-16T20:10:35.807 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:10:35.807 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.807 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.807 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.807 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6805", 2020-11-16T20:10:35.808 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:10:35.808 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.808 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.808 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.808 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:10:35.808 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.808 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.809 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.809 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6808", 2020-11-16T20:10:35.809 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:10:35.809 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.809 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.809 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.809 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6809", 2020-11-16T20:10:35.809 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:10:35.810 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.810 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.810 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.810 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:10:35.810 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.810 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.810 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.810 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6806", 2020-11-16T20:10:35.811 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:10:35.811 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.811 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.811 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.811 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6807", 2020-11-16T20:10:35.811 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 29620 2020-11-16T20:10:35.811 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.812 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.812 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.812 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6803/29620", 2020-11-16T20:10:35.812 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6805/29620", 2020-11-16T20:10:35.812 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6809/29620", 2020-11-16T20:10:35.812 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6807/29620", 2020-11-16T20:10:35.812 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:10:35.812 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:10:35.813 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:10:35.813 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.813 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.813 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.813 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:10:35.813 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "ea764750-7154-4c9e-af61-617257854e05", 2020-11-16T20:10:35.813 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 0, 2020-11-16T20:10:35.813 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:10:35.814 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:10:35.814 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:10:35.814 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 0, 2020-11-16T20:10:35.814 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 0, 2020-11-16T20:10:35.814 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 10, 2020-11-16T20:10:35.814 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 23, 2020-11-16T20:10:35.814 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 31, 2020-11-16T20:10:35.815 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:10:35.815 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:10:35.815 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.815 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.815 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.815 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:10:35.815 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:10:35.815 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.816 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.816 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.816 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:10:35.816 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:10:35.816 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.816 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.816 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.816 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:10:35.817 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.817 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.817 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.817 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:10:35.817 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:10:35.817 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.817 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.818 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.818 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:10:35.818 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:10:35.818 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.818 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.818 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.818 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:10:35.818 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.819 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.819 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.819 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:10:35.819 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:10:35.819 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.819 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.819 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.819 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:10:35.820 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:10:35.820 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.820 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.820 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.820 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:10:35.820 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.820 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.820 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.821 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:10:35.821 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:10:35.821 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.821 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.821 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.821 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:10:35.821 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 24725 2020-11-16T20:10:35.822 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.822 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.822 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.822 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/24725", 2020-11-16T20:10:35.822 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/24725", 2020-11-16T20:10:35.822 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/24725", 2020-11-16T20:10:35.822 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/24725", 2020-11-16T20:10:35.822 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:10:35.823 INFO:tasks.workunit.client.0.smithi189.stdout: "exists" 2020-11-16T20:10:35.823 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.823 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.823 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.823 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:10:35.823 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "83927725-0d3c-4d38-9733-6d8054f1fa2b", 2020-11-16T20:10:35.823 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:10:35.824 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:10:35.824 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:10:35.824 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:10:35.824 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 15, 2020-11-16T20:10:35.824 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 29, 2020-11-16T20:10:35.824 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 36, 2020-11-16T20:10:35.824 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 0, 2020-11-16T20:10:35.824 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 32, 2020-11-16T20:10:35.825 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:10:35.825 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:10:35.825 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.825 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.825 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.825 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:10:35.825 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:10:35.825 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.826 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.826 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.826 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:10:35.826 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:10:35.826 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.826 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.826 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.826 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:10:35.827 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.827 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.827 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.827 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:10:35.827 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:10:35.827 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.827 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.828 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.828 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:10:35.828 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:10:35.828 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.828 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.828 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.828 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:10:35.828 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.829 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.829 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.829 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:10:35.829 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:10:35.829 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.829 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.829 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.829 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:10:35.830 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:10:35.830 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.830 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.830 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.830 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:10:35.830 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:10:35.830 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.831 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:10:35.831 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:10:35.831 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:10:35.831 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.831 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.831 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:10:35.831 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:10:35.831 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 30046 2020-11-16T20:10:35.832 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.832 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.832 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.832 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/30046", 2020-11-16T20:10:35.832 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/30046", 2020-11-16T20:10:35.832 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30046", 2020-11-16T20:10:35.832 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30046", 2020-11-16T20:10:35.832 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:10:35.833 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:10:35.833 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:10:35.833 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.833 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.833 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.833 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_xinfo": [ 2020-11-16T20:10:35.833 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.834 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:10:35.834 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:09:13.490423+0000", 2020-11-16T20:10:35.834 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:10:35.834 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:10:35.834 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:10:35.834 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:10:35.834 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:08:12.796468+0000", 2020-11-16T20:10:35.834 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:10:35.835 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.835 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.835 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:10:35.835 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:09:14.042030+0000", 2020-11-16T20:10:35.835 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:10:35.835 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:10:35.835 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:10:35.836 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:10:35.836 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:08:20.791257+0000", 2020-11-16T20:10:35.836 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:10:35.836 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.836 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.836 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:10:35.836 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:09:14.584643+0000", 2020-11-16T20:10:35.836 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:10:35.837 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:10:35.837 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:10:35.837 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:10:35.837 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:08:28.548492+0000", 2020-11-16T20:10:35.837 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:10:35.837 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.837 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.837 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap": [], 2020-11-16T20:10:35.838 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap_items": [], 2020-11-16T20:10:35.838 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_temp": [], 2020-11-16T20:10:35.838 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_temp": [], 2020-11-16T20:10:35.838 INFO:tasks.workunit.client.0.smithi189.stdout: "blocklist": {}, 2020-11-16T20:10:35.838 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profiles": { 2020-11-16T20:10:35.838 INFO:tasks.workunit.client.0.smithi189.stdout: "default": { 2020-11-16T20:10:35.838 INFO:tasks.workunit.client.0.smithi189.stdout: "k": "2", 2020-11-16T20:10:35.839 INFO:tasks.workunit.client.0.smithi189.stdout: "m": "2", 2020-11-16T20:10:35.839 INFO:tasks.workunit.client.0.smithi189.stdout: "plugin": "jerasure", 2020-11-16T20:10:35.839 INFO:tasks.workunit.client.0.smithi189.stdout: "technique": "reed_sol_van" 2020-11-16T20:10:35.839 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.839 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.839 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps_queue": [], 2020-11-16T20:10:35.839 INFO:tasks.workunit.client.0.smithi189.stdout: "new_removed_snaps": [], 2020-11-16T20:10:35.839 INFO:tasks.workunit.client.0.smithi189.stdout: "new_purged_snaps": [], 2020-11-16T20:10:35.840 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_node_flags": {}, 2020-11-16T20:10:35.840 INFO:tasks.workunit.client.0.smithi189.stdout: "device_class_flags": {}, 2020-11-16T20:10:35.840 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": { 2020-11-16T20:10:35.840 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_enabled": false, 2020-11-16T20:10:35.840 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_bucket_count": 0, 2020-11-16T20:10:35.840 INFO:tasks.workunit.client.0.smithi189.stdout: "degraded_stretch_mode": 0, 2020-11-16T20:10:35.840 INFO:tasks.workunit.client.0.smithi189.stdout: "recovering_stretch_mode": 0, 2020-11-16T20:10:35.840 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_bucket": 0 2020-11-16T20:10:35.841 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.841 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.841 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_metadata": [ 2020-11-16T20:10:35.841 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.841 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:10:35.841 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:10:35.841 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6804/29620,v1:127.0.0.1:6805/29620]", 2020-11-16T20:10:35.842 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:10:35.842 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:10:35.842 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:10:35.842 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:10:35.842 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:10:35.842 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:10:35.842 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:10:35.842 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:10:35.843 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:10:35.843 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-11-16T20:10:35.843 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:10:35.843 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:10:35.843 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:10:35.843 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:10:35.843 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:10:35.844 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:10:35.844 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:10:35.844 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:10:35.844 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:10:35.844 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:10:35.844 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:10:35.844 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:10:35.844 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:10:35.845 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:10:35.845 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:10:35.845 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6802/29620,v1:127.0.0.1:6803/29620]", 2020-11-16T20:10:35.845 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:10:35.845 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/29620,v1:127.0.0.1:6809/29620]", 2020-11-16T20:10:35.845 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/29620,v1:127.0.0.1:6807/29620]", 2020-11-16T20:10:35.845 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:10:35.845 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:10:35.846 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:10:35.846 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:10:35.846 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:10:35.846 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:10:35.846 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:10:35.846 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:10:35.846 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:10:35.847 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/0", 2020-11-16T20:10:35.847 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:10:35.847 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:10:35.847 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:10:35.847 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.847 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.847 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:10:35.847 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:10:35.848 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/24725,v1:127.0.0.1:6813/24725]", 2020-11-16T20:10:35.848 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:10:35.848 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:10:35.848 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:10:35.848 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:10:35.848 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:10:35.848 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:10:35.848 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:10:35.849 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:10:35.849 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:10:35.849 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-11-16T20:10:35.849 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:10:35.849 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:10:35.849 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:10:35.849 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:10:35.850 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:10:35.850 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:10:35.850 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:10:35.850 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:10:35.850 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:10:35.850 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:10:35.850 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:10:35.850 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:10:35.851 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:10:35.851 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:10:35.851 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:10:35.851 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/24725,v1:127.0.0.1:6811/24725]", 2020-11-16T20:10:35.851 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:10:35.851 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/24725,v1:127.0.0.1:6817/24725]", 2020-11-16T20:10:35.851 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/24725,v1:127.0.0.1:6815/24725]", 2020-11-16T20:10:35.851 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:10:35.852 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:10:35.852 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:10:35.852 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:10:35.852 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:10:35.852 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:10:35.852 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:10:35.852 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:10:35.853 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:10:35.853 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/1", 2020-11-16T20:10:35.853 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:10:35.853 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:10:35.853 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:10:35.853 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.853 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.853 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:10:35.854 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:10:35.854 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/30046,v1:127.0.0.1:6813/30046]", 2020-11-16T20:10:35.854 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:10:35.854 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:10:35.854 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:10:35.854 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:10:35.854 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:10:35.854 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:10:35.855 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:10:35.855 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:10:35.855 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:10:35.855 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-11-16T20:10:35.855 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:10:35.855 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:10:35.855 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:10:35.856 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:10:35.856 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:10:35.856 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:10:35.856 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:10:35.856 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:10:35.856 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:10:35.856 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:10:35.856 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:10:35.857 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:10:35.857 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:10:35.857 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:10:35.857 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:10:35.857 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/30046,v1:127.0.0.1:6811/30046]", 2020-11-16T20:10:35.857 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:10:35.857 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30046,v1:127.0.0.1:6817/30046]", 2020-11-16T20:10:35.857 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30046,v1:127.0.0.1:6815/30046]", 2020-11-16T20:10:35.858 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:10:35.858 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:10:35.858 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:10:35.858 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:10:35.858 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:10:35.858 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:10:35.858 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:10:35.858 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:10:35.859 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:10:35.859 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/2", 2020-11-16T20:10:35.859 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:10:35.859 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:10:35.859 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:10:35.859 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.859 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.860 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_clean_epochs": { 2020-11-16T20:10:35.860 INFO:tasks.workunit.client.0.smithi189.stdout: "min_last_epoch_clean": 0, 2020-11-16T20:10:35.860 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": { 2020-11-16T20:10:35.860 INFO:tasks.workunit.client.0.smithi189.stdout: "per_pool": [] 2020-11-16T20:10:35.860 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.860 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_epochs": [ 2020-11-16T20:10:35.860 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.860 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:10:35.861 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 35 2020-11-16T20:10:35.861 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.861 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.861 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:10:35.861 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 10 2020-11-16T20:10:35.861 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.861 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.862 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:10:35.862 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 37 2020-11-16T20:10:35.862 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.862 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.862 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.862 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_first_committed": 1, 2020-11-16T20:10:35.862 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_last_committed": 39, 2020-11-16T20:10:35.862 INFO:tasks.workunit.client.0.smithi189.stdout: "crushmap": { 2020-11-16T20:10:35.863 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": [ 2020-11-16T20:10:35.863 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.863 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:10:35.863 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.0", 2020-11-16T20:10:35.863 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:10:35.863 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.863 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.863 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:10:35.864 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.1", 2020-11-16T20:10:35.864 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:10:35.864 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.864 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.864 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:10:35.864 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.2", 2020-11-16T20:10:35.864 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:10:35.865 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.865 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.865 INFO:tasks.workunit.client.0.smithi189.stdout: "types": [ 2020-11-16T20:10:35.865 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.865 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 0, 2020-11-16T20:10:35.865 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd" 2020-11-16T20:10:35.865 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.865 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.866 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:10:35.866 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "host" 2020-11-16T20:10:35.866 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.866 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.866 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 2, 2020-11-16T20:10:35.866 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "chassis" 2020-11-16T20:10:35.866 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.866 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.867 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 3, 2020-11-16T20:10:35.867 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "rack" 2020-11-16T20:10:35.867 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.867 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.867 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 4, 2020-11-16T20:10:35.867 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "row" 2020-11-16T20:10:35.867 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.868 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.868 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 5, 2020-11-16T20:10:35.868 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pdu" 2020-11-16T20:10:35.868 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.868 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.868 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 6, 2020-11-16T20:10:35.868 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pod" 2020-11-16T20:10:35.868 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.869 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.869 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 7, 2020-11-16T20:10:35.869 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "room" 2020-11-16T20:10:35.869 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.869 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.869 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 8, 2020-11-16T20:10:35.869 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "datacenter" 2020-11-16T20:10:35.869 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.870 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.870 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 9, 2020-11-16T20:10:35.870 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "zone" 2020-11-16T20:10:35.870 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.870 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.870 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 10, 2020-11-16T20:10:35.870 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "region" 2020-11-16T20:10:35.870 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.871 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.871 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:10:35.871 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "root" 2020-11-16T20:10:35.871 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.871 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.871 INFO:tasks.workunit.client.0.smithi189.stdout: "buckets": [ 2020-11-16T20:10:35.871 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.871 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -1, 2020-11-16T20:10:35.872 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default", 2020-11-16T20:10:35.872 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:10:35.872 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:10:35.872 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:10:35.872 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:10:35.872 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:10:35.872 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:10:35.873 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.873 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:10:35.873 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:10:35.873 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:10:35.873 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.873 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.873 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.873 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.874 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -2, 2020-11-16T20:10:35.874 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default~hdd", 2020-11-16T20:10:35.874 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:10:35.874 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:10:35.874 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:10:35.874 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:10:35.874 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:10:35.875 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:10:35.875 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.875 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:10:35.875 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:10:35.875 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:10:35.875 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.875 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.875 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.876 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.876 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:10:35.876 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189", 2020-11-16T20:10:35.876 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:10:35.876 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:10:35.876 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:10:35.876 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:10:35.876 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:10:35.877 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:10:35.877 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.877 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:10:35.877 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:10:35.877 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:10:35.877 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.877 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.878 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:10:35.878 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:10:35.878 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:10:35.878 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.878 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.878 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:10:35.878 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:10:35.878 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:10:35.879 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.879 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.879 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.879 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.879 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:10:35.879 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189~hdd", 2020-11-16T20:10:35.879 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:10:35.879 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:10:35.880 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:10:35.880 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:10:35.880 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:10:35.880 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:10:35.880 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.880 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:10:35.880 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:10:35.880 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:10:35.881 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.881 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.881 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:10:35.881 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:10:35.881 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:10:35.881 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.881 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.882 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:10:35.882 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:10:35.882 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:10:35.882 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.882 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.882 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.882 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.882 INFO:tasks.workunit.client.0.smithi189.stdout: "rules": [ 2020-11-16T20:10:35.883 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.883 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 0, 2020-11-16T20:10:35.883 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "replicated_rule", 2020-11-16T20:10:35.883 INFO:tasks.workunit.client.0.smithi189.stdout: "ruleset": 0, 2020-11-16T20:10:35.883 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:10:35.883 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 1, 2020-11-16T20:10:35.883 INFO:tasks.workunit.client.0.smithi189.stdout: "max_size": 10, 2020-11-16T20:10:35.883 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2020-11-16T20:10:35.884 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.884 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2020-11-16T20:10:35.884 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2020-11-16T20:10:35.884 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2020-11-16T20:10:35.884 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.884 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.884 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_firstn", 2020-11-16T20:10:35.885 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2020-11-16T20:10:35.885 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2020-11-16T20:10:35.885 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.885 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.885 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2020-11-16T20:10:35.885 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.885 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:10:35.885 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.886 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.886 INFO:tasks.workunit.client.0.smithi189.stdout: "tunables": { 2020-11-16T20:10:35.886 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_tries": 0, 2020-11-16T20:10:35.886 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_fallback_tries": 0, 2020-11-16T20:10:35.886 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_total_tries": 50, 2020-11-16T20:10:35.886 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_descend_once": 1, 2020-11-16T20:10:35.886 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_vary_r": 1, 2020-11-16T20:10:35.886 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_stable": 1, 2020-11-16T20:10:35.887 INFO:tasks.workunit.client.0.smithi189.stdout: "straw_calc_version": 1, 2020-11-16T20:10:35.887 INFO:tasks.workunit.client.0.smithi189.stdout: "allowed_bucket_algs": 54, 2020-11-16T20:10:35.887 INFO:tasks.workunit.client.0.smithi189.stdout: "profile": "jewel", 2020-11-16T20:10:35.887 INFO:tasks.workunit.client.0.smithi189.stdout: "optimal_tunables": 1, 2020-11-16T20:10:35.887 INFO:tasks.workunit.client.0.smithi189.stdout: "legacy_tunables": 0, 2020-11-16T20:10:35.887 INFO:tasks.workunit.client.0.smithi189.stdout: "minimum_required_version": "jewel", 2020-11-16T20:10:35.887 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables": 1, 2020-11-16T20:10:35.888 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables2": 1, 2020-11-16T20:10:35.888 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v2_rules": 0, 2020-11-16T20:10:35.888 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables3": 1, 2020-11-16T20:10:35.888 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v3_rules": 0, 2020-11-16T20:10:35.888 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v4_buckets": 1, 2020-11-16T20:10:35.888 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables5": 1, 2020-11-16T20:10:35.888 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v5_rules": 0 2020-11-16T20:10:35.888 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.889 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_args": {} 2020-11-16T20:10:35.889 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.889 INFO:tasks.workunit.client.0.smithi189.stdout: "fsmap": { 2020-11-16T20:10:35.889 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:10:35.889 INFO:tasks.workunit.client.0.smithi189.stdout: "default_fscid": -1, 2020-11-16T20:10:35.889 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": { 2020-11-16T20:10:35.889 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": {}, 2020-11-16T20:10:35.889 INFO:tasks.workunit.client.0.smithi189.stdout: "ro_compat": {}, 2020-11-16T20:10:35.890 INFO:tasks.workunit.client.0.smithi189.stdout: "incompat": { 2020-11-16T20:10:35.890 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_1": "base v0.20", 2020-11-16T20:10:35.890 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_2": "client writeable ranges", 2020-11-16T20:10:35.890 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_3": "default file layouts on dirs", 2020-11-16T20:10:35.890 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_4": "dir inode in separate object", 2020-11-16T20:10:35.890 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_5": "mds uses versioned encoding", 2020-11-16T20:10:35.890 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_6": "dirfrag is stored in omap", 2020-11-16T20:10:35.891 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_8": "no anchor table", 2020-11-16T20:10:35.891 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_9": "file layout v2", 2020-11-16T20:10:35.891 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_10": "snaprealm v2" 2020-11-16T20:10:35.891 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.891 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.891 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_flags": { 2020-11-16T20:10:35.891 INFO:tasks.workunit.client.0.smithi189.stdout: "enable_multiple": false, 2020-11-16T20:10:35.891 INFO:tasks.workunit.client.0.smithi189.stdout: "ever_enabled_multiple": false 2020-11-16T20:10:35.892 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.892 INFO:tasks.workunit.client.0.smithi189.stdout: "standbys": [], 2020-11-16T20:10:35.892 INFO:tasks.workunit.client.0.smithi189.stdout: "filesystems": [] 2020-11-16T20:10:35.892 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.892 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_first_committed": 1, 2020-11-16T20:10:35.892 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_last_committed": 1, 2020-11-16T20:10:35.892 INFO:tasks.workunit.client.0.smithi189.stdout: "auth": { 2020-11-16T20:10:35.892 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:10:35.893 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 5, 2020-11-16T20:10:35.893 INFO:tasks.workunit.client.0.smithi189.stdout: "num_secrets": 3 2020-11-16T20:10:35.893 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.893 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 1, 2020-11-16T20:10:35.893 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_active": 1, 2020-11-16T20:10:35.893 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_unknown": 0, 2020-11-16T20:10:35.893 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osd": 3, 2020-11-16T20:10:35.894 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_sum": { 2020-11-16T20:10:35.894 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:10:35.894 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 61854, 2020-11-16T20:10:35.894 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 100, 2020-11-16T20:10:35.894 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 0, 2020-11-16T20:10:35.894 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 300, 2020-11-16T20:10:35.894 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:10:35.894 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:10:35.895 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 100, 2020-11-16T20:10:35.895 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:10:35.895 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:10:35.895 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 100, 2020-11-16T20:10:35.895 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:10:35.895 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:10:35.895 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:10:35.896 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 101, 2020-11-16T20:10:35.896 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 101, 2020-11-16T20:10:35.896 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:10:35.896 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:10:35.896 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:10:35.896 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:10:35.896 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:10:35.896 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:10:35.897 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:10:35.897 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:10:35.897 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:10:35.897 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:10:35.897 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:10:35.897 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:10:35.897 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:10:35.897 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:10:35.898 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:10:35.898 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:10:35.898 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:10:35.898 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:10:35.898 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:10:35.898 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:10:35.898 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:10:35.899 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:10:35.899 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:10:35.899 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:10:35.899 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:10:35.899 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.899 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:10:35.899 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:10:35.899 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:10:35.900 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:10:35.900 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 0, 2020-11-16T20:10:35.900 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 0, 2020-11-16T20:10:35.900 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:10:35.900 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:10:35.900 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:10:35.900 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:10:35.900 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:10:35.901 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.901 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 17, 2020-11-16T20:10:35.901 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 17, 2020-11-16T20:10:35.901 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 2, 2020-11-16T20:10:35.901 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 2, 2020-11-16T20:10:35.901 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 0 2020-11-16T20:10:35.901 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.902 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum": { 2020-11-16T20:10:35.902 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:10:35.902 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:10:35.902 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 3, 2020-11-16T20:10:35.902 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:10:35.902 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:10:35.902 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:10:35.902 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:10:35.903 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 18168, 2020-11-16T20:10:35.903 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2072, 2020-11-16T20:10:35.903 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:10:35.903 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 16062, 2020-11-16T20:10:35.903 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314554632, 2020-11-16T20:10:35.903 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:10:35.903 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:10:35.903 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322103943168, 2020-11-16T20:10:35.904 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:10:35.904 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2121728, 2020-11-16T20:10:35.904 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 444482, 2020-11-16T20:10:35.904 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:10:35.904 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:10:35.904 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:10:35.904 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:10:35.905 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 16447608 2020-11-16T20:10:35.905 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.905 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:10:35.905 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:10:35.905 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:10:35.905 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:10:35.905 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:10:35.905 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:10:35.906 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:10:35.906 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.906 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:10:35.906 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 0, 2020-11-16T20:10:35.906 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 0, 2020-11-16T20:10:35.906 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 0, 2020-11-16T20:10:35.906 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 0 2020-11-16T20:10:35.906 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.907 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:10:35.907 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:10:35.907 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.907 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum_by_class": { 2020-11-16T20:10:35.907 INFO:tasks.workunit.client.0.smithi189.stdout: "hdd": { 2020-11-16T20:10:35.907 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:10:35.907 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:10:35.908 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 3, 2020-11-16T20:10:35.908 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:10:35.908 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:10:35.908 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:10:35.908 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:10:35.908 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 18168, 2020-11-16T20:10:35.908 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2072, 2020-11-16T20:10:35.908 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:10:35.909 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 16062, 2020-11-16T20:10:35.909 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314554632, 2020-11-16T20:10:35.909 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:10:35.909 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:10:35.909 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322103943168, 2020-11-16T20:10:35.909 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:10:35.909 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2121728, 2020-11-16T20:10:35.909 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 444482, 2020-11-16T20:10:35.910 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:10:35.910 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:10:35.910 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:10:35.910 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:10:35.910 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 16447608 2020-11-16T20:10:35.910 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.910 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:10:35.910 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:10:35.911 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:10:35.911 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:10:35.911 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:10:35.911 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:10:35.911 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:10:35.911 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.911 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:10:35.912 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 0, 2020-11-16T20:10:35.912 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 0, 2020-11-16T20:10:35.912 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 0, 2020-11-16T20:10:35.912 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 0 2020-11-16T20:10:35.912 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.912 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:10:35.912 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:10:35.912 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.913 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.913 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_stats": [ 2020-11-16T20:10:35.913 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.913 INFO:tasks.workunit.client.0.smithi189.stdout: "poolid": 1, 2020-11-16T20:10:35.913 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 1, 2020-11-16T20:10:35.913 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:10:35.913 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 61854, 2020-11-16T20:10:35.913 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 100, 2020-11-16T20:10:35.914 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 0, 2020-11-16T20:10:35.914 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 300, 2020-11-16T20:10:35.914 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:10:35.914 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:10:35.914 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 100, 2020-11-16T20:10:35.914 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:10:35.914 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:10:35.915 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 100, 2020-11-16T20:10:35.915 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:10:35.915 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:10:35.915 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:10:35.915 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 101, 2020-11-16T20:10:35.915 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 101, 2020-11-16T20:10:35.915 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:10:35.915 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:10:35.916 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:10:35.916 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:10:35.916 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:10:35.916 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:10:35.916 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:10:35.916 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:10:35.916 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:10:35.916 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:10:35.917 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:10:35.917 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:10:35.917 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:10:35.917 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:10:35.917 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:10:35.917 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:10:35.917 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:10:35.917 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:10:35.918 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:10:35.918 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:10:35.918 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:10:35.918 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:10:35.918 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:10:35.918 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:10:35.918 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:10:35.918 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.919 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:10:35.919 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:10:35.919 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:10:35.919 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:10:35.919 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 1220608, 2020-11-16T20:10:35.919 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 184482, 2020-11-16T20:10:35.919 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:10:35.920 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:10:35.920 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:10:35.920 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:10:35.920 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:10:35.920 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.920 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 17, 2020-11-16T20:10:35.920 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 17, 2020-11-16T20:10:35.920 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 2, 2020-11-16T20:10:35.921 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 2, 2020-11-16T20:10:35.921 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 3 2020-11-16T20:10:35.921 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.921 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.921 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_stats": [ 2020-11-16T20:10:35.921 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.921 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:10:35.921 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 150323855380 2020-11-16T20:10:35.922 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.922 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.922 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:10:35.922 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 42949672977 2020-11-16T20:10:35.922 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.922 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.922 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:10:35.923 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 154618822675 2020-11-16T20:10:35.923 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.923 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.923 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_state": [ 2020-11-16T20:10:35.923 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.923 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "active+undersized+degraded", 2020-11-16T20:10:35.923 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 1 2020-11-16T20:10:35.923 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.924 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.924 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_osd": [ 2020-11-16T20:10:35.924 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.924 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:10:35.924 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:10:35.924 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:10:35.924 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:10:35.924 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.925 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.925 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:10:35.925 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:10:35.925 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 0, 2020-11-16T20:10:35.925 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:10:35.925 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.925 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.925 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:10:35.926 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 1, 2020-11-16T20:10:35.926 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:10:35.926 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:10:35.926 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.926 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.926 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": [ 2020-11-16T20:10:35.926 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:10:35.927 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:10:35.927 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": {} 2020-11-16T20:10:35.927 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.927 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:10:35.927 INFO:tasks.workunit.client.0.smithi189.stdout: "servicemap": { 2020-11-16T20:10:35.927 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 6, 2020-11-16T20:10:35.927 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:10:22.657862+0000", 2020-11-16T20:10:35.927 INFO:tasks.workunit.client.0.smithi189.stdout: "services": {} 2020-11-16T20:10:35.928 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:10:35.928 INFO:tasks.workunit.client.0.smithi189.stdout: "paxos": { 2020-11-16T20:10:35.928 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:10:35.928 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 447, 2020-11-16T20:10:35.928 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pn": 0, 2020-11-16T20:10:35.928 INFO:tasks.workunit.client.0.smithi189.stdout: "accepted_pn": 0 2020-11-16T20:10:35.928 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:10:35.928 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:10:35.929 INFO:tasks.workunit.client.0.smithi189.stderr:report 3796355393 2020-11-16T20:10:35.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2020-11-16T20:10:35.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2020-11-16T20:10:35.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:10:35.930 INFO:tasks.workunit.client.0.smithi189.stdout:delay recovery 2020-11-16T20:10:36.158 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:10:36.159 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 100 0 0 61854 0 0 17 17 active+undersized+degraded 2020-11-16T20:09:23.568948+0000 39'101 39:121 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:10:36.159 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:10:36.159 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:10:36.162 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:10:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-11-16T20:10:36.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2020-11-16T20:10:36.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T20:10:36.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T20:10:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:10:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:10:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:10:36.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.0.asok 2020-11-16T20:10:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-11-16T20:10:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.0.asok set_recovery_delay 100000 2020-11-16T20:10:36.278 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:10:36.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-11-16T20:10:36.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2020-11-16T20:10:36.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-11-16T20:10:36.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-11-16T20:10:36.288 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:10:36.288 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:10:36.288 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:10:36.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.2.asok 2020-11-16T20:10:36.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-11-16T20:10:36.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.2.asok set_recovery_delay 100000 2020-11-16T20:10:36.389 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:10:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2020-11-16T20:10:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:10:36.397 INFO:tasks.workunit.client.0.smithi189.stdout:revive divergent 1 2020-11-16T20:10:36.745 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:10:36.746 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 100 0 0 61854 0 0 17 17 active+undersized+degraded 2020-11-16T20:09:23.568948+0000 39'101 39:121 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:10:36.746 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:10:36.746 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:10:36.748 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:10:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2020-11-16T20:10:37.441 INFO:tasks.workunit.client.0.smithi189.stderr:noup is set 2020-11-16T20:10:37.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-11-16T20:10:37.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:10:37.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:10:37.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T20:10:37.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:10:37.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:10:37.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:10:37.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:10:37.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:10:37.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:10:37.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:10:37.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:10:37.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:10:37.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:10:37.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:10:37.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:10:37.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:10:37.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:10:37.458 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:10:37.458 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:10:37.459 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:10:37.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:10:37.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:10:37.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:10:37.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:10:37.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:10:37.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:10:37.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:10:37.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:10:37.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:10:37.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:10:37.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:10:37.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T20:10:37.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:10:37.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2020-11-16T20:10:37.461 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:10:37.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T20:10:37.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:10:37.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:10:37.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:10:37.492 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:10:37.495+0000 7f2abf519d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:10:37.494 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:10:37.499+0000 7f2abf519d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:10:37.495 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:10:37.499+0000 7f2abf519d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:10:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2020-11-16T20:10:37.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2020-11-16T20:10:38.028 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:10:38.031+0000 7f2abf519d80 -1 Falling back to public interface 2020-11-16T20:10:39.510 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:10:39.511+0000 7f2abf519d80 -1 osd.1 29 log_to_monitors {default=true} 2020-11-16T20:10:42.848 INFO:tasks.workunit.client.0.smithi189.stdout:delay recovery divergent 2020-11-16T20:10:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2020-11-16T20:10:42.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:10:43.280 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:10:43.281 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 100 0 0 61854 0 0 17 17 active+undersized+degraded 2020-11-16T20:09:23.568948+0000 39'101 39:122 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:10:43.281 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:10:43.281 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:10:43.282 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:10:43.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2020-11-16T20:10:43.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:10:43.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:10:43.299 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:10:43.299 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:10:43.300 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:10:43.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.1.asok 2020-11-16T20:10:43.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2020-11-16T20:10:43.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.1.asok set_recovery_delay 100000 2020-11-16T20:10:43.400 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:10:43.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2020-11-16T20:10:44.067 INFO:tasks.workunit.client.0.smithi189.stderr:noup is unset 2020-11-16T20:10:44.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2020-11-16T20:10:44.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:10:44.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:10:44.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:10:44.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:10:44.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:10:44.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:10:44.081 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:10:44.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:10:44.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:10:44.519 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 35 up_thru 38 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/29620,v1:127.0.0.1:6803/29620] [v2:127.0.0.1:6804/29620,v1:127.0.0.1:6805/29620] exists,up 6f8f3073-dcec-495d-8b27-191eea5f2c57 2020-11-16T20:10:44.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:10:44.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:10:44.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:10:44.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2020-11-16T20:10:44.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:10:44.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:10:44.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:10:44.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:10:44.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:10:44.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:10:44.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:10:44.522 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:10:44.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:10:44.915 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/32570,v1:127.0.0.1:6819/32570] [v2:127.0.0.1:6820/32570,v1:127.0.0.1:6821/32570] exists,up ea764750-7154-4c9e-af61-617257854e05 2020-11-16T20:10:44.915 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:10:44.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:10:44.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:10:44.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:10:44.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2020-11-16T20:10:44.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:10:44.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:10:44.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:10:44.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:10:44.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:10:44.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:10:44.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:10:44.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:10:45.309 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/30046,v1:127.0.0.1:6811/30046] [v2:127.0.0.1:6812/30046,v1:127.0.0.1:6813/30046] exists,up 83927725-0d3c-4d38-9733-6d8054f1fa2b 2020-11-16T20:10:45.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:10:45.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:10:45.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:10:45.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:10:45.682 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:10:45.682 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 60774 0 0 16 16 peering 2020-11-16T20:10:44.206006+0000 29'106 43:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:10:45.682 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:10:45.682 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:10:45.684 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:10:45.694 INFO:tasks.workunit.client.0.smithi189.stdout:wait for peering 2020-11-16T20:10:45.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2020-11-16T20:10:45.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:10:46.074 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:10:46.074 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 60774 0 0 16 16 peering 2020-11-16T20:10:44.206006+0000 29'106 43:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:10:46.075 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:10:46.075 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:10:46.076 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:10:46.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2020-11-16T20:10:59.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2020-11-16T20:10:59.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:10:59.679 INFO:tasks.workunit.client.0.smithi189.stdout:killing divergent 1 2020-11-16T20:11:00.055 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:11:00.056 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 4 4 0 0 61854 0 0 11 11 active+recovering+degraded+wait 2020-11-16T20:10:44.771948+0000 39'101 44:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:11:00.056 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:11:00.056 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:11:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:11:00.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2020-11-16T20:11:00.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:11:00.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:11:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:11:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:11:00.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:11:00.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:11:00.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.12850.out 2020-11-16T20:11:00.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.12850.out 2020-11-16T20:11:00.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-11-16T20:11:00.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T20:11:00.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-11-16T20:11:00.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T20:11:00.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-11-16T20:11:00.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.12850.out 2020-11-16T20:11:01.027 INFO:tasks.workunit.client.0.smithi189.stderr:Exporting 1.0 info 1.0( v 44'102 lc 0'0 (29'90,44'102] local-lis/les=43/44 n=101 ec=23/23 lis/c=43/23 les/c/f=44/24/0 sis=43 pruub=14.803615508s) 2020-11-16T20:11:01.045 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:023dd3d2:::existing_12:head# 2020-11-16T20:11:01.046 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:04c40a0c:::existing_93:head# 2020-11-16T20:11:01.049 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:064ff5fb:::existing_36:head# 2020-11-16T20:11:01.050 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:091f9f1d:::existing_94:head# 2020-11-16T20:11:01.050 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0a865a93:::existing_54:head# 2020-11-16T20:11:01.051 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0fed513c:::existing_41:head# 2020-11-16T20:11:01.052 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:13b9e0b8:::existing_92:head# 2020-11-16T20:11:01.053 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:169157fa:::existing_81:head# 2020-11-16T20:11:01.054 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:17731382:::existing_58:head# 2020-11-16T20:11:01.055 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:18a45a39:::existing_43:head# 2020-11-16T20:11:01.055 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1c160f01:::existing_66:head# 2020-11-16T20:11:01.056 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1dd6c249:::existing_97:head# 2020-11-16T20:11:01.057 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:227e0c77:::existing_7:head# 2020-11-16T20:11:01.057 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:23e1b063:::existing_59:head# 2020-11-16T20:11:01.057 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:28566edb:::existing_78:head# 2020-11-16T20:11:01.058 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2a101182:::existing_20:head# 2020-11-16T20:11:01.059 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2d1519da:::existing_96:head# 2020-11-16T20:11:01.059 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3078b87a:::existing_86:head# 2020-11-16T20:11:01.059 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:30bde01b:::existing_15:head# 2020-11-16T20:11:01.060 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:317a7e19:::existing_56:head# 2020-11-16T20:11:01.060 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:38b9c7fa:::existing_33:head# 2020-11-16T20:11:01.061 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3e16d84b:::existing_48:head# 2020-11-16T20:11:01.061 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:45fe8203:::existing_75:head# 2020-11-16T20:11:01.062 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4e82c9ba:::existing_50:head# 2020-11-16T20:11:01.062 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5298dcdb:::existing_57:head# 2020-11-16T20:11:01.062 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:52ebaa44:::existing_32:head# 2020-11-16T20:11:01.063 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:53588912:::existing_9:head# 2020-11-16T20:11:01.063 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5b7e2de6:::existing_84:head# 2020-11-16T20:11:01.064 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5da0255c:::existing_13:head# 2020-11-16T20:11:01.064 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5da2dab8:::existing_53:head# 2020-11-16T20:11:01.064 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5ee3514b:::existing_30:head# 2020-11-16T20:11:01.064 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:602f83fe:::foo:head# 2020-11-16T20:11:01.074 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:60426cc7:::existing_62:head# 2020-11-16T20:11:01.075 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:628918ca:::existing_79:head# 2020-11-16T20:11:01.076 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:63f8ab2d:::existing_77:head# 2020-11-16T20:11:01.077 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:67e1780f:::existing_27:head# 2020-11-16T20:11:01.079 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:68143ca1:::existing_71:head# 2020-11-16T20:11:01.081 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6bac9509:::existing_29:head# 2020-11-16T20:11:01.081 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6bbec798:::existing_45:head# 2020-11-16T20:11:01.082 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6cb23a40:::existing_65:head# 2020-11-16T20:11:01.082 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7055ec5e:::existing_83:head# 2020-11-16T20:11:01.082 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:70a54e2b:::existing_87:head# 2020-11-16T20:11:01.083 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:750ace21:::existing_19:head# 2020-11-16T20:11:01.083 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:78623b1d:::existing_6:head# 2020-11-16T20:11:01.084 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:79881152:::existing_63:head# 2020-11-16T20:11:01.084 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7a80c0d5:::existing_99:head# 2020-11-16T20:11:01.084 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:81490677:::existing_76:head# 2020-11-16T20:11:01.085 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:85ec8e5d:::existing_85:head# 2020-11-16T20:11:01.085 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8e2abbf0:::existing_49:head# 2020-11-16T20:11:01.086 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8ee0fe0f:::existing_52:head# 2020-11-16T20:11:01.086 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:923c5d4d:::existing_5:head# 2020-11-16T20:11:01.086 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:92731e71:::existing_89:head# 2020-11-16T20:11:01.087 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:928205cf:::existing_82:head# 2020-11-16T20:11:01.087 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:93be2267:::existing_24:head# 2020-11-16T20:11:01.087 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:99ceda2c:::existing_44:head# 2020-11-16T20:11:01.088 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9dd917ee:::existing_26:head# 2020-11-16T20:11:01.088 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a28805a6:::existing_72:head# 2020-11-16T20:11:01.088 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a294a8f6:::existing_16:head# 2020-11-16T20:11:01.089 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a2f5473a:::existing_42:head# 2020-11-16T20:11:01.089 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a504a533:::existing_90:head# 2020-11-16T20:11:01.090 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a691337e:::existing_61:head# 2020-11-16T20:11:01.090 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a6e8d7ac:::existing_28:head# 2020-11-16T20:11:01.090 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a9fde86e:::existing_18:head# 2020-11-16T20:11:01.091 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:aa309faa:::existing_68:head# 2020-11-16T20:11:01.091 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ae40b0ed:::existing_31:head# 2020-11-16T20:11:01.091 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b279b1fe:::existing_10:head# 2020-11-16T20:11:01.092 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b343e277:::existing_88:head# 2020-11-16T20:11:01.092 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b7130e90:::existing_80:head# 2020-11-16T20:11:01.092 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:bc6f7309:::existing_70:head# 2020-11-16T20:11:01.093 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:be8489b7:::existing_38:head# 2020-11-16T20:11:01.093 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:beb4e5e3:::existing_47:head# 2020-11-16T20:11:01.094 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:bebec877:::existing_74:head# 2020-11-16T20:11:01.094 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c04347d8:::existing_60:head# 2020-11-16T20:11:01.094 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c156033e:::existing_100:head# 2020-11-16T20:11:01.094 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c19f5a8e:::existing_8:head# 2020-11-16T20:11:01.095 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c3e6ebe4:::existing_17:head# 2020-11-16T20:11:01.095 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c6c5a7c2:::existing_39:head# 2020-11-16T20:11:01.096 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cc1d0abf:::existing_95:head# 2020-11-16T20:11:01.096 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ccb3865b:::existing_73:head# 2020-11-16T20:11:01.096 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cd84e19e:::existing_40:head# 2020-11-16T20:11:01.097 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d2cd64ce:::existing_37:head# 2020-11-16T20:11:01.097 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d41a25c9:::existing_22:head# 2020-11-16T20:11:01.097 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d7f088da:::existing_91:head# 2020-11-16T20:11:01.097 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d9cbd1d6:::existing_64:head# 2020-11-16T20:11:01.098 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:dcc4ff82:::existing_98:head# 2020-11-16T20:11:01.098 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:dd677bd3:::existing_35:head# 2020-11-16T20:11:01.098 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:dd6d90e7:::existing_46:head# 2020-11-16T20:11:01.099 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e0e22077:::existing_21:head# 2020-11-16T20:11:01.099 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e255b710:::existing_55:head# 2020-11-16T20:11:01.099 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:eff1d5db:::existing_14:head# 2020-11-16T20:11:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f1b156bf:::existing_23:head# 2020-11-16T20:11:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f5ce4199:::existing_34:head# 2020-11-16T20:11:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f6809456:::existing_11:head# 2020-11-16T20:11:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f699a2a9:::existing_67:head# 2020-11-16T20:11:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f9baac43:::existing_25:head# 2020-11-16T20:11:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:fb76688e:::existing_51:head# 2020-11-16T20:11:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:fbb92eec:::existing_69:head# 2020-11-16T20:11:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:Export successful 2020-11-16T20:11:01.102 INFO:tasks.workunit.client.0.smithi189.stdout: marking collection for removal 2020-11-16T20:11:01.102 INFO:tasks.workunit.client.0.smithi189.stdout:setting '_remove' omap key 2020-11-16T20:11:01.102 INFO:tasks.workunit.client.0.smithi189.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-11-16T20:11:01.336 INFO:tasks.workunit.client.0.smithi189.stderr:Remove successful 2020-11-16T20:11:01.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.12850.out 2020-11-16T20:11:01.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-11-16T20:11:01.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T20:11:01.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-11-16T20:11:01.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T20:11:01.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-11-16T20:11:01.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.12850.out 2020-11-16T20:11:02.334 INFO:tasks.workunit.client.0.smithi189.stdout:get_pg_num_history pg_num_history pg_num_history(e44 pg_nums {1={23=1}} deleted_pools ) 2020-11-16T20:11:02.334 INFO:tasks.workunit.client.0.smithi189.stdout:Importing pgid 1.0 2020-11-16T20:11:02.334 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:023dd3d2:::existing_12:head# 2020-11-16T20:11:02.335 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.438 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:04c40a0c:::existing_93:head# 2020-11-16T20:11:02.438 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.463 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:064ff5fb:::existing_36:head# 2020-11-16T20:11:02.463 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.488 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:091f9f1d:::existing_94:head# 2020-11-16T20:11:02.489 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.522 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0a865a93:::existing_54:head# 2020-11-16T20:11:02.522 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.555 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0fed513c:::existing_41:head# 2020-11-16T20:11:02.555 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.589 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:13b9e0b8:::existing_92:head# 2020-11-16T20:11:02.589 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.622 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:169157fa:::existing_81:head# 2020-11-16T20:11:02.622 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.656 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:17731382:::existing_58:head# 2020-11-16T20:11:02.656 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.689 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:18a45a39:::existing_43:head# 2020-11-16T20:11:02.690 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.723 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1c160f01:::existing_66:head# 2020-11-16T20:11:02.723 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.815 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1dd6c249:::existing_97:head# 2020-11-16T20:11:02.815 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.848 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:227e0c77:::existing_7:head# 2020-11-16T20:11:02.848 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.881 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:23e1b063:::existing_59:head# 2020-11-16T20:11:02.882 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.915 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:28566edb:::existing_78:head# 2020-11-16T20:11:02.916 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.949 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2a101182:::existing_20:head# 2020-11-16T20:11:02.949 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:02.982 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2d1519da:::existing_96:head# 2020-11-16T20:11:02.982 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.015 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3078b87a:::existing_86:head# 2020-11-16T20:11:03.016 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.049 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:30bde01b:::existing_15:head# 2020-11-16T20:11:03.049 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.082 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:317a7e19:::existing_56:head# 2020-11-16T20:11:03.082 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.116 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:38b9c7fa:::existing_33:head# 2020-11-16T20:11:03.116 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.157 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3e16d84b:::existing_48:head# 2020-11-16T20:11:03.158 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.216 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:45fe8203:::existing_75:head# 2020-11-16T20:11:03.216 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.250 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4e82c9ba:::existing_50:head# 2020-11-16T20:11:03.250 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.283 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5298dcdb:::existing_57:head# 2020-11-16T20:11:03.283 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.317 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:52ebaa44:::existing_32:head# 2020-11-16T20:11:03.317 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.392 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:53588912:::existing_9:head# 2020-11-16T20:11:03.392 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.425 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5b7e2de6:::existing_84:head# 2020-11-16T20:11:03.426 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.459 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5da0255c:::existing_13:head# 2020-11-16T20:11:03.459 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.493 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5da2dab8:::existing_53:head# 2020-11-16T20:11:03.493 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.526 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5ee3514b:::existing_30:head# 2020-11-16T20:11:03.526 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.568 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:602f83fe:::foo:head# 2020-11-16T20:11:03.568 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.602 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:60426cc7:::existing_62:head# 2020-11-16T20:11:03.603 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.635 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:628918ca:::existing_79:head# 2020-11-16T20:11:03.635 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.668 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:63f8ab2d:::existing_77:head# 2020-11-16T20:11:03.668 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.702 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:67e1780f:::existing_27:head# 2020-11-16T20:11:03.702 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.743 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:68143ca1:::existing_71:head# 2020-11-16T20:11:03.744 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.785 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6bac9509:::existing_29:head# 2020-11-16T20:11:03.785 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.827 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6bbec798:::existing_45:head# 2020-11-16T20:11:03.827 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.869 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6cb23a40:::existing_65:head# 2020-11-16T20:11:03.869 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.911 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7055ec5e:::existing_83:head# 2020-11-16T20:11:03.911 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:03.980 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:70a54e2b:::existing_87:head# 2020-11-16T20:11:03.980 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.027 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:750ace21:::existing_19:head# 2020-11-16T20:11:04.028 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.069 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:78623b1d:::existing_6:head# 2020-11-16T20:11:04.070 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.111 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:79881152:::existing_63:head# 2020-11-16T20:11:04.111 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.153 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7a80c0d5:::existing_99:head# 2020-11-16T20:11:04.153 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.186 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:81490677:::existing_76:head# 2020-11-16T20:11:04.187 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.220 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:85ec8e5d:::existing_85:head# 2020-11-16T20:11:04.220 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.253 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8e2abbf0:::existing_49:head# 2020-11-16T20:11:04.254 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.287 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8ee0fe0f:::existing_52:head# 2020-11-16T20:11:04.287 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.320 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:923c5d4d:::existing_5:head# 2020-11-16T20:11:04.321 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.353 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:92731e71:::existing_89:head# 2020-11-16T20:11:04.354 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.387 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:928205cf:::existing_82:head# 2020-11-16T20:11:04.387 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.421 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:93be2267:::existing_24:head# 2020-11-16T20:11:04.421 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.454 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:99ceda2c:::existing_44:head# 2020-11-16T20:11:04.454 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.487 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9dd917ee:::existing_26:head# 2020-11-16T20:11:04.488 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.521 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a28805a6:::existing_72:head# 2020-11-16T20:11:04.521 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.584 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a294a8f6:::existing_16:head# 2020-11-16T20:11:04.584 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.621 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a2f5473a:::existing_42:head# 2020-11-16T20:11:04.621 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.655 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a504a533:::existing_90:head# 2020-11-16T20:11:04.655 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.688 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a691337e:::existing_61:head# 2020-11-16T20:11:04.688 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.722 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a6e8d7ac:::existing_28:head# 2020-11-16T20:11:04.722 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.780 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a9fde86e:::existing_18:head# 2020-11-16T20:11:04.780 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.805 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:aa309faa:::existing_68:head# 2020-11-16T20:11:04.806 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.830 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ae40b0ed:::existing_31:head# 2020-11-16T20:11:04.831 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.889 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b279b1fe:::existing_10:head# 2020-11-16T20:11:04.889 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.914 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b343e277:::existing_88:head# 2020-11-16T20:11:04.914 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.940 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b7130e90:::existing_80:head# 2020-11-16T20:11:04.940 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.964 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:bc6f7309:::existing_70:head# 2020-11-16T20:11:04.965 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:04.989 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:be8489b7:::existing_38:head# 2020-11-16T20:11:04.990 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.023 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:beb4e5e3:::existing_47:head# 2020-11-16T20:11:05.023 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.056 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:bebec877:::existing_74:head# 2020-11-16T20:11:05.057 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.090 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c04347d8:::existing_60:head# 2020-11-16T20:11:05.090 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.155 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c156033e:::existing_100:head# 2020-11-16T20:11:05.155 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.240 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c19f5a8e:::existing_8:head# 2020-11-16T20:11:05.241 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.299 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c3e6ebe4:::existing_17:head# 2020-11-16T20:11:05.300 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.349 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c6c5a7c2:::existing_39:head# 2020-11-16T20:11:05.349 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.383 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cc1d0abf:::existing_95:head# 2020-11-16T20:11:05.383 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.416 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ccb3865b:::existing_73:head# 2020-11-16T20:11:05.416 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.449 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cd84e19e:::existing_40:head# 2020-11-16T20:11:05.450 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.483 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d2cd64ce:::existing_37:head# 2020-11-16T20:11:05.483 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.525 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d41a25c9:::existing_22:head# 2020-11-16T20:11:05.525 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.558 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d7f088da:::existing_91:head# 2020-11-16T20:11:05.559 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.592 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d9cbd1d6:::existing_64:head# 2020-11-16T20:11:05.592 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.625 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:dcc4ff82:::existing_98:head# 2020-11-16T20:11:05.625 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.675 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:dd677bd3:::existing_35:head# 2020-11-16T20:11:05.676 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.759 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:dd6d90e7:::existing_46:head# 2020-11-16T20:11:05.760 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.810 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e0e22077:::existing_21:head# 2020-11-16T20:11:05.810 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.860 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e255b710:::existing_55:head# 2020-11-16T20:11:05.860 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.910 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:eff1d5db:::existing_14:head# 2020-11-16T20:11:05.910 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:05.960 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f1b156bf:::existing_23:head# 2020-11-16T20:11:05.961 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:06.010 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f5ce4199:::existing_34:head# 2020-11-16T20:11:06.011 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:06.061 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f6809456:::existing_11:head# 2020-11-16T20:11:06.061 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:06.111 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f699a2a9:::existing_67:head# 2020-11-16T20:11:06.111 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:06.161 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f9baac43:::existing_25:head# 2020-11-16T20:11:06.161 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:06.211 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:fb76688e:::existing_51:head# 2020-11-16T20:11:06.212 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:06.253 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:fbb92eec:::existing_69:head# 2020-11-16T20:11:06.253 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:11:06.328 INFO:tasks.workunit.client.0.smithi189.stdout:write_pg epoch 44 info 1.0( v 44'102 lc 0'0 (29'90,44'102] local-lis/les=43/44 n=101 ec=23/23 lis/c=43/23 les/c/f=44/24/0 sis=43 pruub=14.803615508s) 2020-11-16T20:11:06.370 INFO:tasks.workunit.client.0.smithi189.stdout:Import successful 2020-11-16T20:11:06.909 INFO:tasks.workunit.client.0.smithi189.stdout:reviving divergent 1 2020-11-16T20:11:06.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2020-11-16T20:11:06.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:11:07.286 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:11:07.287 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 4 4 0 0 61854 0 0 11 11 active+recovering+degraded+wait 2020-11-16T20:10:44.771948+0000 39'101 44:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:11:07.287 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:11:07.287 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:11:07.289 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:11:07.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-11-16T20:11:07.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:11:07.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:11:07.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T20:11:07.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:11:07.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:11:07.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:11:07.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:11:07.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:11:07.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:11:07.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:11:07.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:11:07.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:11:07.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:11:07.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:11:07.301 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:11:07.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:11:07.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:11:07.302 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:11:07.302 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:07.302 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:07.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:11:07.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:11:07.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:11:07.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:11:07.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:11:07.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:11:07.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:11:07.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:11:07.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:11:07.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:11:07.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:11:07.304 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:11:07.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T20:11:07.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:11:07.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2020-11-16T20:11:07.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T20:11:07.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:11:07.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:11:07.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:11:07.339 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:07.339+0000 7f10b0406d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:07.342 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:07.343+0000 7f10b0406d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:07.343 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:07.343+0000 7f10b0406d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:07.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T20:11:07.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:11:07.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:11:07.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:11:07.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:11:07.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:11:07.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:11:07.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:11:07.693 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:11:07.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:11:07.899 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:07.899+0000 7f10b0406d80 -1 Falling back to public interface 2020-11-16T20:11:08.128 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/32570,v1:127.0.0.1:6819/32570] [v2:127.0.0.1:6820/32570,v1:127.0.0.1:6821/32570] exists,up ea764750-7154-4c9e-af61-617257854e05 2020-11-16T20:11:08.128 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:11:08.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:11:08.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:11:08.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:11:08.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2020-11-16T20:11:08.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:11:08.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:11:08.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:11:08.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:11:08.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:11:08.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:11:08.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:11:08.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:11:08.529 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/32570,v1:127.0.0.1:6819/32570] [v2:127.0.0.1:6820/32570,v1:127.0.0.1:6821/32570] exists,up ea764750-7154-4c9e-af61-617257854e05 2020-11-16T20:11:08.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:11:08.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:11:08.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:11:08.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2020-11-16T20:11:09.032 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:09.031+0000 7f10b0406d80 -1 osd.1 44 log_to_monitors {default=true} 2020-11-16T20:11:28.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2020-11-16T20:11:28.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:11:28.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:11:28.535 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:11:28.535 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:28.535 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:28.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.1.asok 2020-11-16T20:11:28.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2020-11-16T20:11:28.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.1.asok dump_ops_in_flight 2020-11-16T20:11:28.647 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:11:28.647 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [], 2020-11-16T20:11:28.647 INFO:tasks.workunit.client.0.smithi189.stdout: "num_ops": 0 2020-11-16T20:11:28.648 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:11:28.656 INFO:tasks.workunit.client.0.smithi189.stdout:allowing recovery 2020-11-16T20:11:28.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2020-11-16T20:11:28.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:11:29.030 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:11:29.030 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 0 0 0 0 62471 0 0 12 12 active+clean 2020-11-16T20:11:11.321754+0000 44'102 48:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:11:29.030 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:11:29.030 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:11:29.032 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:11:29.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-11-16T20:11:29.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2020-11-16T20:11:29.205 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:11:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-11-16T20:11:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T20:11:29.375 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:11:29.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-11-16T20:11:29.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2020-11-16T20:11:29.549 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:11:29.558 INFO:tasks.workunit.client.0.smithi189.stdout:reading divergent objects 2020-11-16T20:11:29.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2020-11-16T20:11:29.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2020-11-16T20:11:29.933 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:11:29.933 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 0 0 0 0 62471 0 0 12 12 active+clean 2020-11-16T20:11:11.321754+0000 44'102 48:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:08:31.135553+0000 0'0 2020-11-16T20:08:31.135553+0000 0 2020-11-16T20:11:29.933 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:11:29.934 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:11:29.935 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:11:29.946 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2020-11-16T20:11:29.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2020-11-16T20:11:29.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:11:29.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2020-11-16T20:11:29.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:11:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2020-11-16T20:11:30.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:11:30.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2020-11-16T20:11:30.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:11:30.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2020-11-16T20:11:30.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2020-11-16T20:11:30.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-11-16T20:11:30.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2020-11-16T20:11:30.169 INFO:tasks.workunit.client.0.smithi189.stderr:error getting test/newobject_1: (2) No such file or directory 2020-11-16T20:11:30.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-11-16T20:11:30.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2020-11-16T20:11:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:error getting test/newobject_2: (2) No such file or directory 2020-11-16T20:11:30.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2020-11-16T20:11:30.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-11-16T20:11:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2020-11-16T20:11:30.224 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4535.0:1 2020-11-16T20:09:02.670513+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-11-16T20:11:30.225 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: keeping 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4535.0:1 2020-11-16T20:09:02.670513+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-11-16T20:11:30.225 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:11:30.225 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'104 last_divergent_update: 29'104 2020-11-16T20:11:30.225 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: more recent entry found: 39'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4650.0:1 2020-11-16T20:09:59.737107+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0, already merged 2020-11-16T20:11:30.225 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'102 (0'0) modify 1:75ac5091:::newobject_2:head by client.4525.0:1 2020-11-16T20:09:02.661823+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-11-16T20:11:30.226 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: keeping 29'102 (0'0) modify 1:75ac5091:::newobject_2:head by client.4525.0:1 2020-11-16T20:09:02.661823+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-11-16T20:11:30.226 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:11:30.226 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'102 last_divergent_update: 29'102 2020-11-16T20:11:30.226 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-11-16T20:11:30.227 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-11-16T20:11:30.227 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4539.0:1 2020-11-16T20:09:02.674183+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-11-16T20:11:30.227 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: keeping 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4539.0:1 2020-11-16T20:09:02.674183+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-11-16T20:11:30.227 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-11-16T20:11:30.227 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'106 last_divergent_update: 29'106 2020-11-16T20:11:30.228 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-11-16T20:11:30.228 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-11-16T20:11:30.228 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2020-11-16T20:11:30.228 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4539.0:1 2020-11-16T20:09:02.674183+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-11-16T20:11:30.228 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-11-16T20:11:30.228 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2020-11-16T20:11:30.229 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'105 (29'2) modify 1:97db2a0f:::existing_2:head by client.4538.0:1 2020-11-16T20:09:02.673622+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:11:30.229 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: keeping 29'105 (29'2) modify 1:97db2a0f:::existing_2:head by client.4538.0:1 2020-11-16T20:09:02.673622+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:11:30.229 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:11:30.229 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'105 last_divergent_update: 29'105 2020-11-16T20:11:30.230 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-11-16T20:11:30.230 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-11-16T20:11:30.230 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2020-11-16T20:11:30.230 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'105 (29'2) modify 1:97db2a0f:::existing_2:head by client.4538.0:1 2020-11-16T20:09:02.673622+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:11:30.230 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-11-16T20:11:30.230 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2020-11-16T20:11:30.231 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'103 (29'1) modify 1:d858c035:::existing_1:head by client.4534.0:1 2020-11-16T20:09:02.669285+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:11:30.231 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: keeping 29'103 (29'1) modify 1:d858c035:::existing_1:head by client.4534.0:1 2020-11-16T20:09:02.669285+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:11:30.231 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:11:30.232 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'103 last_divergent_update: 29'103 2020-11-16T20:11:30.232 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-11-16T20:11:30.232 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-11-16T20:11:30.232 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2020-11-16T20:11:30.232 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'103 (29'1) modify 1:d858c035:::existing_1:head by client.4534.0:1 2020-11-16T20:09:02.669285+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:11:30.232 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-11-16T20:11:30.233 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2020-11-16T20:11:30.233 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4527.0:1 2020-11-16T20:09:02.654879+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-11-16T20:11:30.233 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4527.0:1 2020-11-16T20:09:02.654879+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-11-16T20:11:30.233 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:11:30.233 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2020-11-16T20:11:30.233 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-11-16T20:11:30.234 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:10:44.318+0000 7f2a9d9a5700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-11-16T20:11:30.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-11-16T20:11:30.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2020-11-16T20:11:30.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-11-16T20:11:30.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2020-11-16T20:11:30.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2020-11-16T20:11:30.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.12850.out 2020-11-16T20:11:30.245 INFO:tasks.workunit.client.0.smithi189.stdout:success 2020-11-16T20:11:30.246 INFO:tasks.workunit.client.0.smithi189.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.12850.out': No such file or directory 2020-11-16T20:11:30.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2020-11-16T20:11:30.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T20:11:30.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T20:11:30.834 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T20:11:30.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2020-11-16T20:11:30.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:11:30.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:11:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:11:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:11:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:11:30.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:11:30.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-11-16T20:11:30.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-11-16T20:11:30.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:11:30.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-11-16T20:11:30.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:11:30.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:11:30.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:11:30.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:11:30.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:11:30.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:11:30.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:11:30.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:11:30.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:11:30.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:11:30.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:11:30.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:11:30.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:11:30.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:11:30.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:11:30.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:11:30.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:11:30.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:11:30.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-11-16T20:11:31.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:11:31.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:31.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:31.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12850 2020-11-16T20:11:31.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:11:31.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:11:31.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-11-16T20:11:31.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-11-16T20:11:31.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-11-16T20:11:31.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-11-16T20:11:31.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-11-16T20:11:31.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:11:31.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-11-16T20:11:31.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:11:31.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:11:31.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:11:31.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:11:31.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:11:31.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:11:31.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:11:31.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:11:31.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:11:31.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:11:31.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:11:31.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:11:31.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:11:31.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:11:31.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:11:31.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:11:31.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:11:31.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:11:31.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-11-16T20:11:31.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:11:31.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:31.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:31.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12850 2020-11-16T20:11:31.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:11:31.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:11:31.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-11-16T20:11:31.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:11:31.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:31.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:31.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12850 2020-11-16T20:11:31.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:11:31.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T20:11:31.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:11:31.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-11-16T20:11:31.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2020-11-16T20:11:31.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2020-11-16T20:11:31.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2020-11-16T20:11:31.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2020-11-16T20:11:31.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2020-11-16T20:11:31.031 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2020-11-16T20:11:31.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2020-11-16T20:11:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2020-11-16T20:11:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:11:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:11:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2020-11-16T20:11:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-11-16T20:11:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:11:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:11:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:11:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-11-16T20:11:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-11-16T20:11:31.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:11:31.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:11:31.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:11:31.368 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:11:31.368 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:31.369 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:31.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:11:31.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:11:31.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:11:31.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:11:31.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:11:31.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:11:31.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:11:31.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:11:31.575 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:11:31.575 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:11:31.575 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:11:31.579 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:11:31.579 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:31.580 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:31.580 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-mon.a.asok 2020-11-16T20:11:31.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:11:31.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.12850/ceph-mon.a.asok config get fsid 2020-11-16T20:11:31.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:11:31.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:11:31.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:11:31.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:11:31.690 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:11:31.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:11:31.691 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:11:31.691 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:11:31.691 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:11:31.691 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:31.691 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:31.692 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-mon.a.asok 2020-11-16T20:11:31.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:11:31.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.12850/ceph-mon.a.asok config get mon_host 2020-11-16T20:11:31.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2020-11-16T20:11:31.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2020-11-16T20:11:31.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:11:31.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:11:31.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:11:31.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2020-11-16T20:11:31.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:11:32.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:11:32.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:11:32.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:11:32.295 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:11:32.295 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:32.295 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:32.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:11:32.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:11:32.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:11:32.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-11-16T20:11:32.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2020-11-16T20:11:32.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:11:32.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:11:32.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:11:32.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:11:32.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2020-11-16T20:11:32.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:11:32.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:11:32.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:11:32.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:11:32.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-11-16T20:11:32.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-11-16T20:11:32.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:11:32.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:11:32.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:11:32.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:11:32.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:11:32.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:11:32.336 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:11:32.336 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:32.337 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:32.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:11:32.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:11:32.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:11:32.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:11:32.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:11:32.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:11:32.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:11:32.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:11:32.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:11:32.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:11:32.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:11:32.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:11:32.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2020-11-16T20:11:32.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:11:32.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=42e192ce-7576-4c00-87e5-e211cf1e1011 2020-11-16T20:11:32.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 42e192ce-7576-4c00-87e5-e211cf1e1011' 2020-11-16T20:11:32.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:11:32.340 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 42e192ce-7576-4c00-87e5-e211cf1e1011 2020-11-16T20:11:32.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD03LJfMjAlFRAADX7QeilZBLWO7X9K1I0LVQ== 2020-11-16T20:11:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD03LJfMjAlFRAADX7QeilZBLWO7X9K1I0LVQ=="}' 2020-11-16T20:11:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 42e192ce-7576-4c00-87e5-e211cf1e1011 -i td/divergent-priors/0/new.json 2020-11-16T20:11:32.705 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:11:32.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2020-11-16T20:11:32.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD03LJfMjAlFRAADX7QeilZBLWO7X9K1I0LVQ== --osd-uuid 42e192ce-7576-4c00-87e5-e211cf1e1011 2020-11-16T20:11:32.755 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:32.753+0000 7fdef0b57d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:32.757 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:32.753+0000 7fdef0b57d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:32.759 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:32.757+0000 7fdef0b57d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:32.761 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:32.757+0000 7fdef0b57d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:11:32.761 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:32.757+0000 7fdef0b57d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:11:32.761 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:32.757+0000 7fdef0b57d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:11:32.813 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:32.809+0000 7fdef0b57d80 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-11-16T20:11:37.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-11-16T20:11:37.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:11:37.342 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:11:37.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:11:37.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:11:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:11:37.861 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:11:37.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:11:37.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:11:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:11:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:11:37.896 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:37.893+0000 7f819be92d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:37.899 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:37.897+0000 7f819be92d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:37.901 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:37.897+0000 7f819be92d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:38.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:11:38.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:11:38.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:11:38.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:11:38.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:11:38.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:11:38.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:11:38.220 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:11:38.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:11:38.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:11:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:38.433+0000 7f819be92d80 -1 Falling back to public interface 2020-11-16T20:11:38.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:11:39.033 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:39.021+0000 7f819be92d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:11:39.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:11:39.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:11:39.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:11:39.617 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:11:39.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:11:39.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:11:40.025 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/35273,v1:127.0.0.1:6803/35273] [v2:127.0.0.1:6804/35273,v1:127.0.0.1:6805/35273] exists,up 42e192ce-7576-4c00-87e5-e211cf1e1011 2020-11-16T20:11:40.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:11:40.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:11:40.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:11:40.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-11-16T20:11:40.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2020-11-16T20:11:40.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:11:40.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:11:40.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:11:40.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:11:40.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:11:40.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:11:40.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:11:40.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:11:40.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:11:40.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:11:40.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:11:40.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:11:40.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:11:40.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:11:40.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:11:40.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:11:40.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:11:40.031 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:11:40.031 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:40.031 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:40.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:11:40.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:11:40.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:11:40.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:11:40.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:11:40.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:11:40.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:11:40.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:11:40.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:11:40.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:11:40.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:11:40.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:11:40.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:11:40.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:11:40.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=dc40a541-b8cd-4c3b-81a7-0d36bc669c17 2020-11-16T20:11:40.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 dc40a541-b8cd-4c3b-81a7-0d36bc669c17' 2020-11-16T20:11:40.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:11:40.034 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 dc40a541-b8cd-4c3b-81a7-0d36bc669c17 2020-11-16T20:11:40.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD83LJfkzL9AhAAza/tRiPSnIyEUffNw+wOhQ== 2020-11-16T20:11:40.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD83LJfkzL9AhAAza/tRiPSnIyEUffNw+wOhQ=="}' 2020-11-16T20:11:40.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new dc40a541-b8cd-4c3b-81a7-0d36bc669c17 -i td/divergent-priors/1/new.json 2020-11-16T20:11:40.608 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:11:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2020-11-16T20:11:40.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD83LJfkzL9AhAAza/tRiPSnIyEUffNw+wOhQ== --osd-uuid dc40a541-b8cd-4c3b-81a7-0d36bc669c17 2020-11-16T20:11:40.656 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:40.652+0000 7fd47b21ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:40.659 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:40.652+0000 7fd47b21ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:40.660 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:40.656+0000 7fd47b21ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:40.661 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:40.656+0000 7fd47b21ed80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:11:40.661 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:40.656+0000 7fd47b21ed80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:11:40.661 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:40.656+0000 7fd47b21ed80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:11:40.701 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:40.696+0000 7fd47b21ed80 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-11-16T20:11:45.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-11-16T20:11:45.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:11:45.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:11:45.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:11:45.323 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:11:45.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:11:45.824 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:11:45.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:11:45.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:11:45.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:11:45.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:11:45.858 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:45.852+0000 7fb833bcdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:45.860 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:45.856+0000 7fb833bcdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:45.861 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:45.856+0000 7fb833bcdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:46.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:11:46.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:11:46.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:11:46.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:11:46.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:11:46.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:11:46.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:11:46.211 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:11:46.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:11:46.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:11:46.406 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:46.400+0000 7fb833bcdd80 -1 Falling back to public interface 2020-11-16T20:11:46.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:11:47.249 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:47.244+0000 7fb833bcdd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:11:47.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:11:47.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:11:47.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:11:47.608 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:11:47.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:11:47.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:11:48.002 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/36267,v1:127.0.0.1:6811/36267] [v2:127.0.0.1:6812/36267,v1:127.0.0.1:6813/36267] exists,up dc40a541-b8cd-4c3b-81a7-0d36bc669c17 2020-11-16T20:11:48.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:11:48.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:11:48.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:11:48.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-11-16T20:11:48.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2020-11-16T20:11:48.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:11:48.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:11:48.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:11:48.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:11:48.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2020-11-16T20:11:48.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:11:48.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:11:48.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:11:48.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:11:48.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-11-16T20:11:48.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-11-16T20:11:48.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:11:48.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:11:48.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:11:48.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:11:48.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:11:48.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:11:48.007 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:11:48.007 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:11:48.007 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:11:48.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:11:48.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:11:48.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:11:48.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:11:48.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:11:48.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:11:48.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:11:48.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:11:48.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:11:48.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:11:48.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:11:48.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:11:48.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2020-11-16T20:11:48.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:11:48.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ca0542d6-a1d5-4f3f-8ddc-4d1c683befc4 2020-11-16T20:11:48.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 ca0542d6-a1d5-4f3f-8ddc-4d1c683befc4' 2020-11-16T20:11:48.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:11:48.010 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 ca0542d6-a1d5-4f3f-8ddc-4d1c683befc4 2020-11-16T20:11:48.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAE3bJf7tqbARAAflVrBX0LJPrBUa4o+wo71A== 2020-11-16T20:11:48.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAE3bJf7tqbARAAflVrBX0LJPrBUa4o+wo71A=="}' 2020-11-16T20:11:48.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ca0542d6-a1d5-4f3f-8ddc-4d1c683befc4 -i td/divergent-priors/2/new.json 2020-11-16T20:11:48.528 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:11:48.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2020-11-16T20:11:48.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAE3bJf7tqbARAAflVrBX0LJPrBUa4o+wo71A== --osd-uuid ca0542d6-a1d5-4f3f-8ddc-4d1c683befc4 2020-11-16T20:11:48.578 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:48.572+0000 7f6788045d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:48.580 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:48.576+0000 7f6788045d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:48.582 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:48.576+0000 7f6788045d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:48.583 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:48.576+0000 7f6788045d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:11:48.583 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:48.576+0000 7f6788045d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:11:48.583 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:48.576+0000 7f6788045d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:11:48.622 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:48.616+0000 7f6788045d80 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-11-16T20:11:53.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-11-16T20:11:53.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:11:53.359 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:11:53.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:11:53.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:11:53.868 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:11:53.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:11:53.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:11:53.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:11:53.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:11:53.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:11:53.903 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:53.896+0000 7fb6fc77ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:53.905 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:53.900+0000 7fb6fc77ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:53.907 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:53.900+0000 7fb6fc77ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:11:54.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:11:54.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:11:54.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:11:54.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:11:54.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:11:54.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:11:54.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:11:54.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:11:54.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:11:54.246 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:11:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:54.435+0000 7fb6fc77ad80 -1 Falling back to public interface 2020-11-16T20:11:54.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:11:55.295 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:11:55.283+0000 7fb6fc77ad80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:11:55.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:11:55.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:11:55.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:11:55.614 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:11:55.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:11:55.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:11:56.033 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/37261,v1:127.0.0.1:6819/37261] [v2:127.0.0.1:6820/37261,v1:127.0.0.1:6821/37261] exists,up ca0542d6-a1d5-4f3f-8ddc-4d1c683befc4 2020-11-16T20:11:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:11:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:11:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:11:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2020-11-16T20:11:56.902 INFO:tasks.workunit.client.0.smithi189.stderr:noout is set 2020-11-16T20:11:56.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2020-11-16T20:11:57.653 INFO:tasks.workunit.client.0.smithi189.stderr:noin is set 2020-11-16T20:11:57.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2020-11-16T20:11:58.430 INFO:tasks.workunit.client.0.smithi189.stderr:nodown is set 2020-11-16T20:11:58.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2020-11-16T20:11:58.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T20:11:58.992 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T20:11:59.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:12:00.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2020-11-16T20:12:01.022 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 3 2020-11-16T20:12:01.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2020-11-16T20:12:01.732 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 min_size to 2 2020-11-16T20:12:01.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2020-11-16T20:12:02.469 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 pg_autoscale_mode to on 2020-11-16T20:12:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2020-11-16T20:12:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:12:02.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:12:02.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:12:02.899 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:12:02.899 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:12:02.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:12:02.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:02.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:12:03.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2020-11-16T20:12:03.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2020-11-16T20:12:03.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2020-11-16T20:12:03.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:03.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:12:03.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2020-11-16T20:12:03.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2020-11-16T20:12:03.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2020-11-16T20:12:03.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:03.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:12:03.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542148 2020-11-16T20:12:03.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542148 2020-11-16T20:12:03.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-60129542148' 2020-11-16T20:12:03.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:12:03.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2020-11-16T20:12:03.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:12:03.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:12:03.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2020-11-16T20:12:03.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:12:03.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2020-11-16T20:12:03.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-11-16T20:12:03.434 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836487 2020-11-16T20:12:03.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:12:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2020-11-16T20:12:03.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:12:04.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:12:04.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:12:05.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2020-11-16T20:12:05.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:12:05.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2020-11-16T20:12:05.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:12:05.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:12:05.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:12:05.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2020-11-16T20:12:05.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2020-11-16T20:12:05.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-11-16T20:12:05.250 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672965 2020-11-16T20:12:05.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:12:05.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672965 2020-11-16T20:12:05.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:12:05.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542148 2020-11-16T20:12:05.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:12:05.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:12:05.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542148 2020-11-16T20:12:05.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:12:05.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542148 2020-11-16T20:12:05.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2020-11-16T20:12:05.652 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542148 2020-11-16T20:12:05.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:12:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542148 -lt 60129542148 2020-11-16T20:12:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2020-11-16T20:12:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:12:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:12:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:12:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:12:06.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:12:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:12:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:12:06.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:12:06.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:12:06.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:12:06.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:12:06.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:12:06.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:12:06.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:12:06.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:12:06.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:12:06.682 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:12:06.682 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:12:06.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:12:06.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:06.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:12:06.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2020-11-16T20:12:06.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2020-11-16T20:12:06.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2020-11-16T20:12:06.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:06.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:12:07.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2020-11-16T20:12:07.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2020-11-16T20:12:07.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2020-11-16T20:12:07.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:07.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:12:07.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542151 2020-11-16T20:12:07.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542151 2020-11-16T20:12:07.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-60129542151' 2020-11-16T20:12:07.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:12:07.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2020-11-16T20:12:07.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:12:07.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:12:07.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2020-11-16T20:12:07.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:12:07.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2020-11-16T20:12:07.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-11-16T20:12:07.213 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836490 2020-11-16T20:12:07.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:12:07.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836490 2020-11-16T20:12:07.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:12:08.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:12:08.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:12:09.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2020-11-16T20:12:09.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:12:09.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2020-11-16T20:12:09.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:12:09.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:12:09.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2020-11-16T20:12:09.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:12:09.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2020-11-16T20:12:09.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-11-16T20:12:09.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:12:09.016 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672968 2020-11-16T20:12:09.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2020-11-16T20:12:09.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:12:09.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542151 2020-11-16T20:12:09.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:12:09.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:12:09.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542151 2020-11-16T20:12:09.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:12:09.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542151 2020-11-16T20:12:09.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2020-11-16T20:12:09.410 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542151 2020-11-16T20:12:09.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:12:09.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542151 -lt 60129542151 2020-11-16T20:12:09.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:12:09.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:12:09.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:12:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:12:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:12:10.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:12:10.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:12:10.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:12:10.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:12:10.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:12:10.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:12:10.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:12:10.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:12:10.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:12:10.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:12:11.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:12:11.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:12:11.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:12:11.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2020-11-16T20:12:11.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2020-11-16T20:12:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:12:11.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2020-11-16T20:12:11.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2020-11-16T20:12:11.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:12:11.499 INFO:tasks.workunit.client.0.smithi189.stdout:primary and soon to be divergent is 1 2020-11-16T20:12:11.867 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:12:11.868 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:12:00.364070+0000 0'0 29:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:11.868 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:12:11.868 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:12:11.870 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:12:11.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2020-11-16T20:12:11.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-11-16T20:12:11.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2020-11-16T20:12:11.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2020-11-16T20:12:11.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-11-16T20:12:11.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2020-11-16T20:12:11.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2020-11-16T20:12:11.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-11-16T20:12:11.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2020-11-16T20:12:11.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2020-11-16T20:12:11.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2020-11-16T20:12:11.882 INFO:tasks.workunit.client.0.smithi189.stdout:writing initial objects 2020-11-16T20:12:11.882 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2020-11-16T20:12:11.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:11.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2020-11-16T20:12:12.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:12.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2020-11-16T20:12:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2020-11-16T20:12:12.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:12.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2020-11-16T20:12:12.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:12.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2020-11-16T20:12:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2020-11-16T20:12:12.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:12.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2020-11-16T20:12:12.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:12.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2020-11-16T20:12:13.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:13.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2020-11-16T20:12:13.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:13.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2020-11-16T20:12:13.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:13.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2020-11-16T20:12:13.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:13.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2020-11-16T20:12:13.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:13.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2020-11-16T20:12:13.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:13.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2020-11-16T20:12:13.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:13.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2020-11-16T20:12:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2020-11-16T20:12:14.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:14.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2020-11-16T20:12:14.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:14.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2020-11-16T20:12:14.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:14.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2020-11-16T20:12:14.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2020-11-16T20:12:14.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:14.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2020-11-16T20:12:14.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:14.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2020-11-16T20:12:15.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:15.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2020-11-16T20:12:15.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:15.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2020-11-16T20:12:15.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:15.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2020-11-16T20:12:15.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:15.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2020-11-16T20:12:15.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:15.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2020-11-16T20:12:15.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:15.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2020-11-16T20:12:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2020-11-16T20:12:20.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2020-11-16T20:12:20.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2020-11-16T20:12:20.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2020-11-16T20:12:20.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2020-11-16T20:12:20.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2020-11-16T20:12:20.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2020-11-16T20:12:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2020-11-16T20:12:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2020-11-16T20:12:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2020-11-16T20:12:20.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2020-11-16T20:12:20.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2020-11-16T20:12:20.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2020-11-16T20:12:20.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2020-11-16T20:12:20.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2020-11-16T20:12:20.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2020-11-16T20:12:20.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2020-11-16T20:12:20.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2020-11-16T20:12:20.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2020-11-16T20:12:20.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2020-11-16T20:12:20.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2020-11-16T20:12:20.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2020-11-16T20:12:20.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2020-11-16T20:12:20.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2020-11-16T20:12:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2020-11-16T20:12:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2020-11-16T20:12:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2020-11-16T20:12:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2020-11-16T20:12:20.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2020-11-16T20:12:20.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2020-11-16T20:12:20.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2020-11-16T20:12:20.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2020-11-16T20:12:20.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2020-11-16T20:12:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2020-11-16T20:12:20.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2020-11-16T20:12:20.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:20.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2020-11-16T20:12:21.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:21.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2020-11-16T20:12:21.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:21.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2020-11-16T20:12:21.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:21.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2020-11-16T20:12:21.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:21.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2020-11-16T20:12:21.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:21.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2020-11-16T20:12:21.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:21.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2020-11-16T20:12:21.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:21.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2020-11-16T20:12:22.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:22.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2020-11-16T20:12:22.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:22.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2020-11-16T20:12:22.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:22.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2020-11-16T20:12:22.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:22.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2020-11-16T20:12:22.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:22.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2020-11-16T20:12:22.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:22.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2020-11-16T20:12:23.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:23.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2020-11-16T20:12:23.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:23.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2020-11-16T20:12:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2020-11-16T20:12:23.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:23.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2020-11-16T20:12:23.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:23.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2020-11-16T20:12:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:23.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2020-11-16T20:12:23.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:23.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2020-11-16T20:12:24.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:24.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2020-11-16T20:12:24.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:24.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2020-11-16T20:12:24.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:24.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2020-11-16T20:12:24.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:24.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2020-11-16T20:12:24.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:24.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2020-11-16T20:12:24.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:24.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2020-11-16T20:12:24.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:24.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2020-11-16T20:12:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2020-11-16T20:12:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:25.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2020-11-16T20:12:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2020-11-16T20:12:25.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:25.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2020-11-16T20:12:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:25.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2020-11-16T20:12:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:25.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2020-11-16T20:12:26.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:26.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2020-11-16T20:12:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2020-11-16T20:12:26.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-11-16T20:12:26.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2020-11-16T20:12:26.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-11-16T20:12:26.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2020-11-16T20:12:26.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:12:26.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:12:26.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:12:26.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:12:26.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2020-11-16T20:12:26.498 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:12:26.499 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:12:26.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:12:26.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:12:26.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:12:26.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:12:26.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:12:26.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:12:26.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:12:26.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:12:27.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:12:27.044 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:12:27.044 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:12:27.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:12:27.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:27.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:12:27.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836496 2020-11-16T20:12:27.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836496 2020-11-16T20:12:27.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496' 2020-11-16T20:12:27.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:27.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:12:27.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2020-11-16T20:12:27.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2020-11-16T20:12:27.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2020-11-16T20:12:27.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:27.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:12:27.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542157 2020-11-16T20:12:27.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542157 2020-11-16T20:12:27.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-60129542157' 2020-11-16T20:12:27.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:12:27.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836496 2020-11-16T20:12:27.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:12:27.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:12:27.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2020-11-16T20:12:27.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:12:27.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836496 2020-11-16T20:12:27.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2020-11-16T20:12:27.580 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836496 2020-11-16T20:12:27.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:12:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836494 -lt 21474836496 2020-11-16T20:12:27.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:12:28.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:12:28.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:12:29.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836496 2020-11-16T20:12:29.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:12:29.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2020-11-16T20:12:29.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:12:29.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:12:29.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2020-11-16T20:12:29.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:12:29.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2020-11-16T20:12:29.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2020-11-16T20:12:29.387 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672974 2020-11-16T20:12:29.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:12:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2020-11-16T20:12:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:12:29.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542157 2020-11-16T20:12:29.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:12:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:12:29.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542157 2020-11-16T20:12:29.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:12:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542157 2020-11-16T20:12:29.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2020-11-16T20:12:29.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:12:29.800 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542157 2020-11-16T20:12:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542157 -lt 60129542157 2020-11-16T20:12:30.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:12:30.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:12:30.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:12:30.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:12:30.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:12:30.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:12:30.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:12:30.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:12:30.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:12:30.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:12:30.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:12:31.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:12:31.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:12:31.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:12:31.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:12:31.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:12:31.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:12:31.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:12:31.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2020-11-16T20:12:31.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T20:12:31.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2020-11-16T20:12:31.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2020-11-16T20:12:31.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T20:12:32.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2020-11-16T20:12:32.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2020-11-16T20:12:32.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:12:32.141 INFO:tasks.workunit.client.0.smithi189.stdout:blackholing osds 0 2 2020-11-16T20:12:32.516 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:12:32.516 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2020-11-16T20:12:00.364070+0000 30'100 30:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:32.516 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:12:32.516 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:12:32.518 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:12:32.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-11-16T20:12:32.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2020-11-16T20:12:32.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T20:12:32.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T20:12:32.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:12:32.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:12:32.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:12:32.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.0.asok 2020-11-16T20:12:32.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-11-16T20:12:32.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.0.asok config set objectstore_blackhole 1 2020-11-16T20:12:32.631 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:12:32.632 INFO:tasks.workunit.client.0.smithi189.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-11-16T20:12:32.632 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:12:32.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-11-16T20:12:32.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2020-11-16T20:12:32.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-11-16T20:12:32.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-11-16T20:12:32.645 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:12:32.645 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:12:32.646 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:12:32.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.2.asok 2020-11-16T20:12:32.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-11-16T20:12:32.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.2.asok config set objectstore_blackhole 1 2020-11-16T20:12:32.738 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:12:32.738 INFO:tasks.workunit.client.0.smithi189.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-11-16T20:12:32.738 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:12:32.746 INFO:tasks.workunit.client.0.smithi189.stdout:create new divergent objects 2020-11-16T20:12:32.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2020-11-16T20:12:32.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2020-11-16T20:12:32.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-11-16T20:12:32.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-11-16T20:12:32.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2020-11-16T20:12:32.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2020-11-16T20:12:32.750 INFO:tasks.workunit.client.0.smithi189.stdout:writing divergent objects 2020-11-16T20:12:32.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2020-11-16T20:12:32.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2020-11-16T20:12:32.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-11-16T20:12:32.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-11-16T20:12:32.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2020-11-16T20:12:32.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2020-11-16T20:12:32.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2020-11-16T20:12:32.752 INFO:tasks.workunit.client.0.smithi189.stdout:remove divergent objects 2020-11-16T20:12:32.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2020-11-16T20:12:32.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-11-16T20:12:32.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2020-11-16T20:12:32.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2020-11-16T20:12:32.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-11-16T20:12:32.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2020-11-16T20:12:32.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2020-11-16T20:12:32.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2020-11-16T20:12:32.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2020-11-16T20:12:32.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2020-11-16T20:12:42.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2020-11-16T20:12:42.763 INFO:tasks.workunit.client.0.smithi189.stderr:rados: no process found 2020-11-16T20:12:42.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2020-11-16T20:12:42.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:12:42.764 INFO:tasks.workunit.client.0.smithi189.stdout:killing all the osds 2020-11-16T20:12:43.147 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:12:43.147 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.467447+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.148 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.467294+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.148 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.468180+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.148 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.467254+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.148 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.467496+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.148 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.470733+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.148 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.470434+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.148 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.468454+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.149 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.467566+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.149 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.468234+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.149 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.469300+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.149 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.469945+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.149 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.468781+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.149 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.467820+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.149 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.469036+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.468986+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 3 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:12:38.469522+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:12:38.469135+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:12:38.469881+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:12:38.470205+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.177773+0000 30'106 32:123 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:12:38.468620+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 3 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:12:38.468335+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.470706+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.471032+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.470523+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.466850+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.469574+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.470472+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.152 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.468486+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.152 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.469395+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.152 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:12:38.468167+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:43.152 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:12:43.152 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:12:43.153 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:12:43.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2020-11-16T20:12:43.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:12:43.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:12:43.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:12:43.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:12:43.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:12:43.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:12:43.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-11-16T20:12:43.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2020-11-16T20:12:43.888 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already down. 2020-11-16T20:12:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-11-16T20:12:43.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2020-11-16T20:12:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already down. 2020-11-16T20:12:44.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-11-16T20:12:44.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2020-11-16T20:12:45.048 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already down. 2020-11-16T20:12:45.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-11-16T20:12:45.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2020-11-16T20:12:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T20:12:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-11-16T20:12:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2020-11-16T20:12:46.198 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already out. 2020-11-16T20:12:46.208 INFO:tasks.workunit.client.0.smithi189.stdout:bringing up non_divergent 0 2 2020-11-16T20:12:46.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2020-11-16T20:12:46.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:12:46.581 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:12:46.581 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.467447+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.581 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.467294+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.581 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.468180+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.581 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.467254+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.467496+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.470733+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.470434+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.468454+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.467566+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.468234+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.469300+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.583 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.469945+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.583 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.468781+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.583 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.467820+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.583 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.469036+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.583 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.468986+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.583 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 3 0 0 0 0 1900 0 0 16 16 stale+peering 2020-11-16T20:12:38.469522+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.583 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 4 0 0 0 0 1900 0 0 16 16 stale+peering 2020-11-16T20:12:38.469135+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.584 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 0 0 0 1900 0 0 16 16 stale+peering 2020-11-16T20:12:38.469881+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.584 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 4 0 0 0 0 1900 0 0 16 16 stale+peering 2020-11-16T20:12:38.470205+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.584 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.177773+0000 30'106 32:123 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.584 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 0 0 0 1900 0 0 16 16 stale+peering 2020-11-16T20:12:38.468620+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.584 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 3 0 0 0 0 1900 0 0 16 16 stale+peering 2020-11-16T20:12:38.468335+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.584 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.470706+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.584 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.471032+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.470523+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.466850+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.469574+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.470472+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.468486+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.469395+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 0 0 0 1899 0 0 16 16 stale+peering 2020-11-16T20:12:38.468167+0000 30'106 32:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:12:46.586 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:12:46.586 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:12:46.586 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:12:46.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-11-16T20:12:46.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2020-11-16T20:12:46.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:12:46.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:12:46.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T20:12:46.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:12:46.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2020-11-16T20:12:46.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:12:46.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:12:46.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:12:46.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:12:46.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-11-16T20:12:46.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-11-16T20:12:46.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:12:46.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:12:46.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:12:46.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:12:46.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:12:46.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:12:46.604 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:12:46.604 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:12:46.604 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:12:46.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:12:46.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:12:46.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:12:46.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:12:46.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:12:46.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:12:46.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:12:46.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:12:46.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:12:46.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:12:46.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2020-11-16T20:12:46.607 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:12:46.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T20:12:46.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:12:46.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2020-11-16T20:12:46.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T20:12:46.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:12:46.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:12:46.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:12:46.637 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:12:46.633+0000 7f08b3d30d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:12:46.640 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:12:46.633+0000 7f08b3d30d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:12:46.642 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:12:46.637+0000 7f08b3d30d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:12:47.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T20:12:47.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:12:47.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:12:47.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:12:47.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:12:47.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:12:47.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:12:47.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:12:47.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:12:47.008 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:12:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:12:47.169+0000 7f08b3d30d80 -1 Falling back to public interface 2020-11-16T20:12:47.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:12:48.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:12:48.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:12:48.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:12:48.395 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:12:48.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:12:48.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:12:48.555 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:12:48.549+0000 7f08b3d30d80 -1 osd.0 30 log_to_monitors {default=true} 2020-11-16T20:12:48.831 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up out weight 0 up_from 39 up_thru 0 down_at 34 last_clean_interval [5,30) [v2:127.0.0.1:6802/41211,v1:127.0.0.1:6803/41211] [v2:127.0.0.1:6804/41211,v1:127.0.0.1:6805/41211] exists,up 42e192ce-7576-4c00-87e5-e211cf1e1011 2020-11-16T20:12:48.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:12:48.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:12:48.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:12:48.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-11-16T20:12:48.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2020-11-16T20:12:48.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:12:48.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:12:48.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-11-16T20:12:48.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:12:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2020-11-16T20:12:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:12:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:12:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:12:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:12:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-11-16T20:12:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-11-16T20:12:48.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:12:48.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:12:48.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:12:48.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:12:48.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:12:48.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:12:48.836 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:12:48.836 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:12:48.836 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:12:48.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:12:48.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:12:48.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:12:48.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:12:48.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:12:48.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:12:48.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:12:48.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:12:48.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:12:48.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:12:48.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2020-11-16T20:12:48.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-11-16T20:12:48.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:12:48.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2020-11-16T20:12:48.839 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:12:48.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-11-16T20:12:48.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:12:48.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:12:48.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:12:48.866 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:12:48.861+0000 7f41dd7e4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:12:48.867 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:12:48.861+0000 7f41dd7e4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:12:48.869 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:12:48.865+0000 7f41dd7e4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:12:49.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-11-16T20:12:49.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:12:49.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:12:49.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:12:49.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:12:49.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:12:49.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:12:49.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:12:49.282 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:12:49.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:12:49.418 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:12:49.413+0000 7f41dd7e4d80 -1 Falling back to public interface 2020-11-16T20:12:49.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:12:50.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:12:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:12:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:12:50.670 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:12:50.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:12:50.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:12:50.819 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:12:50.813+0000 7f41dd7e4d80 -1 osd.2 30 log_to_monitors {default=true} 2020-11-16T20:12:55.060 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up out weight 0 up_from 40 up_thru 0 down_at 36 last_clean_interval [14,30) [v2:127.0.0.1:6810/41610,v1:127.0.0.1:6811/41610] [v2:127.0.0.1:6812/41610,v1:127.0.0.1:6813/41610] exists,up ca0542d6-a1d5-4f3f-8ddc-4d1c683befc4 2020-11-16T20:12:55.061 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 167503724547 2020-11-16T20:12:55.061 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 171798691843 2020-11-16T20:12:55.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:12:55.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:12:55.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:12:55.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-11-16T20:12:55.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2020-11-16T20:12:55.062 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already in. 2020-11-16T20:12:55.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-11-16T20:12:55.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2020-11-16T20:12:55.062 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already in. 2020-11-16T20:12:55.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-11-16T20:12:55.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2020-11-16T20:12:55.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:12:55.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:12:55.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:12:55.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:12:55.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2020-11-16T20:12:55.063 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:12:55.063 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:12:55.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:12:55.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:12:55.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:12:55.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:12:55.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:12:55.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:12:55.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:12:55.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:12:55.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:12:55.065 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:12:55.065 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:12:55.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:12:55.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:55.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:12:55.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=167503724547 2020-11-16T20:12:55.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 167503724547 2020-11-16T20:12:55.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-167503724547' 2020-11-16T20:12:55.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:55.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:12:55.067 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-11-16T20:12:55.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T20:12:55.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T20:12:55.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T20:12:55.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:12:55.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:12:55.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691843 2020-11-16T20:12:55.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691843 2020-11-16T20:12:55.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-167503724547 2-171798691843' 2020-11-16T20:12:55.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:12:55.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-167503724547 2020-11-16T20:12:55.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:12:55.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:12:55.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-167503724547 2020-11-16T20:12:55.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:12:55.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=167503724547 2020-11-16T20:12:55.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 167503724547' 2020-11-16T20:12:55.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:12:55.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724548 -lt 167503724547 2020-11-16T20:12:55.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:12:55.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-171798691843 2020-11-16T20:12:55.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:12:55.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:12:55.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-171798691843 2020-11-16T20:12:55.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:12:55.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691843 2020-11-16T20:12:55.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 171798691843' 2020-11-16T20:12:55.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:12:55.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691843 -lt 171798691843 2020-11-16T20:12:55.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:12:55.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:12:55.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:12:55.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 32 == 0 2020-11-16T20:12:55.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:12:55.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:12:55.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:12:55.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:12:55.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:12:55.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:12:55.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:12:55.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:12:55.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:12:55.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:12:55.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:12:55.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:12:55.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:12:55.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:12:55.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:12:55.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:12:55.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:12:56.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:12:56.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:12:56.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:12:56.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:12:56.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:12:56.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:12:56.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:12:56.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:12:56.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:12:56.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:12:56.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:12:56.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:12:56.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:12:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:12:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:12:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:12:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:12:56.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:12:56.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:12:56.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:12:56.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:12:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:12:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:12:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2020-11-16T20:12:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:12:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-11-16T20:12:57.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:12:57.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:12:57.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:12:57.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:12:57.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:12:57.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:12:57.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:12:57.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:12:57.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:12:58.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:12:58.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:12:58.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:12:58.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:12:58.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:12:58.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:12:58.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:12:58.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:12:58.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:12:58.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:12:58.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:12:58.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:12:58.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:12:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:12:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2020-11-16T20:12:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:12:58.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2020-11-16T20:12:59.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:12:59.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:12:59.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:12:59.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:12:59.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:12:59.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:12:59.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:12:59.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:12:59.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:12:59.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:12:59.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:12:59.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:00.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:00.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:00.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:00.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:00.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:00.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:00.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:00.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:00.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:00.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:00.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:00.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2020-11-16T20:13:00.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:00.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2020-11-16T20:13:01.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:01.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:01.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:01.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:01.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:01.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:01.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:01.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:01.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:01.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:01.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:01.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:02.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:02.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:02.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:02.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:02.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:02.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:02.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:02.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:02.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:02.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:02.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:02.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2020-11-16T20:13:02.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:02.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2020-11-16T20:13:04.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:04.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:04.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:04.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:04.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:04.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:04.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:04.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:04.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:04.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:04.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:04.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:05.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:05.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:05.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:05.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:05.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:05.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:05.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:05.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:05.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:05.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:05.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:05.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2020-11-16T20:13:05.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:05.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2020-11-16T20:13:09.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:09.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:09.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:09.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:09.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:09.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:09.057 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:09.057 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:09.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:09.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:09.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:09.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:09.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:09.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:09.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:09.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:09.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:09.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:09.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:09.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:09.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:10.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2020-11-16T20:13:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2020-11-16T20:13:16.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:16.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:16.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:16.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:16.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:16.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:16.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:16.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:17.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:17.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:17.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:17.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:17.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:17.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:17.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2020-11-16T20:13:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2020-11-16T20:13:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:25.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:25.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:25.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:25.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:25.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:25.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:25.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:25.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:25.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:25.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:25.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:26.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:26.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:26.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:26.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:26.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:26.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:26.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:26.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:26.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:26.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:26.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:26.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2020-11-16T20:13:26.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2020-11-16T20:13:27.409 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:13:27.409 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_fingerprint": "de742bec-3667-4f2c-9f37-c103e68426dc", 2020-11-16T20:13:27.410 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "16.0.0-7219-g911cc357", 2020-11-16T20:13:27.410 INFO:tasks.workunit.client.0.smithi189.stdout: "commit": "911cc35794d871071a8bebfb0919482263b1d18b", 2020-11-16T20:13:27.410 INFO:tasks.workunit.client.0.smithi189.stdout: "timestamp": "2020-11-16T20:13:27.407383+0000", 2020-11-16T20:13:27.410 INFO:tasks.workunit.client.0.smithi189.stdout: "tag": "", 2020-11-16T20:13:27.410 INFO:tasks.workunit.client.0.smithi189.stdout: "health": { 2020-11-16T20:13:27.410 INFO:tasks.workunit.client.0.smithi189.stdout: "status": "HEALTH_WARN", 2020-11-16T20:13:27.411 INFO:tasks.workunit.client.0.smithi189.stdout: "checks": { 2020-11-16T20:13:27.411 INFO:tasks.workunit.client.0.smithi189.stdout: "OSDMAP_FLAGS": { 2020-11-16T20:13:27.411 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:13:27.411 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:13:27.411 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "nodown,noout,noin flag(s) set", 2020-11-16T20:13:27.411 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 17 2020-11-16T20:13:27.411 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.412 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [], 2020-11-16T20:13:27.412 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:13:27.412 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.412 INFO:tasks.workunit.client.0.smithi189.stdout: "OSD_DOWN": { 2020-11-16T20:13:27.412 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:13:27.412 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:13:27.413 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 osds down", 2020-11-16T20:13:27.413 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:13:27.413 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.413 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:13:27.413 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.413 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "osd.1 (root=default,host=smithi189) is down" 2020-11-16T20:13:27.413 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.414 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.414 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:13:27.414 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.414 INFO:tasks.workunit.client.0.smithi189.stdout: "PG_DEGRADED": { 2020-11-16T20:13:27.414 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:13:27.414 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:13:27.414 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2020-11-16T20:13:27.415 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 32 2020-11-16T20:13:27.415 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.415 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:13:27.415 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.415 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.415 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.415 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.415 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.416 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.416 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.416 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.416 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.416 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.416 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.416 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.417 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.417 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.417 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.417 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.417 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.417 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.417 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.417 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.418 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.418 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.418 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.418 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.418 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.418 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.418 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.419 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.419 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.419 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.419 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.419 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.a is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.419 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.419 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.419 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.420 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.420 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.420 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.420 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.420 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.420 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.d is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.420 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.421 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.421 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.421 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.421 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.421 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.f is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.421 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.421 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.422 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.422 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.422 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.422 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.422 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.422 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.422 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.422 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.423 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.423 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.423 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.423 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.423 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.423 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.423 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.423 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.424 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.424 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.424 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.424 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.424 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.424 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.424 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.425 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.425 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.425 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.425 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.425 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.425 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.425 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.426 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.426 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.426 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.426 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.426 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.426 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.426 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.426 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.427 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.427 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.427 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.427 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.427 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.427 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.427 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.428 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2020-11-16T20:13:27.428 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.428 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.428 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:13:27.428 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.428 INFO:tasks.workunit.client.0.smithi189.stdout: "POOL_APP_NOT_ENABLED": { 2020-11-16T20:13:27.428 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:13:27.428 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:13:27.429 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 pool(s) do not have an application enabled", 2020-11-16T20:13:27.429 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:13:27.429 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.429 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:13:27.429 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.429 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "application not enabled on pool 'test'" 2020-11-16T20:13:27.429 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.430 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.430 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-11-16T20:13:27.430 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.430 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.430 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:13:27.430 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.430 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.431 INFO:tasks.workunit.client.0.smithi189.stdout: "mutes": [] 2020-11-16T20:13:27.431 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.431 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_first_committed": 1, 2020-11-16T20:13:27.431 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_last_committed": 1, 2020-11-16T20:13:27.431 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap": { 2020-11-16T20:13:27.431 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:13:27.431 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:13:27.431 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:11:31.065413Z", 2020-11-16T20:13:27.432 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:11:31.065413Z", 2020-11-16T20:13:27.432 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release": 16, 2020-11-16T20:13:27.432 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release_name": "pacific", 2020-11-16T20:13:27.432 INFO:tasks.workunit.client.0.smithi189.stdout: "election_strategy": 1, 2020-11-16T20:13:27.432 INFO:tasks.workunit.client.0.smithi189.stdout: "disallowed_leaders: ": "", 2020-11-16T20:13:27.432 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": false, 2020-11-16T20:13:27.432 INFO:tasks.workunit.client.0.smithi189.stdout: "features": { 2020-11-16T20:13:27.432 INFO:tasks.workunit.client.0.smithi189.stdout: "persistent": [ 2020-11-16T20:13:27.433 INFO:tasks.workunit.client.0.smithi189.stdout: "kraken", 2020-11-16T20:13:27.433 INFO:tasks.workunit.client.0.smithi189.stdout: "luminous", 2020-11-16T20:13:27.433 INFO:tasks.workunit.client.0.smithi189.stdout: "mimic", 2020-11-16T20:13:27.433 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap-prune", 2020-11-16T20:13:27.433 INFO:tasks.workunit.client.0.smithi189.stdout: "nautilus", 2020-11-16T20:13:27.433 INFO:tasks.workunit.client.0.smithi189.stdout: "octopus", 2020-11-16T20:13:27.433 INFO:tasks.workunit.client.0.smithi189.stdout: "pacific", 2020-11-16T20:13:27.434 INFO:tasks.workunit.client.0.smithi189.stdout: "elector-pinging" 2020-11-16T20:13:27.434 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.434 INFO:tasks.workunit.client.0.smithi189.stdout: "optional": [] 2020-11-16T20:13:27.434 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.435 INFO:tasks.workunit.client.0.smithi189.stdout: "mons": [ 2020-11-16T20:13:27.435 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.435 INFO:tasks.workunit.client.0.smithi189.stdout: "rank": 0, 2020-11-16T20:13:27.435 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "a", 2020-11-16T20:13:27.435 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:13:27.435 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.435 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.435 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.436 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115", 2020-11-16T20:13:27.436 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 0 2020-11-16T20:13:27.436 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.436 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.436 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.436 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115/0", 2020-11-16T20:13:27.436 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:7115/0", 2020-11-16T20:13:27.437 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 0, 2020-11-16T20:13:27.437 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 0, 2020-11-16T20:13:27.437 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_location": "{}" 2020-11-16T20:13:27.437 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.437 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.437 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.437 INFO:tasks.workunit.client.0.smithi189.stdout: "quorum": [ 2020-11-16T20:13:27.437 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2020-11-16T20:13:27.438 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.438 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap": { 2020-11-16T20:13:27.438 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 45, 2020-11-16T20:13:27.438 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:13:27.438 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:11:31.869939+0000", 2020-11-16T20:13:27.438 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:12:52.797777+0000", 2020-11-16T20:13:27.438 INFO:tasks.workunit.client.0.smithi189.stdout: "last_up_change": "2020-11-16T20:12:50.890228+0000", 2020-11-16T20:13:27.438 INFO:tasks.workunit.client.0.smithi189.stdout: "last_in_change": "2020-11-16T20:12:52.354351+0000", 2020-11-16T20:13:27.439 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-11-16T20:13:27.439 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_num": 5800384, 2020-11-16T20:13:27.439 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_set": [ 2020-11-16T20:13:27.439 INFO:tasks.workunit.client.0.smithi189.stdout: "nodown", 2020-11-16T20:13:27.439 INFO:tasks.workunit.client.0.smithi189.stdout: "noin", 2020-11-16T20:13:27.439 INFO:tasks.workunit.client.0.smithi189.stdout: "noout", 2020-11-16T20:13:27.439 INFO:tasks.workunit.client.0.smithi189.stdout: "pglog_hardlimit", 2020-11-16T20:13:27.440 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snapdirs", 2020-11-16T20:13:27.440 INFO:tasks.workunit.client.0.smithi189.stdout: "recovery_deletes", 2020-11-16T20:13:27.440 INFO:tasks.workunit.client.0.smithi189.stdout: "sortbitwise" 2020-11-16T20:13:27.440 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.440 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_version": 7, 2020-11-16T20:13:27.440 INFO:tasks.workunit.client.0.smithi189.stdout: "full_ratio": 0.99000000953674316, 2020-11-16T20:13:27.440 INFO:tasks.workunit.client.0.smithi189.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-11-16T20:13:27.441 INFO:tasks.workunit.client.0.smithi189.stdout: "nearfull_ratio": 0.85000002384185791, 2020-11-16T20:13:27.441 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_snapshot": "", 2020-11-16T20:13:27.441 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_max": 1, 2020-11-16T20:13:27.441 INFO:tasks.workunit.client.0.smithi189.stdout: "max_osd": 3, 2020-11-16T20:13:27.441 INFO:tasks.workunit.client.0.smithi189.stdout: "require_min_compat_client": "luminous", 2020-11-16T20:13:27.441 INFO:tasks.workunit.client.0.smithi189.stdout: "min_compat_client": "jewel", 2020-11-16T20:13:27.441 INFO:tasks.workunit.client.0.smithi189.stdout: "require_osd_release": "pacific", 2020-11-16T20:13:27.441 INFO:tasks.workunit.client.0.smithi189.stdout: "pools": [ 2020-11-16T20:13:27.442 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.442 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:13:27.442 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_name": "test", 2020-11-16T20:13:27.442 INFO:tasks.workunit.client.0.smithi189.stdout: "create_time": "2020-11-16T20:11:58.751915+0000", 2020-11-16T20:13:27.442 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": 1, 2020-11-16T20:13:27.442 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_names": "hashpspool", 2020-11-16T20:13:27.442 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:13:27.443 INFO:tasks.workunit.client.0.smithi189.stdout: "size": 3, 2020-11-16T20:13:27.443 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 2, 2020-11-16T20:13:27.443 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_rule": 0, 2020-11-16T20:13:27.443 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_count": 0, 2020-11-16T20:13:27.443 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_target": 0, 2020-11-16T20:13:27.443 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_barrier": 0, 2020-11-16T20:13:27.443 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2020-11-16T20:13:27.444 INFO:tasks.workunit.client.0.smithi189.stdout: "object_hash": 2, 2020-11-16T20:13:27.444 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_autoscale_mode": "on", 2020-11-16T20:13:27.444 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num": 32, 2020-11-16T20:13:27.444 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num": 1, 2020-11-16T20:13:27.444 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num_target": 32, 2020-11-16T20:13:27.444 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_target": 32, 2020-11-16T20:13:27.444 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_pending": 32, 2020-11-16T20:13:27.444 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pg_merge_meta": { 2020-11-16T20:13:27.445 INFO:tasks.workunit.client.0.smithi189.stdout: "source_pgid": "0.0", 2020-11-16T20:13:27.445 INFO:tasks.workunit.client.0.smithi189.stdout: "ready_epoch": 0, 2020-11-16T20:13:27.445 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_started": 0, 2020-11-16T20:13:27.445 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": 0, 2020-11-16T20:13:27.445 INFO:tasks.workunit.client.0.smithi189.stdout: "source_version": "0'0", 2020-11-16T20:13:27.445 INFO:tasks.workunit.client.0.smithi189.stdout: "target_version": "0'0" 2020-11-16T20:13:27.445 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.446 INFO:tasks.workunit.client.0.smithi189.stdout: "last_change": "32", 2020-11-16T20:13:27.446 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend": "0", 2020-11-16T20:13:27.446 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_prenautilus": "0", 2020-11-16T20:13:27.446 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_preluminous": "32", 2020-11-16T20:13:27.446 INFO:tasks.workunit.client.0.smithi189.stdout: "auid": 0, 2020-11-16T20:13:27.446 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_mode": "selfmanaged", 2020-11-16T20:13:27.446 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_seq": 0, 2020-11-16T20:13:27.446 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_epoch": 0, 2020-11-16T20:13:27.447 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_snaps": [], 2020-11-16T20:13:27.447 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps": "[]", 2020-11-16T20:13:27.447 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_bytes": 0, 2020-11-16T20:13:27.447 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_objects": 0, 2020-11-16T20:13:27.447 INFO:tasks.workunit.client.0.smithi189.stdout: "tiers": [], 2020-11-16T20:13:27.447 INFO:tasks.workunit.client.0.smithi189.stdout: "tier_of": -1, 2020-11-16T20:13:27.447 INFO:tasks.workunit.client.0.smithi189.stdout: "read_tier": -1, 2020-11-16T20:13:27.448 INFO:tasks.workunit.client.0.smithi189.stdout: "write_tier": -1, 2020-11-16T20:13:27.448 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_mode": "none", 2020-11-16T20:13:27.448 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_bytes": 0, 2020-11-16T20:13:27.448 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_objects": 0, 2020-11-16T20:13:27.448 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-11-16T20:13:27.448 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-11-16T20:13:27.448 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_full_ratio_micro": 800000, 2020-11-16T20:13:27.448 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_flush_age": 0, 2020-11-16T20:13:27.449 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_evict_age": 0, 2020-11-16T20:13:27.449 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profile": "", 2020-11-16T20:13:27.449 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_params": { 2020-11-16T20:13:27.449 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "none" 2020-11-16T20:13:27.449 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.449 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_period": 0, 2020-11-16T20:13:27.449 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_count": 0, 2020-11-16T20:13:27.450 INFO:tasks.workunit.client.0.smithi189.stdout: "use_gmt_hitset": true, 2020-11-16T20:13:27.450 INFO:tasks.workunit.client.0.smithi189.stdout: "min_read_recency_for_promote": 0, 2020-11-16T20:13:27.450 INFO:tasks.workunit.client.0.smithi189.stdout: "min_write_recency_for_promote": 0, 2020-11-16T20:13:27.450 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_grade_decay_rate": 0, 2020-11-16T20:13:27.450 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_search_last_n": 0, 2020-11-16T20:13:27.450 INFO:tasks.workunit.client.0.smithi189.stdout: "grade_table": [], 2020-11-16T20:13:27.450 INFO:tasks.workunit.client.0.smithi189.stdout: "stripe_width": 0, 2020-11-16T20:13:27.451 INFO:tasks.workunit.client.0.smithi189.stdout: "expected_num_objects": 0, 2020-11-16T20:13:27.451 INFO:tasks.workunit.client.0.smithi189.stdout: "fast_read": false, 2020-11-16T20:13:27.451 INFO:tasks.workunit.client.0.smithi189.stdout: "options": {}, 2020-11-16T20:13:27.451 INFO:tasks.workunit.client.0.smithi189.stdout: "application_metadata": {} 2020-11-16T20:13:27.451 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.451 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.451 INFO:tasks.workunit.client.0.smithi189.stdout: "osds": [ 2020-11-16T20:13:27.451 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.452 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:13:27.452 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "42e192ce-7576-4c00-87e5-e211cf1e1011", 2020-11-16T20:13:27.452 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:13:27.452 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:13:27.452 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:13:27.452 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:13:27.452 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 5, 2020-11-16T20:13:27.453 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 30, 2020-11-16T20:13:27.453 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 39, 2020-11-16T20:13:27.453 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 44, 2020-11-16T20:13:27.453 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 34, 2020-11-16T20:13:27.453 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:13:27.453 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:13:27.453 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.453 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.454 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.454 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6802", 2020-11-16T20:13:27.454 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:13:27.454 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.454 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.454 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.454 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6803", 2020-11-16T20:13:27.455 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:13:27.455 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.455 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.455 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.455 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:13:27.455 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.455 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.455 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.456 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6804", 2020-11-16T20:13:27.456 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:13:27.456 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.456 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.456 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.456 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6805", 2020-11-16T20:13:27.456 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:13:27.457 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.457 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.457 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.457 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:13:27.457 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.457 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.457 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.458 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6808", 2020-11-16T20:13:27.458 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:13:27.458 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.458 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.458 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.458 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6809", 2020-11-16T20:13:27.458 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:13:27.459 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.459 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.459 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.459 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:13:27.459 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.459 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.459 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.459 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6806", 2020-11-16T20:13:27.460 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:13:27.460 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.460 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.460 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.460 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6807", 2020-11-16T20:13:27.460 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:13:27.460 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.461 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.461 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.461 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6803/41211", 2020-11-16T20:13:27.461 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6805/41211", 2020-11-16T20:13:27.461 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6809/41211", 2020-11-16T20:13:27.461 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6807/41211", 2020-11-16T20:13:27.461 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:13:27.461 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:13:27.462 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:13:27.462 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.462 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.462 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.462 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:13:27.462 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "dc40a541-b8cd-4c3b-81a7-0d36bc669c17", 2020-11-16T20:13:27.462 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 0, 2020-11-16T20:13:27.463 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:13:27.463 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:13:27.463 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:13:27.463 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 0, 2020-11-16T20:13:27.463 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 0, 2020-11-16T20:13:27.463 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 10, 2020-11-16T20:13:27.463 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 32, 2020-11-16T20:13:27.464 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 35, 2020-11-16T20:13:27.464 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:13:27.464 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:13:27.464 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.464 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.464 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.464 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:13:27.464 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:13:27.465 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.465 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.465 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.465 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:13:27.465 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:13:27.465 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.465 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.465 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.466 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:13:27.466 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.466 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.466 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.466 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:13:27.466 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:13:27.466 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.467 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.467 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.467 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:13:27.467 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:13:27.467 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.467 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.467 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.467 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:13:27.468 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.468 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.468 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.468 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:13:27.468 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:13:27.468 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.468 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.469 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.469 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:13:27.469 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:13:27.469 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.469 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.469 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.469 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:13:27.470 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.470 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.470 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.470 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:13:27.470 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:13:27.470 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.470 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.471 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.471 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:13:27.471 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:13:27.471 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.471 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.471 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.471 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/36267", 2020-11-16T20:13:27.471 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/36267", 2020-11-16T20:13:27.472 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/36267", 2020-11-16T20:13:27.472 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/36267", 2020-11-16T20:13:27.472 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:13:27.476 INFO:tasks.workunit.client.0.smithi189.stdout: "exists" 2020-11-16T20:13:27.477 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.477 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.478 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.478 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:13:27.478 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "ca0542d6-a1d5-4f3f-8ddc-4d1c683befc4", 2020-11-16T20:13:27.479 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:13:27.479 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:13:27.479 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:13:27.480 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:13:27.480 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 14, 2020-11-16T20:13:27.480 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 30, 2020-11-16T20:13:27.480 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 40, 2020-11-16T20:13:27.480 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 0, 2020-11-16T20:13:27.480 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 36, 2020-11-16T20:13:27.480 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:13:27.481 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:13:27.481 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.481 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.481 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.481 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:13:27.481 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:13:27.481 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.481 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.482 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.482 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:13:27.482 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:13:27.482 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.482 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.482 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.482 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:13:27.483 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.483 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.483 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.483 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:13:27.483 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:13:27.483 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.483 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.484 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.484 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:13:27.484 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:13:27.484 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.484 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.484 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.484 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:13:27.484 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.485 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.485 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.485 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:13:27.485 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:13:27.485 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.485 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.485 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.486 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:13:27.486 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:13:27.486 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.486 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.486 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.486 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:13:27.486 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:13:27.486 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.487 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:13:27.487 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:13:27.487 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:13:27.487 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.487 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.487 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:13:27.487 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:13:27.488 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:13:27.488 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.488 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.488 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.488 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/41610", 2020-11-16T20:13:27.488 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/41610", 2020-11-16T20:13:27.488 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/41610", 2020-11-16T20:13:27.488 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/41610", 2020-11-16T20:13:27.489 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:13:27.489 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:13:27.489 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:13:27.489 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.489 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.489 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.489 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_xinfo": [ 2020-11-16T20:13:27.490 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.490 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:13:27.490 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:12:43.647800+0000", 2020-11-16T20:13:27.490 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:13:27.490 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:13:27.490 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:13:27.490 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:13:27.490 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:11:40.042383+0000", 2020-11-16T20:13:27.491 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:13:27.491 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.491 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.491 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:13:27.491 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:12:44.237289+0000", 2020-11-16T20:13:27.491 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:13:27.491 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:13:27.492 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:13:27.492 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:13:27.492 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:11:48.292944+0000", 2020-11-16T20:13:27.492 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:13:27.492 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.492 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.492 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:13:27.492 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:12:44.821561+0000", 2020-11-16T20:13:27.493 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:13:27.493 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:13:27.493 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:13:27.493 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:13:27.493 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:11:56.248668+0000", 2020-11-16T20:13:27.493 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:13:27.493 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.493 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.494 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap": [], 2020-11-16T20:13:27.494 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap_items": [], 2020-11-16T20:13:27.494 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_temp": [], 2020-11-16T20:13:27.494 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_temp": [], 2020-11-16T20:13:27.494 INFO:tasks.workunit.client.0.smithi189.stdout: "blocklist": {}, 2020-11-16T20:13:27.494 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profiles": { 2020-11-16T20:13:27.494 INFO:tasks.workunit.client.0.smithi189.stdout: "default": { 2020-11-16T20:13:27.495 INFO:tasks.workunit.client.0.smithi189.stdout: "k": "2", 2020-11-16T20:13:27.495 INFO:tasks.workunit.client.0.smithi189.stdout: "m": "2", 2020-11-16T20:13:27.495 INFO:tasks.workunit.client.0.smithi189.stdout: "plugin": "jerasure", 2020-11-16T20:13:27.495 INFO:tasks.workunit.client.0.smithi189.stdout: "technique": "reed_sol_van" 2020-11-16T20:13:27.495 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.495 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.495 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps_queue": [], 2020-11-16T20:13:27.495 INFO:tasks.workunit.client.0.smithi189.stdout: "new_removed_snaps": [], 2020-11-16T20:13:27.496 INFO:tasks.workunit.client.0.smithi189.stdout: "new_purged_snaps": [], 2020-11-16T20:13:27.496 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_node_flags": {}, 2020-11-16T20:13:27.496 INFO:tasks.workunit.client.0.smithi189.stdout: "device_class_flags": {}, 2020-11-16T20:13:27.496 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": { 2020-11-16T20:13:27.496 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_enabled": false, 2020-11-16T20:13:27.496 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_bucket_count": 0, 2020-11-16T20:13:27.496 INFO:tasks.workunit.client.0.smithi189.stdout: "degraded_stretch_mode": 0, 2020-11-16T20:13:27.497 INFO:tasks.workunit.client.0.smithi189.stdout: "recovering_stretch_mode": 0, 2020-11-16T20:13:27.497 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_bucket": 0 2020-11-16T20:13:27.497 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.497 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.497 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_metadata": [ 2020-11-16T20:13:27.497 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.497 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:13:27.497 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:13:27.498 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6804/41211,v1:127.0.0.1:6805/41211]", 2020-11-16T20:13:27.498 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:13:27.498 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:13:27.498 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:13:27.498 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:13:27.498 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:13:27.498 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:13:27.499 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:13:27.499 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:13:27.499 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:13:27.499 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-11-16T20:13:27.499 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:13:27.499 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:13:27.499 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:13:27.499 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:13:27.500 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:13:27.500 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:13:27.500 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:13:27.500 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:13:27.500 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:13:27.500 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:13:27.500 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:13:27.501 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:13:27.501 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:13:27.501 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:13:27.501 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:13:27.501 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6802/41211,v1:127.0.0.1:6803/41211]", 2020-11-16T20:13:27.501 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:13:27.501 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/41211,v1:127.0.0.1:6809/41211]", 2020-11-16T20:13:27.501 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/41211,v1:127.0.0.1:6807/41211]", 2020-11-16T20:13:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:13:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:13:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:13:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:13:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:13:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:13:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:13:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:13:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:13:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/0", 2020-11-16T20:13:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:13:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:13:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:13:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.504 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:13:27.504 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:13:27.504 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/36267,v1:127.0.0.1:6813/36267]", 2020-11-16T20:13:27.504 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:13:27.504 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:13:27.504 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:13:27.504 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:13:27.505 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:13:27.505 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:13:27.505 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:13:27.505 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:13:27.505 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:13:27.505 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-11-16T20:13:27.505 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:13:27.506 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:13:27.506 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:13:27.506 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:13:27.506 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:13:27.506 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:13:27.506 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:13:27.506 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:13:27.506 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:13:27.507 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:13:27.507 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:13:27.507 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:13:27.507 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:13:27.507 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:13:27.507 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:13:27.507 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/36267,v1:127.0.0.1:6811/36267]", 2020-11-16T20:13:27.507 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:13:27.508 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/36267,v1:127.0.0.1:6817/36267]", 2020-11-16T20:13:27.508 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/36267,v1:127.0.0.1:6815/36267]", 2020-11-16T20:13:27.508 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:13:27.508 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:13:27.508 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:13:27.508 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:13:27.508 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:13:27.509 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:13:27.509 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:13:27.509 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:13:27.509 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:13:27.509 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/1", 2020-11-16T20:13:27.509 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:13:27.509 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:13:27.510 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:13:27.510 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.510 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.510 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:13:27.510 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:13:27.510 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/41610,v1:127.0.0.1:6813/41610]", 2020-11-16T20:13:27.510 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:13:27.510 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:13:27.511 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:13:27.511 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:13:27.511 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:13:27.511 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:13:27.511 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:13:27.511 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:13:27.511 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:13:27.511 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-11-16T20:13:27.512 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:13:27.512 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:13:27.512 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:13:27.512 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:13:27.512 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:13:27.512 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:13:27.512 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:13:27.513 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:13:27.513 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:13:27.513 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:13:27.513 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:13:27.513 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:13:27.513 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:13:27.513 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:13:27.513 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:13:27.514 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/41610,v1:127.0.0.1:6811/41610]", 2020-11-16T20:13:27.514 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:13:27.514 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/41610,v1:127.0.0.1:6817/41610]", 2020-11-16T20:13:27.514 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/41610,v1:127.0.0.1:6815/41610]", 2020-11-16T20:13:27.514 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:13:27.514 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:13:27.514 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:13:27.515 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:13:27.515 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:13:27.515 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:13:27.515 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:13:27.515 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:13:27.515 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:13:27.515 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/2", 2020-11-16T20:13:27.515 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:13:27.516 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:13:27.516 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:13:27.516 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.516 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.516 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_clean_epochs": { 2020-11-16T20:13:27.516 INFO:tasks.workunit.client.0.smithi189.stdout: "min_last_epoch_clean": 0, 2020-11-16T20:13:27.516 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": { 2020-11-16T20:13:27.516 INFO:tasks.workunit.client.0.smithi189.stdout: "per_pool": [] 2020-11-16T20:13:27.517 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.517 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_epochs": [ 2020-11-16T20:13:27.517 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.517 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:13:27.517 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 39 2020-11-16T20:13:27.517 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.517 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.518 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:13:27.518 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 10 2020-11-16T20:13:27.518 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.518 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.518 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:13:27.518 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 40 2020-11-16T20:13:27.518 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.518 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.519 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.519 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_first_committed": 1, 2020-11-16T20:13:27.519 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_last_committed": 45, 2020-11-16T20:13:27.519 INFO:tasks.workunit.client.0.smithi189.stdout: "crushmap": { 2020-11-16T20:13:27.519 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": [ 2020-11-16T20:13:27.519 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.519 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:13:27.520 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.0", 2020-11-16T20:13:27.520 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:13:27.520 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.520 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.520 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:13:27.520 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.1", 2020-11-16T20:13:27.520 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:13:27.520 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.521 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.521 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:13:27.521 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.2", 2020-11-16T20:13:27.521 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:13:27.521 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.521 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.521 INFO:tasks.workunit.client.0.smithi189.stdout: "types": [ 2020-11-16T20:13:27.521 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.522 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 0, 2020-11-16T20:13:27.522 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd" 2020-11-16T20:13:27.522 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.522 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.522 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:13:27.522 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "host" 2020-11-16T20:13:27.522 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.523 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.523 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 2, 2020-11-16T20:13:27.523 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "chassis" 2020-11-16T20:13:27.523 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.523 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.523 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 3, 2020-11-16T20:13:27.523 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "rack" 2020-11-16T20:13:27.523 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.524 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.524 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 4, 2020-11-16T20:13:27.524 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "row" 2020-11-16T20:13:27.524 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.524 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.524 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 5, 2020-11-16T20:13:27.524 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pdu" 2020-11-16T20:13:27.524 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.525 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.525 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 6, 2020-11-16T20:13:27.525 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pod" 2020-11-16T20:13:27.525 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.525 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.525 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 7, 2020-11-16T20:13:27.525 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "room" 2020-11-16T20:13:27.526 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.526 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.526 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 8, 2020-11-16T20:13:27.526 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "datacenter" 2020-11-16T20:13:27.526 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.526 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.526 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 9, 2020-11-16T20:13:27.526 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "zone" 2020-11-16T20:13:27.527 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.527 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.527 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 10, 2020-11-16T20:13:27.527 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "region" 2020-11-16T20:13:27.527 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.527 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.527 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:13:27.527 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "root" 2020-11-16T20:13:27.528 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.528 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.528 INFO:tasks.workunit.client.0.smithi189.stdout: "buckets": [ 2020-11-16T20:13:27.528 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.528 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -1, 2020-11-16T20:13:27.528 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default", 2020-11-16T20:13:27.528 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:13:27.529 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:13:27.529 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:13:27.529 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:13:27.529 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:13:27.529 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:13:27.529 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.529 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:13:27.529 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:13:27.530 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:13:27.530 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.530 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.530 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.530 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.530 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -2, 2020-11-16T20:13:27.530 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default~hdd", 2020-11-16T20:13:27.531 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:13:27.531 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:13:27.531 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:13:27.531 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:13:27.531 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:13:27.531 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:13:27.531 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.531 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:13:27.532 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:13:27.532 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:13:27.532 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.532 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.532 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.532 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.532 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:13:27.533 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189", 2020-11-16T20:13:27.533 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:13:27.533 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:13:27.533 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:13:27.533 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:13:27.533 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:13:27.533 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:13:27.533 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.534 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:13:27.534 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:13:27.534 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:13:27.534 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.534 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.534 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:13:27.534 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:13:27.534 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:13:27.535 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.535 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.535 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:13:27.535 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:13:27.535 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:13:27.535 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.535 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.536 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.536 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.536 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:13:27.536 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189~hdd", 2020-11-16T20:13:27.536 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:13:27.536 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:13:27.536 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:13:27.536 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:13:27.537 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:13:27.537 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:13:27.537 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.537 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:13:27.537 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:13:27.538 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:13:27.538 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.538 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.538 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:13:27.538 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:13:27.538 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:13:27.538 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.539 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.539 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:13:27.539 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:13:27.539 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:13:27.539 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.539 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.539 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.539 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.540 INFO:tasks.workunit.client.0.smithi189.stdout: "rules": [ 2020-11-16T20:13:27.540 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.540 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 0, 2020-11-16T20:13:27.540 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "replicated_rule", 2020-11-16T20:13:27.540 INFO:tasks.workunit.client.0.smithi189.stdout: "ruleset": 0, 2020-11-16T20:13:27.540 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:13:27.540 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 1, 2020-11-16T20:13:27.541 INFO:tasks.workunit.client.0.smithi189.stdout: "max_size": 10, 2020-11-16T20:13:27.541 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2020-11-16T20:13:27.541 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.541 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2020-11-16T20:13:27.541 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2020-11-16T20:13:27.541 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2020-11-16T20:13:27.541 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.541 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.542 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_firstn", 2020-11-16T20:13:27.542 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2020-11-16T20:13:27.542 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2020-11-16T20:13:27.542 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.542 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.542 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2020-11-16T20:13:27.542 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.543 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:13:27.543 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.543 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.543 INFO:tasks.workunit.client.0.smithi189.stdout: "tunables": { 2020-11-16T20:13:27.543 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_tries": 0, 2020-11-16T20:13:27.543 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_fallback_tries": 0, 2020-11-16T20:13:27.543 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_total_tries": 50, 2020-11-16T20:13:27.543 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_descend_once": 1, 2020-11-16T20:13:27.544 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_vary_r": 1, 2020-11-16T20:13:27.544 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_stable": 1, 2020-11-16T20:13:27.544 INFO:tasks.workunit.client.0.smithi189.stdout: "straw_calc_version": 1, 2020-11-16T20:13:27.544 INFO:tasks.workunit.client.0.smithi189.stdout: "allowed_bucket_algs": 54, 2020-11-16T20:13:27.544 INFO:tasks.workunit.client.0.smithi189.stdout: "profile": "jewel", 2020-11-16T20:13:27.544 INFO:tasks.workunit.client.0.smithi189.stdout: "optimal_tunables": 1, 2020-11-16T20:13:27.544 INFO:tasks.workunit.client.0.smithi189.stdout: "legacy_tunables": 0, 2020-11-16T20:13:27.544 INFO:tasks.workunit.client.0.smithi189.stdout: "minimum_required_version": "jewel", 2020-11-16T20:13:27.545 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables": 1, 2020-11-16T20:13:27.545 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables2": 1, 2020-11-16T20:13:27.545 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v2_rules": 0, 2020-11-16T20:13:27.545 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables3": 1, 2020-11-16T20:13:27.545 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v3_rules": 0, 2020-11-16T20:13:27.545 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v4_buckets": 1, 2020-11-16T20:13:27.545 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables5": 1, 2020-11-16T20:13:27.546 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v5_rules": 0 2020-11-16T20:13:27.546 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.546 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_args": {} 2020-11-16T20:13:27.546 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.546 INFO:tasks.workunit.client.0.smithi189.stdout: "fsmap": { 2020-11-16T20:13:27.546 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:13:27.546 INFO:tasks.workunit.client.0.smithi189.stdout: "default_fscid": -1, 2020-11-16T20:13:27.546 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": { 2020-11-16T20:13:27.547 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": {}, 2020-11-16T20:13:27.547 INFO:tasks.workunit.client.0.smithi189.stdout: "ro_compat": {}, 2020-11-16T20:13:27.547 INFO:tasks.workunit.client.0.smithi189.stdout: "incompat": { 2020-11-16T20:13:27.547 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_1": "base v0.20", 2020-11-16T20:13:27.547 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_2": "client writeable ranges", 2020-11-16T20:13:27.547 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_3": "default file layouts on dirs", 2020-11-16T20:13:27.547 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_4": "dir inode in separate object", 2020-11-16T20:13:27.547 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_5": "mds uses versioned encoding", 2020-11-16T20:13:27.548 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_6": "dirfrag is stored in omap", 2020-11-16T20:13:27.548 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_8": "no anchor table", 2020-11-16T20:13:27.548 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_9": "file layout v2", 2020-11-16T20:13:27.548 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_10": "snaprealm v2" 2020-11-16T20:13:27.548 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.548 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.548 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_flags": { 2020-11-16T20:13:27.549 INFO:tasks.workunit.client.0.smithi189.stdout: "enable_multiple": false, 2020-11-16T20:13:27.549 INFO:tasks.workunit.client.0.smithi189.stdout: "ever_enabled_multiple": false 2020-11-16T20:13:27.549 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.549 INFO:tasks.workunit.client.0.smithi189.stdout: "standbys": [], 2020-11-16T20:13:27.549 INFO:tasks.workunit.client.0.smithi189.stdout: "filesystems": [] 2020-11-16T20:13:27.549 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.549 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_first_committed": 1, 2020-11-16T20:13:27.549 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_last_committed": 1, 2020-11-16T20:13:27.550 INFO:tasks.workunit.client.0.smithi189.stdout: "auth": { 2020-11-16T20:13:27.550 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:13:27.550 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 5, 2020-11-16T20:13:27.550 INFO:tasks.workunit.client.0.smithi189.stdout: "num_secrets": 3 2020-11-16T20:13:27.550 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.550 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 32, 2020-11-16T20:13:27.550 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_active": 32, 2020-11-16T20:13:27.551 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_unknown": 0, 2020-11-16T20:13:27.551 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osd": 3, 2020-11-16T20:13:27.551 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_sum": { 2020-11-16T20:13:27.551 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:13:27.551 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 61700, 2020-11-16T20:13:27.551 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 100, 2020-11-16T20:13:27.551 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 0, 2020-11-16T20:13:27.551 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 300, 2020-11-16T20:13:27.552 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:13:27.552 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:13:27.552 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 100, 2020-11-16T20:13:27.552 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:13:27.552 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:13:27.552 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 100, 2020-11-16T20:13:27.552 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:13:27.552 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:13:27.553 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:13:27.553 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 100, 2020-11-16T20:13:27.553 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 100, 2020-11-16T20:13:27.553 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:13:27.553 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:13:27.553 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:13:27.553 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:13:27.553 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:13:27.554 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:13:27.554 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:13:27.554 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:13:27.554 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:13:27.554 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:13:27.554 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:13:27.554 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:13:27.555 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:13:27.555 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:13:27.555 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:13:27.555 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:13:27.555 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:13:27.555 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:13:27.555 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:13:27.555 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:13:27.556 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:13:27.556 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:13:27.556 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:13:27.556 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:13:27.556 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:13:27.556 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.556 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:13:27.556 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:13:27.557 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:13:27.557 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:13:27.557 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 0, 2020-11-16T20:13:27.557 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 0, 2020-11-16T20:13:27.557 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:13:27.557 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:13:27.557 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:13:27.558 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:13:27.558 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:13:27.558 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.558 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 512, 2020-11-16T20:13:27.558 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 512, 2020-11-16T20:13:27.558 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 64, 2020-11-16T20:13:27.558 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 64, 2020-11-16T20:13:27.558 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 0 2020-11-16T20:13:27.559 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.559 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum": { 2020-11-16T20:13:27.559 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:13:27.559 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:13:27.559 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 65, 2020-11-16T20:13:27.559 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:13:27.559 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:13:27.560 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:13:27.560 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:13:27.560 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 19948, 2020-11-16T20:13:27.560 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2196, 2020-11-16T20:13:27.560 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:13:27.560 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 17726, 2020-11-16T20:13:27.560 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314552852, 2020-11-16T20:13:27.560 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:13:27.561 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:13:27.561 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322102120448, 2020-11-16T20:13:27.561 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:13:27.561 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2248704, 2020-11-16T20:13:27.561 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 500223, 2020-11-16T20:13:27.561 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:13:27.561 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:13:27.561 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:13:27.562 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:13:27.562 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 18151544 2020-11-16T20:13:27.562 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.562 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:13:27.562 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:13:27.562 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:13:27.562 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:13:27.563 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:13:27.563 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:13:27.563 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:13:27.563 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.563 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:13:27.563 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 96, 2020-11-16T20:13:27.563 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 96, 2020-11-16T20:13:27.563 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 96000000, 2020-11-16T20:13:27.564 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 96000000 2020-11-16T20:13:27.564 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.564 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:13:27.564 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:13:27.564 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.564 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum_by_class": { 2020-11-16T20:13:27.564 INFO:tasks.workunit.client.0.smithi189.stdout: "hdd": { 2020-11-16T20:13:27.564 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:13:27.565 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:13:27.565 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 65, 2020-11-16T20:13:27.565 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:13:27.565 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:13:27.565 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:13:27.565 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:13:27.565 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 19948, 2020-11-16T20:13:27.565 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2196, 2020-11-16T20:13:27.566 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:13:27.566 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 17726, 2020-11-16T20:13:27.566 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314552852, 2020-11-16T20:13:27.566 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:13:27.566 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:13:27.566 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322102120448, 2020-11-16T20:13:27.566 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:13:27.567 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2248704, 2020-11-16T20:13:27.567 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 500223, 2020-11-16T20:13:27.567 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:13:27.567 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:13:27.567 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:13:27.567 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:13:27.567 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 18151544 2020-11-16T20:13:27.567 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.568 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:13:27.568 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:13:27.568 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:13:27.568 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:13:27.568 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:13:27.568 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:13:27.568 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:13:27.568 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.569 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:13:27.569 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 96, 2020-11-16T20:13:27.569 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 96, 2020-11-16T20:13:27.569 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 96000000, 2020-11-16T20:13:27.569 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 96000000 2020-11-16T20:13:27.569 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.569 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:13:27.569 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:13:27.570 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.570 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.570 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_stats": [ 2020-11-16T20:13:27.570 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.570 INFO:tasks.workunit.client.0.smithi189.stdout: "poolid": 1, 2020-11-16T20:13:27.570 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 32, 2020-11-16T20:13:27.570 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:13:27.571 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 61700, 2020-11-16T20:13:27.571 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 100, 2020-11-16T20:13:27.571 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 0, 2020-11-16T20:13:27.571 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 300, 2020-11-16T20:13:27.571 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:13:27.571 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:13:27.571 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 100, 2020-11-16T20:13:27.571 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:13:27.572 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:13:27.572 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 100, 2020-11-16T20:13:27.572 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:13:27.572 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:13:27.572 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:13:27.572 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 100, 2020-11-16T20:13:27.572 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 100, 2020-11-16T20:13:27.573 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:13:27.573 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:13:27.573 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:13:27.573 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:13:27.573 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:13:27.573 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:13:27.573 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:13:27.573 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:13:27.574 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:13:27.574 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:13:27.574 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:13:27.574 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:13:27.574 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:13:27.574 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:13:27.574 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:13:27.575 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:13:27.575 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:13:27.575 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:13:27.575 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:13:27.575 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:13:27.575 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:13:27.575 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:13:27.575 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:13:27.576 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:13:27.576 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:13:27.576 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.576 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:13:27.576 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:13:27.576 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:13:27.576 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:13:27.576 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 1220608, 2020-11-16T20:13:27.577 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 184174, 2020-11-16T20:13:27.577 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:13:27.577 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:13:27.577 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:13:27.577 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:13:27.577 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:13:27.577 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.577 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 512, 2020-11-16T20:13:27.578 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 512, 2020-11-16T20:13:27.578 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 64, 2020-11-16T20:13:27.578 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 64, 2020-11-16T20:13:27.578 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 3 2020-11-16T20:13:27.578 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.578 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.578 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_stats": [ 2020-11-16T20:13:27.579 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.579 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:13:27.579 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 167503724554 2020-11-16T20:13:27.579 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.579 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.579 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:13:27.579 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 42949672977 2020-11-16T20:13:27.579 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.580 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.580 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:13:27.580 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 171798691850 2020-11-16T20:13:27.580 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.580 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.580 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_state": [ 2020-11-16T20:13:27.580 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.580 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "active+undersized+degraded", 2020-11-16T20:13:27.581 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 32 2020-11-16T20:13:27.581 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.581 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.581 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_osd": [ 2020-11-16T20:13:27.581 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.581 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:13:27.581 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:13:27.582 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 32, 2020-11-16T20:13:27.582 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:13:27.582 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.582 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.582 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:13:27.582 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:13:27.582 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 0, 2020-11-16T20:13:27.582 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:13:27.583 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.583 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.583 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:13:27.583 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 32, 2020-11-16T20:13:27.583 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 32, 2020-11-16T20:13:27.583 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:13:27.583 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.584 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.584 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": [ 2020-11-16T20:13:27.584 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:13:27.584 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:13:27.584 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": {} 2020-11-16T20:13:27.584 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.584 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:13:27.584 INFO:tasks.workunit.client.0.smithi189.stdout: "servicemap": { 2020-11-16T20:13:27.585 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 4, 2020-11-16T20:13:27.585 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:12:49.945544+0000", 2020-11-16T20:13:27.585 INFO:tasks.workunit.client.0.smithi189.stdout: "services": { 2020-11-16T20:13:27.585 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": { 2020-11-16T20:13:27.585 INFO:tasks.workunit.client.0.smithi189.stdout: "daemons": { 2020-11-16T20:13:27.585 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": "", 2020-11-16T20:13:27.585 INFO:tasks.workunit.client.0.smithi189.stdout: "0": { 2020-11-16T20:13:27.585 INFO:tasks.workunit.client.0.smithi189.stdout: "start_epoch": 0, 2020-11-16T20:13:27.586 INFO:tasks.workunit.client.0.smithi189.stdout: "start_stamp": "0.000000", 2020-11-16T20:13:27.586 INFO:tasks.workunit.client.0.smithi189.stdout: "gid": 0, 2020-11-16T20:13:27.586 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "(unrecognized address family 0)/0", 2020-11-16T20:13:27.586 INFO:tasks.workunit.client.0.smithi189.stdout: "metadata": {}, 2020-11-16T20:13:27.586 INFO:tasks.workunit.client.0.smithi189.stdout: "task_status": {} 2020-11-16T20:13:27.586 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.586 INFO:tasks.workunit.client.0.smithi189.stdout: "2": { 2020-11-16T20:13:27.587 INFO:tasks.workunit.client.0.smithi189.stdout: "start_epoch": 0, 2020-11-16T20:13:27.587 INFO:tasks.workunit.client.0.smithi189.stdout: "start_stamp": "0.000000", 2020-11-16T20:13:27.587 INFO:tasks.workunit.client.0.smithi189.stdout: "gid": 0, 2020-11-16T20:13:27.587 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "(unrecognized address family 0)/0", 2020-11-16T20:13:27.587 INFO:tasks.workunit.client.0.smithi189.stdout: "metadata": {}, 2020-11-16T20:13:27.587 INFO:tasks.workunit.client.0.smithi189.stdout: "task_status": {} 2020-11-16T20:13:27.587 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.587 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.588 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.588 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.588 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:13:27.588 INFO:tasks.workunit.client.0.smithi189.stdout: "paxos": { 2020-11-16T20:13:27.588 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:13:27.588 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 358, 2020-11-16T20:13:27.588 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pn": 0, 2020-11-16T20:13:27.588 INFO:tasks.workunit.client.0.smithi189.stdout: "accepted_pn": 0 2020-11-16T20:13:27.589 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:13:27.589 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:13:27.590 INFO:tasks.workunit.client.0.smithi189.stderr:report 458174950 2020-11-16T20:13:27.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2020-11-16T20:13:27.590 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2020-11-16T20:13:27.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2020-11-16T20:13:27.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2020-11-16T20:13:27.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:13:27.591 INFO:tasks.workunit.client.0.smithi189.stdout:writing non-divergent object existing_4 2020-11-16T20:13:27.790 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:13:27.790 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296348+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.790 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296667+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.790 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296226+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.790 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296303+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.791 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.301035+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.791 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.209719+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.791 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.210484+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.791 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295745+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.791 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295821+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.791 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.299731+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.791 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294133+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293842+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295030+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295320+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.298933+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294639+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294770+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294494+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293545+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.297924+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.297851+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295243+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295820+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293591+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.209633+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293479+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.298754+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.210266+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294924+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294554+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294029+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295697+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:13:27.795 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:13:27.795 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:13:27.795 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:13:27.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2020-11-16T20:13:27.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-11-16T20:13:27.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2020-11-16T20:13:27.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:13:27.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:13:27.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:13:27.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:13:27.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2020-11-16T20:13:27.928 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:13:27.928 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:13:27.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:13:27.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:13:27.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:13:28.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:13:28.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:13:28.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:13:28.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:13:28.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:13:28.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:13:28.447 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:13:28.447 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:13:28.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:13:28.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:13:28.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:13:28.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=167503724557 2020-11-16T20:13:28.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 167503724557 2020-11-16T20:13:28.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-167503724557' 2020-11-16T20:13:28.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:13:28.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:13:28.748 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-11-16T20:13:28.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T20:13:28.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T20:13:28.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T20:13:28.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:13:28.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:13:28.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691852 2020-11-16T20:13:28.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691852 2020-11-16T20:13:28.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-167503724557 2-171798691852' 2020-11-16T20:13:28.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:13:28.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-167503724557 2020-11-16T20:13:28.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:13:28.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:13:28.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-167503724557 2020-11-16T20:13:28.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:13:28.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=167503724557 2020-11-16T20:13:28.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 167503724557' 2020-11-16T20:13:28.933 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 167503724557 2020-11-16T20:13:28.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:13:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724554 -lt 167503724557 2020-11-16T20:13:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:13:30.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:13:30.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:13:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724557 -lt 167503724557 2020-11-16T20:13:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:13:30.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:13:30.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-171798691852 2020-11-16T20:13:30.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:13:30.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-171798691852 2020-11-16T20:13:30.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:13:30.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691852 2020-11-16T20:13:30.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 171798691852' 2020-11-16T20:13:30.761 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 171798691852 2020-11-16T20:13:30.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:13:31.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691852 -lt 171798691852 2020-11-16T20:13:31.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:13:31.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:31.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:31.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 32 == 0 2020-11-16T20:13:31.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:31.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:31.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:31.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:31.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:31.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:31.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:32.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:32.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:32.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:32.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:32.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:13:32.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:13:32.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:13:32.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:32.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:13:32.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:32.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:32.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:32.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:32.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:32.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:32.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:32.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:33.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:33.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:33.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:33.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:33.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:33.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:33.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:33.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:33.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:33.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:33.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:33.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:33.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2020-11-16T20:13:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-11-16T20:13:34.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:34.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:34.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:34.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:34.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:34.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:34.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:34.263 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:34.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:34.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:34.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:34.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:35.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:35.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:35.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:35.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:35.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:35.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:35.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:35.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:35.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:35.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:35.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2020-11-16T20:13:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:35.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2020-11-16T20:13:35.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:35.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:36.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:36.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:36.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:36.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:36.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:36.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:36.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:36.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:36.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:36.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:36.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:36.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:37.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:37.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:37.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2020-11-16T20:13:37.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:37.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2020-11-16T20:13:38.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:38.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:38.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:38.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:38.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:38.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:38.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:38.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:38.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:38.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:38.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:38.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:39.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:39.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:39.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:39.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:39.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:39.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:39.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:39.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:39.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:39.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:39.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:39.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2020-11-16T20:13:39.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:39.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2020-11-16T20:13:41.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:41.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:41.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:41.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:41.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:41.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:41.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:41.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:41.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:41.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:41.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:41.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:42.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:42.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:42.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:42.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:42.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:42.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:42.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:42.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:42.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:42.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:42.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:42.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2020-11-16T20:13:42.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:42.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2020-11-16T20:13:45.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:45.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:45.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:45.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:45.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:45.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:45.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:45.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:46.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:46.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:46.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:46.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:46.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:46.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:46.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:46.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:46.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:46.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:46.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:46.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:46.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:47.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:47.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:47.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2020-11-16T20:13:47.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:47.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2020-11-16T20:13:53.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:13:53.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:13:53.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:13:53.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:13:53.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:13:53.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:13:53.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:13:53.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:13:53.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:13:53.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:13:53.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:13:53.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:13:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:13:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:13:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:13:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:13:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:13:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:13:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:13:54.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:13:54.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:13:54.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:13:54.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:13:54.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2020-11-16T20:13:54.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:13:54.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2020-11-16T20:14:02.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:14:02.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:14:02.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:14:02.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:14:02.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:14:02.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:14:02.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:14:02.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:14:02.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:14:02.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:14:02.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:14:02.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:14:03.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2020-11-16T20:14:03.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:14:03.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:14:03.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:14:03.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:14:03.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:14:03.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:14:03.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:14:03.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:14:03.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:14:03.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:14:03.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2020-11-16T20:14:03.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2020-11-16T20:14:04.126 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:14:04.126 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_fingerprint": "de742bec-3667-4f2c-9f37-c103e68426dc", 2020-11-16T20:14:04.127 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "16.0.0-7219-g911cc357", 2020-11-16T20:14:04.127 INFO:tasks.workunit.client.0.smithi189.stdout: "commit": "911cc35794d871071a8bebfb0919482263b1d18b", 2020-11-16T20:14:04.127 INFO:tasks.workunit.client.0.smithi189.stdout: "timestamp": "2020-11-16T20:14:04.124311+0000", 2020-11-16T20:14:04.127 INFO:tasks.workunit.client.0.smithi189.stdout: "tag": "", 2020-11-16T20:14:04.127 INFO:tasks.workunit.client.0.smithi189.stdout: "health": { 2020-11-16T20:14:04.127 INFO:tasks.workunit.client.0.smithi189.stdout: "status": "HEALTH_WARN", 2020-11-16T20:14:04.127 INFO:tasks.workunit.client.0.smithi189.stdout: "checks": { 2020-11-16T20:14:04.128 INFO:tasks.workunit.client.0.smithi189.stdout: "OSDMAP_FLAGS": { 2020-11-16T20:14:04.128 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:14:04.128 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:14:04.128 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "nodown,noout,noin flag(s) set", 2020-11-16T20:14:04.128 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 17 2020-11-16T20:14:04.128 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.128 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [], 2020-11-16T20:14:04.128 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:14:04.129 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.129 INFO:tasks.workunit.client.0.smithi189.stdout: "OSD_DOWN": { 2020-11-16T20:14:04.129 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:14:04.129 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:14:04.129 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 osds down", 2020-11-16T20:14:04.129 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:14:04.129 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.130 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:14:04.130 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.130 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "osd.1 (root=default,host=smithi189) is down" 2020-11-16T20:14:04.130 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.130 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.130 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:14:04.130 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.131 INFO:tasks.workunit.client.0.smithi189.stdout: "PG_DEGRADED": { 2020-11-16T20:14:04.131 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:14:04.131 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:14:04.131 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2020-11-16T20:14:04.131 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 64 2020-11-16T20:14:04.131 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.131 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:14:04.132 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.132 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.0 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.132 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.132 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.132 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.132 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.132 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.132 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.2 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.133 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.133 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.133 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.3 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.133 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.133 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.133 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.4 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.133 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.134 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.134 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.5 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.134 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.134 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.134 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.6 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.134 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.134 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.135 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.7 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.135 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.135 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.135 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.8 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.135 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.135 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.135 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.9 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.136 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.136 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.136 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.a is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.136 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.136 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.136 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.b is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.136 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.136 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.137 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.c is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.137 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.137 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.137 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.d is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.137 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.137 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.137 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.e is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.138 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.138 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.138 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.f is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.138 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.138 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.138 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.10 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.138 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.139 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.139 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.11 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.139 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.139 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.139 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.12 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.139 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.139 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.139 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.13 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.140 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.140 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.140 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.14 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.140 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.140 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.140 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.15 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.140 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.141 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.141 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.16 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.141 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.141 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.141 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.17 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.141 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.141 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.141 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.18 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.142 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.142 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.142 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.19 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.142 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.142 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.142 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1a is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.142 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.143 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.143 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1b is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.143 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.143 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.143 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1c is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.143 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.143 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.144 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1d is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.144 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.144 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.144 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1e is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.144 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.144 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.144 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.1f is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2020-11-16T20:14:04.144 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.145 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.145 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:14:04.145 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.145 INFO:tasks.workunit.client.0.smithi189.stdout: "POOL_APP_NOT_ENABLED": { 2020-11-16T20:14:04.145 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:14:04.145 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:14:04.145 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 pool(s) do not have an application enabled", 2020-11-16T20:14:04.146 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:14:04.146 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.146 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:14:04.146 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.146 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "application not enabled on pool 'test'" 2020-11-16T20:14:04.146 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.146 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.146 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-11-16T20:14:04.147 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.147 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.147 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:14:04.147 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.147 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.147 INFO:tasks.workunit.client.0.smithi189.stdout: "mutes": [] 2020-11-16T20:14:04.147 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.148 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_first_committed": 1, 2020-11-16T20:14:04.148 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_last_committed": 1, 2020-11-16T20:14:04.148 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap": { 2020-11-16T20:14:04.148 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:14:04.148 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:14:04.148 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:11:31.065413Z", 2020-11-16T20:14:04.148 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:11:31.065413Z", 2020-11-16T20:14:04.148 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release": 16, 2020-11-16T20:14:04.149 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release_name": "pacific", 2020-11-16T20:14:04.149 INFO:tasks.workunit.client.0.smithi189.stdout: "election_strategy": 1, 2020-11-16T20:14:04.149 INFO:tasks.workunit.client.0.smithi189.stdout: "disallowed_leaders: ": "", 2020-11-16T20:14:04.149 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": false, 2020-11-16T20:14:04.149 INFO:tasks.workunit.client.0.smithi189.stdout: "features": { 2020-11-16T20:14:04.149 INFO:tasks.workunit.client.0.smithi189.stdout: "persistent": [ 2020-11-16T20:14:04.149 INFO:tasks.workunit.client.0.smithi189.stdout: "kraken", 2020-11-16T20:14:04.150 INFO:tasks.workunit.client.0.smithi189.stdout: "luminous", 2020-11-16T20:14:04.150 INFO:tasks.workunit.client.0.smithi189.stdout: "mimic", 2020-11-16T20:14:04.150 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap-prune", 2020-11-16T20:14:04.150 INFO:tasks.workunit.client.0.smithi189.stdout: "nautilus", 2020-11-16T20:14:04.150 INFO:tasks.workunit.client.0.smithi189.stdout: "octopus", 2020-11-16T20:14:04.150 INFO:tasks.workunit.client.0.smithi189.stdout: "pacific", 2020-11-16T20:14:04.150 INFO:tasks.workunit.client.0.smithi189.stdout: "elector-pinging" 2020-11-16T20:14:04.150 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.151 INFO:tasks.workunit.client.0.smithi189.stdout: "optional": [] 2020-11-16T20:14:04.151 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.151 INFO:tasks.workunit.client.0.smithi189.stdout: "mons": [ 2020-11-16T20:14:04.151 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.151 INFO:tasks.workunit.client.0.smithi189.stdout: "rank": 0, 2020-11-16T20:14:04.151 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "a", 2020-11-16T20:14:04.151 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:14:04.152 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.152 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.152 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.152 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115", 2020-11-16T20:14:04.152 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 0 2020-11-16T20:14:04.152 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.152 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.152 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.153 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115/0", 2020-11-16T20:14:04.153 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:7115/0", 2020-11-16T20:14:04.153 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 0, 2020-11-16T20:14:04.153 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 0, 2020-11-16T20:14:04.153 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_location": "{}" 2020-11-16T20:14:04.153 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.153 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.153 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.154 INFO:tasks.workunit.client.0.smithi189.stdout: "quorum": [ 2020-11-16T20:14:04.154 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2020-11-16T20:14:04.154 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.154 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap": { 2020-11-16T20:14:04.154 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 45, 2020-11-16T20:14:04.154 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:14:04.154 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:11:31.869939+0000", 2020-11-16T20:14:04.155 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:12:52.797777+0000", 2020-11-16T20:14:04.155 INFO:tasks.workunit.client.0.smithi189.stdout: "last_up_change": "2020-11-16T20:12:50.890228+0000", 2020-11-16T20:14:04.155 INFO:tasks.workunit.client.0.smithi189.stdout: "last_in_change": "2020-11-16T20:12:52.354351+0000", 2020-11-16T20:14:04.155 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-11-16T20:14:04.155 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_num": 5800384, 2020-11-16T20:14:04.155 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_set": [ 2020-11-16T20:14:04.155 INFO:tasks.workunit.client.0.smithi189.stdout: "nodown", 2020-11-16T20:14:04.155 INFO:tasks.workunit.client.0.smithi189.stdout: "noin", 2020-11-16T20:14:04.156 INFO:tasks.workunit.client.0.smithi189.stdout: "noout", 2020-11-16T20:14:04.156 INFO:tasks.workunit.client.0.smithi189.stdout: "pglog_hardlimit", 2020-11-16T20:14:04.156 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snapdirs", 2020-11-16T20:14:04.156 INFO:tasks.workunit.client.0.smithi189.stdout: "recovery_deletes", 2020-11-16T20:14:04.156 INFO:tasks.workunit.client.0.smithi189.stdout: "sortbitwise" 2020-11-16T20:14:04.156 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.156 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_version": 7, 2020-11-16T20:14:04.157 INFO:tasks.workunit.client.0.smithi189.stdout: "full_ratio": 0.99000000953674316, 2020-11-16T20:14:04.157 INFO:tasks.workunit.client.0.smithi189.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-11-16T20:14:04.157 INFO:tasks.workunit.client.0.smithi189.stdout: "nearfull_ratio": 0.85000002384185791, 2020-11-16T20:14:04.157 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_snapshot": "", 2020-11-16T20:14:04.157 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_max": 1, 2020-11-16T20:14:04.157 INFO:tasks.workunit.client.0.smithi189.stdout: "max_osd": 3, 2020-11-16T20:14:04.157 INFO:tasks.workunit.client.0.smithi189.stdout: "require_min_compat_client": "luminous", 2020-11-16T20:14:04.157 INFO:tasks.workunit.client.0.smithi189.stdout: "min_compat_client": "jewel", 2020-11-16T20:14:04.158 INFO:tasks.workunit.client.0.smithi189.stdout: "require_osd_release": "pacific", 2020-11-16T20:14:04.158 INFO:tasks.workunit.client.0.smithi189.stdout: "pools": [ 2020-11-16T20:14:04.158 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.158 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:14:04.158 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_name": "test", 2020-11-16T20:14:04.158 INFO:tasks.workunit.client.0.smithi189.stdout: "create_time": "2020-11-16T20:11:58.751915+0000", 2020-11-16T20:14:04.158 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": 1, 2020-11-16T20:14:04.159 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_names": "hashpspool", 2020-11-16T20:14:04.159 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:14:04.159 INFO:tasks.workunit.client.0.smithi189.stdout: "size": 3, 2020-11-16T20:14:04.159 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 2, 2020-11-16T20:14:04.159 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_rule": 0, 2020-11-16T20:14:04.159 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_count": 0, 2020-11-16T20:14:04.159 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_target": 0, 2020-11-16T20:14:04.160 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_barrier": 0, 2020-11-16T20:14:04.160 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2020-11-16T20:14:04.160 INFO:tasks.workunit.client.0.smithi189.stdout: "object_hash": 2, 2020-11-16T20:14:04.160 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_autoscale_mode": "on", 2020-11-16T20:14:04.160 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num": 32, 2020-11-16T20:14:04.160 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num": 1, 2020-11-16T20:14:04.160 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num_target": 32, 2020-11-16T20:14:04.160 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_target": 32, 2020-11-16T20:14:04.161 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_pending": 32, 2020-11-16T20:14:04.161 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pg_merge_meta": { 2020-11-16T20:14:04.161 INFO:tasks.workunit.client.0.smithi189.stdout: "source_pgid": "0.0", 2020-11-16T20:14:04.161 INFO:tasks.workunit.client.0.smithi189.stdout: "ready_epoch": 0, 2020-11-16T20:14:04.161 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_started": 0, 2020-11-16T20:14:04.161 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": 0, 2020-11-16T20:14:04.161 INFO:tasks.workunit.client.0.smithi189.stdout: "source_version": "0'0", 2020-11-16T20:14:04.161 INFO:tasks.workunit.client.0.smithi189.stdout: "target_version": "0'0" 2020-11-16T20:14:04.162 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.162 INFO:tasks.workunit.client.0.smithi189.stdout: "last_change": "32", 2020-11-16T20:14:04.162 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend": "0", 2020-11-16T20:14:04.162 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_prenautilus": "0", 2020-11-16T20:14:04.162 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_preluminous": "32", 2020-11-16T20:14:04.162 INFO:tasks.workunit.client.0.smithi189.stdout: "auid": 0, 2020-11-16T20:14:04.162 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_mode": "selfmanaged", 2020-11-16T20:14:04.163 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_seq": 0, 2020-11-16T20:14:04.163 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_epoch": 0, 2020-11-16T20:14:04.163 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_snaps": [], 2020-11-16T20:14:04.163 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps": "[]", 2020-11-16T20:14:04.163 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_bytes": 0, 2020-11-16T20:14:04.163 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_objects": 0, 2020-11-16T20:14:04.163 INFO:tasks.workunit.client.0.smithi189.stdout: "tiers": [], 2020-11-16T20:14:04.163 INFO:tasks.workunit.client.0.smithi189.stdout: "tier_of": -1, 2020-11-16T20:14:04.164 INFO:tasks.workunit.client.0.smithi189.stdout: "read_tier": -1, 2020-11-16T20:14:04.164 INFO:tasks.workunit.client.0.smithi189.stdout: "write_tier": -1, 2020-11-16T20:14:04.164 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_mode": "none", 2020-11-16T20:14:04.164 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_bytes": 0, 2020-11-16T20:14:04.164 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_objects": 0, 2020-11-16T20:14:04.164 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-11-16T20:14:04.164 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-11-16T20:14:04.165 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_full_ratio_micro": 800000, 2020-11-16T20:14:04.165 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_flush_age": 0, 2020-11-16T20:14:04.165 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_evict_age": 0, 2020-11-16T20:14:04.165 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profile": "", 2020-11-16T20:14:04.165 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_params": { 2020-11-16T20:14:04.165 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "none" 2020-11-16T20:14:04.165 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.165 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_period": 0, 2020-11-16T20:14:04.166 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_count": 0, 2020-11-16T20:14:04.166 INFO:tasks.workunit.client.0.smithi189.stdout: "use_gmt_hitset": true, 2020-11-16T20:14:04.166 INFO:tasks.workunit.client.0.smithi189.stdout: "min_read_recency_for_promote": 0, 2020-11-16T20:14:04.166 INFO:tasks.workunit.client.0.smithi189.stdout: "min_write_recency_for_promote": 0, 2020-11-16T20:14:04.166 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_grade_decay_rate": 0, 2020-11-16T20:14:04.166 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_search_last_n": 0, 2020-11-16T20:14:04.166 INFO:tasks.workunit.client.0.smithi189.stdout: "grade_table": [], 2020-11-16T20:14:04.167 INFO:tasks.workunit.client.0.smithi189.stdout: "stripe_width": 0, 2020-11-16T20:14:04.167 INFO:tasks.workunit.client.0.smithi189.stdout: "expected_num_objects": 0, 2020-11-16T20:14:04.167 INFO:tasks.workunit.client.0.smithi189.stdout: "fast_read": false, 2020-11-16T20:14:04.167 INFO:tasks.workunit.client.0.smithi189.stdout: "options": {}, 2020-11-16T20:14:04.167 INFO:tasks.workunit.client.0.smithi189.stdout: "application_metadata": {} 2020-11-16T20:14:04.168 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.168 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.168 INFO:tasks.workunit.client.0.smithi189.stdout: "osds": [ 2020-11-16T20:14:04.168 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.168 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:14:04.169 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "42e192ce-7576-4c00-87e5-e211cf1e1011", 2020-11-16T20:14:04.169 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:14:04.169 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:14:04.169 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:14:04.170 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:14:04.170 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 5, 2020-11-16T20:14:04.170 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 30, 2020-11-16T20:14:04.170 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 39, 2020-11-16T20:14:04.170 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 44, 2020-11-16T20:14:04.171 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 34, 2020-11-16T20:14:04.171 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:14:04.171 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:14:04.171 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.171 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.171 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.171 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6802", 2020-11-16T20:14:04.172 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:14:04.172 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.172 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.172 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.172 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6803", 2020-11-16T20:14:04.172 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:14:04.173 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.173 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.173 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.173 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:14:04.173 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.173 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.174 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.174 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6804", 2020-11-16T20:14:04.174 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:14:04.174 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.174 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.174 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.175 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6805", 2020-11-16T20:14:04.175 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:14:04.175 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.175 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.175 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.175 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:14:04.176 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.176 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.176 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.176 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6808", 2020-11-16T20:14:04.176 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:14:04.176 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.177 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.177 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.177 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6809", 2020-11-16T20:14:04.177 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:14:04.177 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.177 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.178 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.178 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:14:04.178 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.178 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.178 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.178 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6806", 2020-11-16T20:14:04.179 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:14:04.179 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.179 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.179 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.179 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6807", 2020-11-16T20:14:04.180 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41211 2020-11-16T20:14:04.180 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.180 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.180 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.180 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6803/41211", 2020-11-16T20:14:04.181 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6805/41211", 2020-11-16T20:14:04.181 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6809/41211", 2020-11-16T20:14:04.181 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6807/41211", 2020-11-16T20:14:04.181 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:14:04.181 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:14:04.182 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:14:04.182 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.182 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.182 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.183 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:14:04.183 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "dc40a541-b8cd-4c3b-81a7-0d36bc669c17", 2020-11-16T20:14:04.183 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 0, 2020-11-16T20:14:04.183 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:14:04.183 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:14:04.183 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:14:04.184 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 0, 2020-11-16T20:14:04.184 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 0, 2020-11-16T20:14:04.184 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 10, 2020-11-16T20:14:04.184 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 32, 2020-11-16T20:14:04.184 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 35, 2020-11-16T20:14:04.184 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:14:04.185 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:14:04.185 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.185 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.185 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.186 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:14:04.186 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:14:04.186 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.186 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.186 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.187 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:14:04.187 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:14:04.187 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.187 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.187 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.188 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:14:04.188 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.188 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.188 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.188 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:14:04.188 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:14:04.189 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.189 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.189 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.189 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:14:04.190 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:14:04.190 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.190 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.190 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.190 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:14:04.191 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.191 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.191 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.191 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:14:04.191 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:14:04.192 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.192 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.192 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.192 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:14:04.192 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:14:04.192 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.193 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.193 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.193 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:14:04.193 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.193 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.193 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.194 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:14:04.194 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:14:04.194 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.194 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.195 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.195 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:14:04.195 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 36267 2020-11-16T20:14:04.195 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.195 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.196 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.196 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/36267", 2020-11-16T20:14:04.196 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/36267", 2020-11-16T20:14:04.196 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/36267", 2020-11-16T20:14:04.196 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/36267", 2020-11-16T20:14:04.197 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:14:04.197 INFO:tasks.workunit.client.0.smithi189.stdout: "exists" 2020-11-16T20:14:04.197 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.197 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.197 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.197 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:14:04.198 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "ca0542d6-a1d5-4f3f-8ddc-4d1c683befc4", 2020-11-16T20:14:04.198 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:14:04.198 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:14:04.198 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:14:04.198 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:14:04.199 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 14, 2020-11-16T20:14:04.199 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 30, 2020-11-16T20:14:04.199 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 40, 2020-11-16T20:14:04.199 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 0, 2020-11-16T20:14:04.199 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 36, 2020-11-16T20:14:04.200 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:14:04.200 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:14:04.200 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.200 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.200 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.200 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:14:04.200 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:14:04.201 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.201 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.201 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.201 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:14:04.201 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:14:04.201 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.201 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.202 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.202 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:14:04.202 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.202 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.202 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.202 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:14:04.203 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:14:04.203 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.203 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.203 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.203 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:14:04.203 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:14:04.203 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.204 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.204 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.204 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:14:04.204 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.204 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.204 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.205 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:14:04.205 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:14:04.205 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.205 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.205 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.205 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:14:04.206 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:14:04.206 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.206 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.206 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.206 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:14:04.206 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:14:04.206 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.207 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:14:04.207 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:14:04.207 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:14:04.207 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.207 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.207 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:14:04.207 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:14:04.208 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 41610 2020-11-16T20:14:04.208 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.208 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.208 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.208 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/41610", 2020-11-16T20:14:04.208 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/41610", 2020-11-16T20:14:04.208 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/41610", 2020-11-16T20:14:04.209 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/41610", 2020-11-16T20:14:04.209 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:14:04.209 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:14:04.209 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:14:04.209 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.209 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.209 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.210 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_xinfo": [ 2020-11-16T20:14:04.210 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.210 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:14:04.210 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:12:43.647800+0000", 2020-11-16T20:14:04.210 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:14:04.210 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:14:04.210 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:14:04.211 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:14:04.211 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:11:40.042383+0000", 2020-11-16T20:14:04.211 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:14:04.211 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.211 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.211 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:14:04.211 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:12:44.237289+0000", 2020-11-16T20:14:04.212 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:14:04.212 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:14:04.212 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:14:04.212 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:14:04.212 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:11:48.292944+0000", 2020-11-16T20:14:04.212 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:14:04.212 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.213 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.213 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:14:04.213 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:12:44.821561+0000", 2020-11-16T20:14:04.213 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:14:04.213 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:14:04.213 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:14:04.214 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:14:04.214 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:11:56.248668+0000", 2020-11-16T20:14:04.214 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:14:04.214 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.214 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.214 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap": [], 2020-11-16T20:14:04.214 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap_items": [], 2020-11-16T20:14:04.215 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_temp": [], 2020-11-16T20:14:04.215 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_temp": [], 2020-11-16T20:14:04.215 INFO:tasks.workunit.client.0.smithi189.stdout: "blocklist": {}, 2020-11-16T20:14:04.215 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profiles": { 2020-11-16T20:14:04.215 INFO:tasks.workunit.client.0.smithi189.stdout: "default": { 2020-11-16T20:14:04.215 INFO:tasks.workunit.client.0.smithi189.stdout: "k": "2", 2020-11-16T20:14:04.216 INFO:tasks.workunit.client.0.smithi189.stdout: "m": "2", 2020-11-16T20:14:04.216 INFO:tasks.workunit.client.0.smithi189.stdout: "plugin": "jerasure", 2020-11-16T20:14:04.216 INFO:tasks.workunit.client.0.smithi189.stdout: "technique": "reed_sol_van" 2020-11-16T20:14:04.216 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.216 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.216 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps_queue": [], 2020-11-16T20:14:04.216 INFO:tasks.workunit.client.0.smithi189.stdout: "new_removed_snaps": [], 2020-11-16T20:14:04.217 INFO:tasks.workunit.client.0.smithi189.stdout: "new_purged_snaps": [], 2020-11-16T20:14:04.217 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_node_flags": {}, 2020-11-16T20:14:04.217 INFO:tasks.workunit.client.0.smithi189.stdout: "device_class_flags": {}, 2020-11-16T20:14:04.217 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": { 2020-11-16T20:14:04.217 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_enabled": false, 2020-11-16T20:14:04.217 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_bucket_count": 0, 2020-11-16T20:14:04.217 INFO:tasks.workunit.client.0.smithi189.stdout: "degraded_stretch_mode": 0, 2020-11-16T20:14:04.218 INFO:tasks.workunit.client.0.smithi189.stdout: "recovering_stretch_mode": 0, 2020-11-16T20:14:04.218 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_bucket": 0 2020-11-16T20:14:04.218 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.218 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.218 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_metadata": [ 2020-11-16T20:14:04.218 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.219 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:14:04.219 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:14:04.219 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6804/41211,v1:127.0.0.1:6805/41211]", 2020-11-16T20:14:04.219 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:14:04.219 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:14:04.219 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:14:04.219 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:14:04.220 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:14:04.220 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:14:04.220 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:14:04.220 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:14:04.220 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:14:04.220 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-11-16T20:14:04.220 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:14:04.221 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:14:04.221 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:14:04.221 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:14:04.221 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:14:04.221 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:14:04.221 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:14:04.221 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:14:04.221 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:14:04.222 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:14:04.222 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:14:04.222 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:14:04.222 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:14:04.222 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:14:04.223 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:14:04.223 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6802/41211,v1:127.0.0.1:6803/41211]", 2020-11-16T20:14:04.223 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:14:04.223 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/41211,v1:127.0.0.1:6809/41211]", 2020-11-16T20:14:04.223 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/41211,v1:127.0.0.1:6807/41211]", 2020-11-16T20:14:04.223 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:14:04.223 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:14:04.224 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:14:04.224 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:14:04.224 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:14:04.224 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:14:04.224 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:14:04.224 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:14:04.225 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:14:04.225 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/0", 2020-11-16T20:14:04.225 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:14:04.225 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:14:04.225 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:14:04.225 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.225 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.226 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:14:04.226 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:14:04.226 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/36267,v1:127.0.0.1:6813/36267]", 2020-11-16T20:14:04.226 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:14:04.226 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:14:04.226 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:14:04.226 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:14:04.227 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:14:04.227 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:14:04.227 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:14:04.227 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:14:04.227 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:14:04.227 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-11-16T20:14:04.228 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:14:04.228 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:14:04.228 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:14:04.228 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:14:04.228 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:14:04.228 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:14:04.228 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:14:04.229 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:14:04.229 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:14:04.229 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:14:04.229 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:14:04.229 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:14:04.229 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:14:04.230 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:14:04.230 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:14:04.230 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/36267,v1:127.0.0.1:6811/36267]", 2020-11-16T20:14:04.230 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:14:04.230 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/36267,v1:127.0.0.1:6817/36267]", 2020-11-16T20:14:04.230 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/36267,v1:127.0.0.1:6815/36267]", 2020-11-16T20:14:04.230 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:14:04.231 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:14:04.231 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:14:04.231 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:14:04.231 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:14:04.231 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:14:04.231 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:14:04.232 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:14:04.232 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:14:04.232 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/1", 2020-11-16T20:14:04.232 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:14:04.232 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:14:04.232 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:14:04.233 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.233 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.233 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:14:04.233 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:14:04.233 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/41610,v1:127.0.0.1:6813/41610]", 2020-11-16T20:14:04.233 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:14:04.233 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:14:04.234 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:14:04.234 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:14:04.234 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:14:04.234 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:14:04.234 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:14:04.234 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:14:04.234 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:14:04.235 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-11-16T20:14:04.235 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:14:04.235 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:14:04.235 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:14:04.235 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:14:04.235 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:14:04.236 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:14:04.236 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:14:04.236 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:14:04.236 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:14:04.236 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:14:04.236 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:14:04.236 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:14:04.237 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:14:04.237 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:14:04.237 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:14:04.237 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/41610,v1:127.0.0.1:6811/41610]", 2020-11-16T20:14:04.237 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:14:04.237 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/41610,v1:127.0.0.1:6817/41610]", 2020-11-16T20:14:04.238 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/41610,v1:127.0.0.1:6815/41610]", 2020-11-16T20:14:04.238 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:14:04.238 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:14:04.238 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:14:04.238 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:14:04.238 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:14:04.238 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:14:04.239 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:14:04.239 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:14:04.239 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:14:04.239 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/2", 2020-11-16T20:14:04.239 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:14:04.239 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:14:04.240 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:14:04.240 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.240 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.240 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_clean_epochs": { 2020-11-16T20:14:04.240 INFO:tasks.workunit.client.0.smithi189.stdout: "min_last_epoch_clean": 0, 2020-11-16T20:14:04.240 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": { 2020-11-16T20:14:04.240 INFO:tasks.workunit.client.0.smithi189.stdout: "per_pool": [] 2020-11-16T20:14:04.241 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.241 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_epochs": [ 2020-11-16T20:14:04.241 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.241 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:14:04.241 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 39 2020-11-16T20:14:04.241 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.241 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.242 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:14:04.242 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 10 2020-11-16T20:14:04.242 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.242 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.242 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:14:04.242 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 40 2020-11-16T20:14:04.243 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.243 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.243 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.243 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_first_committed": 1, 2020-11-16T20:14:04.243 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_last_committed": 45, 2020-11-16T20:14:04.243 INFO:tasks.workunit.client.0.smithi189.stdout: "crushmap": { 2020-11-16T20:14:04.243 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": [ 2020-11-16T20:14:04.244 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.244 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:14:04.244 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.0", 2020-11-16T20:14:04.244 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:14:04.244 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.244 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.244 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:14:04.245 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.1", 2020-11-16T20:14:04.245 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:14:04.245 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.245 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.245 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:14:04.245 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.2", 2020-11-16T20:14:04.245 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:14:04.246 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.246 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.246 INFO:tasks.workunit.client.0.smithi189.stdout: "types": [ 2020-11-16T20:14:04.246 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.246 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 0, 2020-11-16T20:14:04.246 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd" 2020-11-16T20:14:04.247 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.247 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.247 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:14:04.247 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "host" 2020-11-16T20:14:04.247 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.247 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.248 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 2, 2020-11-16T20:14:04.248 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "chassis" 2020-11-16T20:14:04.248 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.248 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.248 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 3, 2020-11-16T20:14:04.248 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "rack" 2020-11-16T20:14:04.249 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.249 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.249 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 4, 2020-11-16T20:14:04.249 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "row" 2020-11-16T20:14:04.249 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.249 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.250 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 5, 2020-11-16T20:14:04.250 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pdu" 2020-11-16T20:14:04.250 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.250 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.250 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 6, 2020-11-16T20:14:04.250 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pod" 2020-11-16T20:14:04.250 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.251 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.251 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 7, 2020-11-16T20:14:04.251 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "room" 2020-11-16T20:14:04.251 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.251 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.251 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 8, 2020-11-16T20:14:04.251 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "datacenter" 2020-11-16T20:14:04.252 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.252 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.252 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 9, 2020-11-16T20:14:04.252 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "zone" 2020-11-16T20:14:04.252 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.252 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.252 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 10, 2020-11-16T20:14:04.253 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "region" 2020-11-16T20:14:04.253 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.253 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.253 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:14:04.253 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "root" 2020-11-16T20:14:04.253 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.254 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.254 INFO:tasks.workunit.client.0.smithi189.stdout: "buckets": [ 2020-11-16T20:14:04.254 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.254 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -1, 2020-11-16T20:14:04.254 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default", 2020-11-16T20:14:04.254 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:14:04.254 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:14:04.255 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:14:04.255 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:14:04.255 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:14:04.255 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:14:04.255 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.255 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:14:04.256 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:14:04.256 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:14:04.256 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.256 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.256 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.256 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.256 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -2, 2020-11-16T20:14:04.257 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default~hdd", 2020-11-16T20:14:04.257 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:14:04.257 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:14:04.257 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:14:04.257 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:14:04.258 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:14:04.258 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:14:04.258 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.258 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:14:04.258 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:14:04.258 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:14:04.258 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.259 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.259 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.259 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.259 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:14:04.259 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189", 2020-11-16T20:14:04.259 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:14:04.260 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:14:04.260 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:14:04.260 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:14:04.260 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:14:04.260 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:14:04.260 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.261 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:14:04.261 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:14:04.261 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:14:04.261 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.261 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.261 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:14:04.261 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:14:04.262 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:14:04.262 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.262 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.262 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:14:04.262 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:14:04.262 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:14:04.262 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.263 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.263 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.263 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.263 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:14:04.263 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189~hdd", 2020-11-16T20:14:04.263 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:14:04.263 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:14:04.264 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:14:04.264 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:14:04.264 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:14:04.264 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:14:04.264 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.264 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:14:04.264 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:14:04.265 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:14:04.265 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.265 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.265 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:14:04.265 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:14:04.265 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:14:04.266 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.266 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.266 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:14:04.266 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:14:04.266 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:14:04.266 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.266 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.267 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.267 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.267 INFO:tasks.workunit.client.0.smithi189.stdout: "rules": [ 2020-11-16T20:14:04.267 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.267 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 0, 2020-11-16T20:14:04.267 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "replicated_rule", 2020-11-16T20:14:04.268 INFO:tasks.workunit.client.0.smithi189.stdout: "ruleset": 0, 2020-11-16T20:14:04.268 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:14:04.268 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 1, 2020-11-16T20:14:04.268 INFO:tasks.workunit.client.0.smithi189.stdout: "max_size": 10, 2020-11-16T20:14:04.268 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2020-11-16T20:14:04.268 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.269 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2020-11-16T20:14:04.269 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2020-11-16T20:14:04.269 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2020-11-16T20:14:04.269 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.269 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.269 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_firstn", 2020-11-16T20:14:04.269 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2020-11-16T20:14:04.270 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2020-11-16T20:14:04.270 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.270 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.270 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2020-11-16T20:14:04.270 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.270 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:14:04.270 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.271 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.271 INFO:tasks.workunit.client.0.smithi189.stdout: "tunables": { 2020-11-16T20:14:04.271 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_tries": 0, 2020-11-16T20:14:04.271 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_fallback_tries": 0, 2020-11-16T20:14:04.271 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_total_tries": 50, 2020-11-16T20:14:04.271 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_descend_once": 1, 2020-11-16T20:14:04.271 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_vary_r": 1, 2020-11-16T20:14:04.272 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_stable": 1, 2020-11-16T20:14:04.272 INFO:tasks.workunit.client.0.smithi189.stdout: "straw_calc_version": 1, 2020-11-16T20:14:04.272 INFO:tasks.workunit.client.0.smithi189.stdout: "allowed_bucket_algs": 54, 2020-11-16T20:14:04.272 INFO:tasks.workunit.client.0.smithi189.stdout: "profile": "jewel", 2020-11-16T20:14:04.272 INFO:tasks.workunit.client.0.smithi189.stdout: "optimal_tunables": 1, 2020-11-16T20:14:04.272 INFO:tasks.workunit.client.0.smithi189.stdout: "legacy_tunables": 0, 2020-11-16T20:14:04.273 INFO:tasks.workunit.client.0.smithi189.stdout: "minimum_required_version": "jewel", 2020-11-16T20:14:04.273 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables": 1, 2020-11-16T20:14:04.273 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables2": 1, 2020-11-16T20:14:04.273 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v2_rules": 0, 2020-11-16T20:14:04.273 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables3": 1, 2020-11-16T20:14:04.273 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v3_rules": 0, 2020-11-16T20:14:04.273 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v4_buckets": 1, 2020-11-16T20:14:04.274 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables5": 1, 2020-11-16T20:14:04.274 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v5_rules": 0 2020-11-16T20:14:04.274 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.274 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_args": {} 2020-11-16T20:14:04.274 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.274 INFO:tasks.workunit.client.0.smithi189.stdout: "fsmap": { 2020-11-16T20:14:04.275 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:14:04.275 INFO:tasks.workunit.client.0.smithi189.stdout: "default_fscid": -1, 2020-11-16T20:14:04.275 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": { 2020-11-16T20:14:04.275 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": {}, 2020-11-16T20:14:04.275 INFO:tasks.workunit.client.0.smithi189.stdout: "ro_compat": {}, 2020-11-16T20:14:04.275 INFO:tasks.workunit.client.0.smithi189.stdout: "incompat": { 2020-11-16T20:14:04.275 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_1": "base v0.20", 2020-11-16T20:14:04.276 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_2": "client writeable ranges", 2020-11-16T20:14:04.276 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_3": "default file layouts on dirs", 2020-11-16T20:14:04.276 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_4": "dir inode in separate object", 2020-11-16T20:14:04.276 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_5": "mds uses versioned encoding", 2020-11-16T20:14:04.276 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_6": "dirfrag is stored in omap", 2020-11-16T20:14:04.276 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_8": "no anchor table", 2020-11-16T20:14:04.277 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_9": "file layout v2", 2020-11-16T20:14:04.277 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_10": "snaprealm v2" 2020-11-16T20:14:04.277 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.277 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.277 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_flags": { 2020-11-16T20:14:04.277 INFO:tasks.workunit.client.0.smithi189.stdout: "enable_multiple": false, 2020-11-16T20:14:04.277 INFO:tasks.workunit.client.0.smithi189.stdout: "ever_enabled_multiple": false 2020-11-16T20:14:04.278 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.278 INFO:tasks.workunit.client.0.smithi189.stdout: "standbys": [], 2020-11-16T20:14:04.278 INFO:tasks.workunit.client.0.smithi189.stdout: "filesystems": [] 2020-11-16T20:14:04.278 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.278 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_first_committed": 1, 2020-11-16T20:14:04.278 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_last_committed": 1, 2020-11-16T20:14:04.278 INFO:tasks.workunit.client.0.smithi189.stdout: "auth": { 2020-11-16T20:14:04.278 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:14:04.279 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 5, 2020-11-16T20:14:04.279 INFO:tasks.workunit.client.0.smithi189.stdout: "num_secrets": 3 2020-11-16T20:14:04.279 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.279 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 32, 2020-11-16T20:14:04.279 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_active": 32, 2020-11-16T20:14:04.279 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_unknown": 0, 2020-11-16T20:14:04.279 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osd": 3, 2020-11-16T20:14:04.280 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_sum": { 2020-11-16T20:14:04.280 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:14:04.280 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 61854, 2020-11-16T20:14:04.280 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 100, 2020-11-16T20:14:04.280 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 0, 2020-11-16T20:14:04.280 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 300, 2020-11-16T20:14:04.281 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:14:04.281 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:14:04.281 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 100, 2020-11-16T20:14:04.281 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:14:04.281 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:14:04.281 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 100, 2020-11-16T20:14:04.281 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:14:04.282 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:14:04.282 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:14:04.282 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 101, 2020-11-16T20:14:04.282 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 101, 2020-11-16T20:14:04.282 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:14:04.282 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:14:04.283 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:14:04.283 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:14:04.283 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:14:04.283 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:14:04.283 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:14:04.283 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:14:04.284 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:14:04.284 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:14:04.284 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:14:04.284 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:14:04.284 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:14:04.284 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:14:04.284 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:14:04.285 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:14:04.285 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:14:04.285 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:14:04.285 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:14:04.285 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:14:04.285 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:14:04.286 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:14:04.286 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:14:04.286 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:14:04.286 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:14:04.286 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.286 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:14:04.286 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:14:04.287 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:14:04.287 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:14:04.287 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 0, 2020-11-16T20:14:04.287 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 0, 2020-11-16T20:14:04.287 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:14:04.287 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:14:04.288 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:14:04.288 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:14:04.288 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:14:04.288 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.288 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 513, 2020-11-16T20:14:04.288 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 513, 2020-11-16T20:14:04.288 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 64, 2020-11-16T20:14:04.289 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 64, 2020-11-16T20:14:04.289 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 0 2020-11-16T20:14:04.289 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.289 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum": { 2020-11-16T20:14:04.289 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:14:04.289 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:14:04.290 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 65, 2020-11-16T20:14:04.290 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:14:04.290 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:14:04.290 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:14:04.290 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:14:04.290 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 19948, 2020-11-16T20:14:04.291 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2196, 2020-11-16T20:14:04.291 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:14:04.291 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 17726, 2020-11-16T20:14:04.291 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314552852, 2020-11-16T20:14:04.291 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:14:04.291 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:14:04.291 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322102120448, 2020-11-16T20:14:04.292 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:14:04.292 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2248704, 2020-11-16T20:14:04.292 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 500531, 2020-11-16T20:14:04.292 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:14:04.292 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:14:04.292 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:14:04.292 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:14:04.292 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 18151544 2020-11-16T20:14:04.293 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.293 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:14:04.293 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:14:04.293 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:14:04.293 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:14:04.293 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:14:04.293 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:14:04.294 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:14:04.294 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.294 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:14:04.294 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 96, 2020-11-16T20:14:04.294 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 96, 2020-11-16T20:14:04.294 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 96000000, 2020-11-16T20:14:04.295 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 96000000 2020-11-16T20:14:04.295 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.295 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:14:04.295 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:14:04.295 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.295 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum_by_class": { 2020-11-16T20:14:04.296 INFO:tasks.workunit.client.0.smithi189.stdout: "hdd": { 2020-11-16T20:14:04.296 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:14:04.296 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:14:04.296 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 65, 2020-11-16T20:14:04.296 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:14:04.296 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:14:04.296 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:14:04.297 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:14:04.297 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 19948, 2020-11-16T20:14:04.297 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2196, 2020-11-16T20:14:04.297 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:14:04.297 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 17726, 2020-11-16T20:14:04.297 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314552852, 2020-11-16T20:14:04.298 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:14:04.298 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:14:04.298 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322102120448, 2020-11-16T20:14:04.298 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:14:04.298 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2248704, 2020-11-16T20:14:04.298 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 500531, 2020-11-16T20:14:04.298 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:14:04.299 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:14:04.299 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:14:04.299 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:14:04.299 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 18151544 2020-11-16T20:14:04.299 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.299 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:14:04.300 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:14:04.300 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:14:04.300 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:14:04.300 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:14:04.300 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:14:04.300 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:14:04.300 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:14:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 96, 2020-11-16T20:14:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 96, 2020-11-16T20:14:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 96000000, 2020-11-16T20:14:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 96000000 2020-11-16T20:14:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:14:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:14:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_stats": [ 2020-11-16T20:14:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: "poolid": 1, 2020-11-16T20:14:04.303 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 32, 2020-11-16T20:14:04.303 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:14:04.303 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 61854, 2020-11-16T20:14:04.303 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 100, 2020-11-16T20:14:04.303 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 0, 2020-11-16T20:14:04.303 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 300, 2020-11-16T20:14:04.303 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:14:04.304 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:14:04.304 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 100, 2020-11-16T20:14:04.304 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:14:04.304 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:14:04.304 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 100, 2020-11-16T20:14:04.304 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:14:04.304 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:14:04.305 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:14:04.305 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 101, 2020-11-16T20:14:04.305 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 101, 2020-11-16T20:14:04.305 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:14:04.305 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:14:04.305 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:14:04.305 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:14:04.306 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:14:04.306 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:14:04.306 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:14:04.306 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:14:04.306 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:14:04.306 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:14:04.307 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:14:04.307 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:14:04.307 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:14:04.307 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:14:04.307 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:14:04.307 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:14:04.307 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:14:04.308 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:14:04.308 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:14:04.308 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:14:04.308 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:14:04.308 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:14:04.308 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:14:04.309 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:14:04.309 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:14:04.309 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.309 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:14:04.309 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:14:04.309 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:14:04.309 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:14:04.310 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 1220608, 2020-11-16T20:14:04.310 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 184482, 2020-11-16T20:14:04.310 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:14:04.310 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:14:04.310 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:14:04.310 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:14:04.311 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:14:04.311 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.311 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 513, 2020-11-16T20:14:04.311 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 513, 2020-11-16T20:14:04.311 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 64, 2020-11-16T20:14:04.311 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 64, 2020-11-16T20:14:04.311 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 3 2020-11-16T20:14:04.312 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.312 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.312 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_stats": [ 2020-11-16T20:14:04.312 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.312 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:14:04.312 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 167503724564 2020-11-16T20:14:04.313 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.313 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.313 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:14:04.313 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 42949672977 2020-11-16T20:14:04.313 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.313 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.313 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:14:04.314 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 171798691859 2020-11-16T20:14:04.314 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.314 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.314 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_state": [ 2020-11-16T20:14:04.314 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.314 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "active+undersized+degraded", 2020-11-16T20:14:04.314 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 32 2020-11-16T20:14:04.315 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.315 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.315 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_osd": [ 2020-11-16T20:14:04.315 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.315 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:14:04.315 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:14:04.315 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 32, 2020-11-16T20:14:04.315 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:14:04.316 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.316 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.316 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:14:04.316 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:14:04.316 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 0, 2020-11-16T20:14:04.316 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:14:04.316 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.317 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.317 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:14:04.317 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 32, 2020-11-16T20:14:04.317 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 32, 2020-11-16T20:14:04.317 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:14:04.318 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.318 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.318 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": [ 2020-11-16T20:14:04.318 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:14:04.318 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:14:04.318 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": {} 2020-11-16T20:14:04.319 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.319 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:14:04.319 INFO:tasks.workunit.client.0.smithi189.stdout: "servicemap": { 2020-11-16T20:14:04.319 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 6, 2020-11-16T20:14:04.319 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:13:51.972309+0000", 2020-11-16T20:14:04.319 INFO:tasks.workunit.client.0.smithi189.stdout: "services": {} 2020-11-16T20:14:04.319 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:14:04.320 INFO:tasks.workunit.client.0.smithi189.stdout: "paxos": { 2020-11-16T20:14:04.320 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:14:04.320 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 450, 2020-11-16T20:14:04.320 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pn": 0, 2020-11-16T20:14:04.320 INFO:tasks.workunit.client.0.smithi189.stdout: "accepted_pn": 0 2020-11-16T20:14:04.320 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:14:04.320 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:14:04.321 INFO:tasks.workunit.client.0.smithi189.stderr:report 3628709881 2020-11-16T20:14:04.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2020-11-16T20:14:04.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2020-11-16T20:14:04.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:14:04.322 INFO:tasks.workunit.client.0.smithi189.stdout:delay recovery 2020-11-16T20:14:04.491 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:14:04.492 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296348+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.492 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296667+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.492 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296226+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.492 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296303+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.492 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.301035+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.492 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.209719+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.493 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.210484+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.493 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295745+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.493 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295821+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.493 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.299731+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.493 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294133+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.493 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293842+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.493 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295030+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.494 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295320+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.494 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.298933+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.494 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294639+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.494 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294770+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.494 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294494+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.494 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293545+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.494 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.297924+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.495 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.297851+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.495 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295243+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.495 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295820+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.495 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293591+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.495 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.209633+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.495 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293479+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.495 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.298754+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.496 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.210266+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.496 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294924+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.496 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294554+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.496 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 3 0 3 0 0 2082 0 0 17 17 active+undersized+degraded 2020-11-16T20:12:53.294029+0000 45'101 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.496 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295697+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:04.496 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:14:04.496 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:14:04.497 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:14:04.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-11-16T20:14:04.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2020-11-16T20:14:04.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T20:14:04.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T20:14:04.505 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:14:04.505 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:04.505 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:04.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.0.asok 2020-11-16T20:14:04.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-11-16T20:14:04.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.0.asok set_recovery_delay 100000 2020-11-16T20:14:04.613 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:14:04.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-11-16T20:14:04.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2020-11-16T20:14:04.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-11-16T20:14:04.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-11-16T20:14:04.623 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:14:04.624 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:04.624 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:04.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.2.asok 2020-11-16T20:14:04.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-11-16T20:14:04.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.2.asok set_recovery_delay 100000 2020-11-16T20:14:04.725 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:14:04.733 INFO:tasks.workunit.client.0.smithi189.stdout:revive divergent 1 2020-11-16T20:14:04.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2020-11-16T20:14:09.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:14:09.006 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:14:09.006 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296348+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.006 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296667+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.006 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296226+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.007 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296303+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.007 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.301035+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.007 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.209719+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.007 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.210484+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.007 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295745+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.007 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295821+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.007 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.299731+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.008 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294133+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.008 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293842+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.008 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295030+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.008 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295320+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.008 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.298933+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.008 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294639+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.008 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294770+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.008 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294494+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.009 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293545+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.009 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.297924+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.009 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.297851+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.009 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295243+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.009 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295820+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.009 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293591+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.009 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.209633+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.010 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293479+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.010 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.298754+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.010 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.210266+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.010 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294924+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.010 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294554+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.010 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 3 0 3 0 0 2082 0 0 17 17 active+undersized+degraded 2020-11-16T20:12:53.294029+0000 45'101 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.010 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295697+0000 30'100 45:126 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:09.011 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:14:09.011 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:14:09.011 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:14:09.011 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:14:09.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2020-11-16T20:14:09.011 INFO:tasks.workunit.client.0.smithi189.stderr:noup is set 2020-11-16T20:14:09.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-11-16T20:14:09.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:14:09.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:14:09.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T20:14:09.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:14:09.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:14:09.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:14:09.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:14:09.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:14:09.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:14:09.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:14:09.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:14:09.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:14:09.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:14:09.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:14:09.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:14:09.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:14:09.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:14:09.014 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:14:09.014 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:09.014 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:09.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:14:09.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:14:09.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:14:09.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:14:09.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:14:09.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:14:09.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:14:09.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:14:09.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:14:09.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:14:09.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:14:09.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T20:14:09.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:14:09.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2020-11-16T20:14:09.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T20:14:09.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:14:09.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:14:09.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:14:09.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:05.917+0000 7f4ad1e32d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:09.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:05.917+0000 7f4ad1e32d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:09.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:05.921+0000 7f4ad1e32d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:09.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2020-11-16T20:14:09.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2020-11-16T20:14:09.018 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:06.457+0000 7f4ad1e32d80 -1 Falling back to public interface 2020-11-16T20:14:09.018 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:07.965+0000 7f4ad1e32d80 -1 osd.1 33 log_to_monitors {default=true} 2020-11-16T20:14:11.290 INFO:tasks.workunit.client.0.smithi189.stdout:delay recovery divergent 2020-11-16T20:14:11.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2020-11-16T20:14:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:14:15.578 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:14:15.578 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296348+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.578 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296667+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.579 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296226+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.579 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.296303+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.579 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.301035+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.579 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.209719+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.579 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.210484+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.579 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295745+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.580 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295821+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.580 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.299731+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.580 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294133+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.580 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293842+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.580 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295030+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.580 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295320+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.580 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.298933+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.580 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294639+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.581 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294770+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.581 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294494+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.581 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293545+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.581 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.297924+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.581 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.297851+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.581 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295243+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.581 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295820+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293591+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.209633+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.293479+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.298754+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.210266+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294924+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.582 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.294554+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.583 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 3 0 3 0 0 2082 0 0 17 17 active+undersized+degraded 2020-11-16T20:12:53.294029+0000 45'101 45:128 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.583 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-11-16T20:12:53.295697+0000 30'100 45:127 [0,2] 0 [0,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.583 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:14:15.583 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:14:15.583 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:14:15.583 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:14:15.583 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 39 up_thru 44 down_at 34 last_clean_interval [5,30) [v2:127.0.0.1:6802/41211,v1:127.0.0.1:6803/41211] [v2:127.0.0.1:6804/41211,v1:127.0.0.1:6805/41211] exists,up 42e192ce-7576-4c00-87e5-e211cf1e1011 2020-11-16T20:14:15.584 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:14:15.584 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 35 last_clean_interval [10,33) [v2:127.0.0.1:6818/44109,v1:127.0.0.1:6819/44109] [v2:127.0.0.1:6820/44109,v1:127.0.0.1:6821/44109] exists,up dc40a541-b8cd-4c3b-81a7-0d36bc669c17 2020-11-16T20:14:15.584 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:14:15.584 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 40 up_thru 0 down_at 36 last_clean_interval [14,30) [v2:127.0.0.1:6810/41610,v1:127.0.0.1:6811/41610] [v2:127.0.0.1:6812/41610,v1:127.0.0.1:6813/41610] exists,up ca0542d6-a1d5-4f3f-8ddc-4d1c683befc4 2020-11-16T20:14:15.584 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:14:15.584 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.553171+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.584 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.553171+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.550394+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.550989+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.550560+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.550456+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.549670+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.552254+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.585 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.552635+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.586 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.552235+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.586 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.553863+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.586 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.551128+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.586 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.551325+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.586 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.554344+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.586 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.553885+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.586 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.552495+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.587 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 3 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.551810+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.587 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.553172+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.587 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.557923+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.587 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.555649+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.587 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.553864+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.587 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.551807+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.587 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 3 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.549610+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.588 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.559433+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.588 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.559762+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.588 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.555090+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.588 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.550456+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.588 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.551123+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.588 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.557553+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.588 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.556105+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.588 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.554100+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.589 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.552510+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.589 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:14:15.589 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:14:15.589 INFO:tasks.workunit.client.0.smithi189.stdout:wait for peering 2020-11-16T20:14:15.589 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:14:15.589 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.553171+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.589 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.553171+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.590 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.550394+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.590 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.550989+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.590 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.550560+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.590 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.550456+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.590 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.549670+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.590 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.552254+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.590 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.552635+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.591 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.552235+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.591 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.553863+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.591 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.551128+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.591 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.551325+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.591 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.554344+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.591 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.553885+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.591 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.552495+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.592 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 3 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.551810+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.592 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.553172+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.592 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.557923+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.592 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.555649+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.592 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.553864+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.592 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.551807+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.592 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 3 0 0 0 0 1900 0 0 16 16 peering 2020-11-16T20:14:12.549610+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.592 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.559433+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.593 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.559762+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.593 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.555090+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.593 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.550456+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.593 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.551123+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.593 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.557553+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.593 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.556105+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.593 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.554100+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.594 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 0 0 0 1899 0 0 16 16 peering 2020-11-16T20:14:12.552510+0000 30'106 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:15.594 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:14:15.594 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:14:15.594 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:14:15.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2020-11-16T20:14:15.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:14:15.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:14:15.595 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:14:15.595 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:15.595 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:15.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.1.asok 2020-11-16T20:14:15.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2020-11-16T20:14:15.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.1.asok set_recovery_delay 100000 2020-11-16T20:14:15.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2020-11-16T20:14:15.596 INFO:tasks.workunit.client.0.smithi189.stderr:noup is unset 2020-11-16T20:14:15.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2020-11-16T20:14:15.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:14:15.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:14:15.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:14:15.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:14:15.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:14:15.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:14:15.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:14:15.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:14:15.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:14:15.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:14:15.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:14:15.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2020-11-16T20:14:15.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:14:15.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:14:15.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:14:15.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:14:15.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:14:15.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:14:15.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:14:15.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:14:15.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:14:15.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:14:15.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:14:15.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2020-11-16T20:14:15.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:14:15.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:14:15.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:14:15.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:14:15.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:14:15.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:14:15.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:14:15.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:14:15.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:14:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:14:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:14:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:14:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:14:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2020-11-16T20:14:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:14:15.601 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:14:15.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2020-11-16T20:14:21.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2020-11-16T20:14:21.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:14:21.404 INFO:tasks.workunit.client.0.smithi189.stdout:killing divergent 1 2020-11-16T20:14:21.791 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:14:21.791 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.564715+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.791 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.567231+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.791 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.569596+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.562397+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 1 1 0 0 1928 0 0 10 10 active+recovery_wait+degraded+wait 2020-11-16T20:14:13.570751+0000 30'100 50:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.414218+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.414418+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.569303+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.562168+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.792 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.570551+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.563504+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.559771+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.569058+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.561932+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 1 1 0 0 1928 0 0 16 16 active+recovery_wait+degraded+wait 2020-11-16T20:14:13.569921+0000 30'100 50:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.563746+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.793 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.566617+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 4 0 0 0 0 1929 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.568628+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 0 0 0 1929 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.417234+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 4 0 0 0 0 1929 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.569521+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.567510+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 0 0 0 1929 0 0 15 15 active+clean+wait 2020-11-16T20:14:13.564214+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.564494+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.794 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.561730+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.795 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.416406+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.795 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 3 1 1 0 0 1928 0 0 11 11 active+recovering+degraded+wait 2020-11-16T20:14:13.570997+0000 30'100 50:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.795 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.567799+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.795 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.559573+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.795 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 0 0 0 1928 0 0 14 14 active+clean+wait 2020-11-16T20:14:13.568903+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.795 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 0 0 0 1928 0 0 12 12 active+clean+wait 2020-11-16T20:14:13.559085+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.795 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 3 1 1 0 0 2082 0 0 14 14 active+recovery_wait+degraded+wait 2020-11-16T20:14:13.568269+0000 45'101 50:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.796 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.566862+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:21.796 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:14:21.796 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:14:21.796 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:14:21.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2020-11-16T20:14:21.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:14:21.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:14:21.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:14:21.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:14:21.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:14:21.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:14:21.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.12850.out 2020-11-16T20:14:21.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.12850.out 2020-11-16T20:14:21.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-11-16T20:14:21.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T20:14:21.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-11-16T20:14:21.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T20:14:21.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-11-16T20:14:21.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.12850.out 2020-11-16T20:14:22.778 INFO:tasks.workunit.client.0.smithi189.stderr:Exporting 1.0 info 1.0( v 30'100 (30'90,30'100] local-lis/les=49/50 n=3 ec=22/22 lis/c=49/49 les/c/f=50/50/0 sis=49 pruub=7.732460076s) 2020-11-16T20:14:22.796 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:023dd3d2:::existing_12:head# 2020-11-16T20:14:22.796 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:04c40a0c:::existing_93:head# 2020-11-16T20:14:22.797 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:064ff5fb:::existing_36:head# 2020-11-16T20:14:22.798 INFO:tasks.workunit.client.0.smithi189.stderr:Export successful 2020-11-16T20:14:22.799 INFO:tasks.workunit.client.0.smithi189.stdout: marking collection for removal 2020-11-16T20:14:22.800 INFO:tasks.workunit.client.0.smithi189.stdout:setting '_remove' omap key 2020-11-16T20:14:22.800 INFO:tasks.workunit.client.0.smithi189.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-11-16T20:14:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:Remove successful 2020-11-16T20:14:23.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.12850.out 2020-11-16T20:14:23.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-11-16T20:14:23.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T20:14:23.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-11-16T20:14:23.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T20:14:23.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-11-16T20:14:23.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.12850.out 2020-11-16T20:14:23.965 INFO:tasks.workunit.client.0.smithi189.stdout:get_pg_num_history pg_num_history pg_num_history(e50 pg_nums {1={22=1,32=32}} deleted_pools ) 2020-11-16T20:14:23.965 INFO:tasks.workunit.client.0.smithi189.stdout:Importing pgid 1.0 2020-11-16T20:14:23.966 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:023dd3d2:::existing_12:head# 2020-11-16T20:14:23.966 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:14:24.086 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:04c40a0c:::existing_93:head# 2020-11-16T20:14:24.086 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:14:24.170 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:064ff5fb:::existing_36:head# 2020-11-16T20:14:24.170 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T20:14:24.203 INFO:tasks.workunit.client.0.smithi189.stdout:write_pg epoch 50 info 1.0( v 30'100 (30'90,30'100] local-lis/les=49/50 n=3 ec=22/22 lis/c=49/49 les/c/f=50/50/0 sis=49 pruub=7.732460076s) 2020-11-16T20:14:24.236 INFO:tasks.workunit.client.0.smithi189.stdout:Import successful 2020-11-16T20:14:24.774 INFO:tasks.workunit.client.0.smithi189.stdout:reviving divergent 1 2020-11-16T20:14:24.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2020-11-16T20:14:24.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:14:25.149 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:14:25.149 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.564715+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.149 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.567231+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.149 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.569596+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.562397+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 1 1 0 0 1928 0 0 10 10 active+recovery_wait+degraded+wait 2020-11-16T20:14:13.570751+0000 30'100 50:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.414218+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.414418+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.569303+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.150 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.562168+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.570551+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.563504+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.559771+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.569058+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.561932+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 1 1 0 0 1928 0 0 16 16 active+recovery_wait+degraded+wait 2020-11-16T20:14:13.569921+0000 30'100 50:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.151 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.563746+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.152 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.566617+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.152 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 4 0 0 0 0 1929 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.568628+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.152 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 0 0 0 1929 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.417234+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.152 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 4 0 0 0 0 1929 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.569521+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.152 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.567510+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.152 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 0 0 0 1929 0 0 15 15 active+clean+wait 2020-11-16T20:14:13.564214+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.152 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.564494+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.152 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.561730+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.153 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.416406+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.153 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 3 1 1 0 0 1928 0 0 11 11 active+recovering+degraded+wait 2020-11-16T20:14:13.570997+0000 30'100 50:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.153 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.567799+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.153 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-11-16T20:14:13.559573+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.153 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 0 0 0 1928 0 0 14 14 active+clean+wait 2020-11-16T20:14:13.568903+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.153 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 0 0 0 1928 0 0 12 12 active+clean+wait 2020-11-16T20:14:13.559085+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.153 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 3 1 1 0 0 2082 0 0 14 14 active+recovery_wait+degraded+wait 2020-11-16T20:14:13.568269+0000 45'101 50:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.154 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-11-16T20:14:13.566862+0000 30'100 50:135 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:25.154 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:14:25.154 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:14:25.154 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:14:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-11-16T20:14:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:14:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:14:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T20:14:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:14:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:14:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:14:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:14:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:14:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:14:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:14:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:14:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:14:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:14:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:14:25.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:14:25.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:14:25.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:14:25.164 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:14:25.164 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:25.164 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:25.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:14:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:14:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:14:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:14:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:14:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:14:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:14:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:14:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:14:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:14:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:14:25.167 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:14:25.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T20:14:25.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:14:25.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2020-11-16T20:14:25.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T20:14:25.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:14:25.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:14:25.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:14:25.199 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:25.192+0000 7f55864d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:25.201 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:25.196+0000 7f55864d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:25.201 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:25.196+0000 7f55864d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:25.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T20:14:25.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:14:25.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:14:25.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:14:25.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:14:25.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:14:25.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:14:25.572 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:14:25.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:14:25.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:14:25.738 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:25.732+0000 7f55864d6d80 -1 Falling back to public interface 2020-11-16T20:14:25.969 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 35 last_clean_interval [10,33) [v2:127.0.0.1:6818/44109,v1:127.0.0.1:6819/44109] [v2:127.0.0.1:6820/44109,v1:127.0.0.1:6821/44109] exists,up dc40a541-b8cd-4c3b-81a7-0d36bc669c17 2020-11-16T20:14:25.969 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:14:25.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:14:25.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:14:25.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:14:25.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2020-11-16T20:14:25.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:14:25.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:14:25.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:14:25.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:14:25.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:14:25.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:14:25.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:14:25.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:14:26.349 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 35 last_clean_interval [10,33) [v2:127.0.0.1:6818/44109,v1:127.0.0.1:6819/44109] [v2:127.0.0.1:6820/44109,v1:127.0.0.1:6821/44109] exists,up dc40a541-b8cd-4c3b-81a7-0d36bc669c17 2020-11-16T20:14:26.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:14:26.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:14:26.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:14:26.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2020-11-16T20:14:27.042 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:27.036+0000 7f55864d6d80 -1 osd.1 50 log_to_monitors {default=true} 2020-11-16T20:14:46.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2020-11-16T20:14:46.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:14:46.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:14:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:14:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:46.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.1.asok 2020-11-16T20:14:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2020-11-16T20:14:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.1.asok dump_ops_in_flight 2020-11-16T20:14:46.461 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:14:46.461 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [], 2020-11-16T20:14:46.461 INFO:tasks.workunit.client.0.smithi189.stdout: "num_ops": 0 2020-11-16T20:14:46.461 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:14:46.470 INFO:tasks.workunit.client.0.smithi189.stdout:allowing recovery 2020-11-16T20:14:46.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2020-11-16T20:14:46.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:14:46.847 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:14:46.847 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:39.171386+0000 30'100 59:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.848 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 0 0 0 1851 0 0 16 16 active+clean 2020-11-16T20:14:40.742386+0000 30'100 60:148 [0,1,2] 0 [0,1,2] 0 30'100 2020-11-16T20:14:40.742256+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.848 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:34.990244+0000 30'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.848 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:28.269119+0000 30'100 59:152 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.848 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-11-16T20:14:39.169263+0000 30'100 59:161 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.848 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 4 0 0 0 0 2468 0 0 16 16 active+clean 2020-11-16T20:14:42.672595+0000 30'100 61:148 [0,1,2] 0 [0,1,2] 0 30'100 2020-11-16T20:14:42.672418+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.848 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 2 0 0 0 0 1234 0 0 16 16 active+clean 2020-11-16T20:14:38.327064+0000 30'100 62:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-11-16T20:14:38.326883+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.848 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-11-16T20:14:28.271895+0000 30'100 59:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.849 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:39.171274+0000 30'100 59:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.849 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:37.030982+0000 30'100 60:145 [0,1,2] 0 [0,1,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.849 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 0 0 0 1851 0 0 16 16 active+clean 2020-11-16T20:14:44.429091+0000 30'100 62:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-11-16T20:14:44.429032+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.849 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 0 0 0 1851 0 0 10 10 active+clean 2020-11-16T20:14:38.137377+0000 30'100 59:156 [1,0,2] 1 [1,0,2] 1 30'100 2020-11-16T20:14:38.137233+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.849 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-11-16T20:14:39.167884+0000 30'100 59:159 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.849 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:37.027837+0000 30'100 60:145 [0,1,2] 0 [0,1,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.850 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 0 0 0 0 1851 0 0 16 16 active+clean 2020-11-16T20:14:40.323399+0000 30'100 62:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-11-16T20:14:40.323277+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.850 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 0 0 0 1851 0 0 10 10 active+clean 2020-11-16T20:14:37.203834+0000 30'100 59:156 [1,0,2] 1 [1,0,2] 1 30'100 2020-11-16T20:14:37.203644+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.850 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 6 0 0 0 0 3702 0 0 16 16 active+clean 2020-11-16T20:14:45.322683+0000 30'100 63:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-11-16T20:14:45.322554+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.850 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 2 0 0 0 0 1234 0 0 16 16 active+clean 2020-11-16T20:14:36.121889+0000 30'100 59:155 [1,0,2] 1 [1,0,2] 1 30'100 2020-11-16T20:14:36.121754+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.850 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 0 0 0 1929 0 0 10 10 active+clean 2020-11-16T20:14:39.170431+0000 30'100 59:159 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.850 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 1 0 0 0 0 617 0 0 16 16 active+clean 2020-11-16T20:14:38.716338+0000 30'100 60:148 [0,1,2] 0 [0,1,2] 0 30'100 2020-11-16T20:14:38.716202+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.850 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-11-16T20:14:28.272767+0000 30'100 59:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.851 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 0 0 0 1929 0 0 16 16 active+clean 2020-11-16T20:14:34.993190+0000 30'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.851 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 4 0 0 0 0 2545 0 0 17 17 active+clean 2020-11-16T20:14:37.031238+0000 50'101 60:153 [0,1,2] 0 [0,1,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.851 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:39.168878+0000 30'100 59:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.851 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 0 0 0 1851 0 0 10 10 active+clean 2020-11-16T20:14:40.151790+0000 30'100 59:156 [1,0,2] 1 [1,0,2] 1 30'100 2020-11-16T20:14:40.151655+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.851 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 5 0 0 0 0 3085 0 0 16 16 active+clean 2020-11-16T20:14:39.345879+0000 30'100 62:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-11-16T20:14:39.345749+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.851 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:37.028803+0000 30'100 60:145 [0,1,2] 0 [0,1,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.851 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-11-16T20:14:39.170580+0000 30'100 59:159 [1,2,0] 1 [1,2,0] 1 30'100 2020-11-16T20:14:37.430179+0000 30'100 2020-11-16T20:14:37.430179+0000 0 2020-11-16T20:14:46.852 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 0 0 0 1928 0 0 14 14 active+clean 2020-11-16T20:14:28.269110+0000 30'100 59:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.852 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:34.987792+0000 30'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.852 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 4 0 0 0 0 2622 0 0 17 17 active+clean 2020-11-16T20:14:41.701970+0000 45'101 60:149 [0,1,2] 0 [0,1,2] 0 45'101 2020-11-16T20:14:41.701890+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.852 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:39.061198+0000 30'100 59:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:46.852 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:14:46.852 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:14:46.853 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:14:46.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-11-16T20:14:46.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2020-11-16T20:14:47.018 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:14:47.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-11-16T20:14:47.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T20:14:47.196 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:14:47.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-11-16T20:14:47.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2020-11-16T20:14:47.362 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:14:47.372 INFO:tasks.workunit.client.0.smithi189.stdout:reading divergent objects 2020-11-16T20:14:47.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2020-11-16T20:14:47.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2020-11-16T20:14:47.731 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:14:47.731 INFO:tasks.workunit.client.0.smithi189.stdout:1.1f 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:39.171386+0000 30'100 59:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.731 INFO:tasks.workunit.client.0.smithi189.stdout:1.1e 3 0 0 0 0 1851 0 0 16 16 active+clean 2020-11-16T20:14:40.742386+0000 30'100 60:148 [0,1,2] 0 [0,1,2] 0 30'100 2020-11-16T20:14:40.742256+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.731 INFO:tasks.workunit.client.0.smithi189.stdout:1.1d 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:34.990244+0000 30'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.732 INFO:tasks.workunit.client.0.smithi189.stdout:1.1c 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:28.269119+0000 30'100 59:152 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.732 INFO:tasks.workunit.client.0.smithi189.stdout:1.1b 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-11-16T20:14:39.169263+0000 30'100 59:161 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.732 INFO:tasks.workunit.client.0.smithi189.stdout:1.1a 4 0 0 0 0 2468 0 0 16 16 active+clean 2020-11-16T20:14:42.672595+0000 30'100 61:148 [0,1,2] 0 [0,1,2] 0 30'100 2020-11-16T20:14:42.672418+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.732 INFO:tasks.workunit.client.0.smithi189.stdout:1.19 2 0 0 0 0 1234 0 0 16 16 active+clean 2020-11-16T20:14:38.327064+0000 30'100 62:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-11-16T20:14:38.326883+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.732 INFO:tasks.workunit.client.0.smithi189.stdout:1.18 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-11-16T20:14:28.271895+0000 30'100 59:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.732 INFO:tasks.workunit.client.0.smithi189.stdout:1.17 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:39.171274+0000 30'100 59:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.732 INFO:tasks.workunit.client.0.smithi189.stdout:1.16 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:37.030982+0000 30'100 60:145 [0,1,2] 0 [0,1,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.733 INFO:tasks.workunit.client.0.smithi189.stdout:1.15 3 0 0 0 0 1851 0 0 16 16 active+clean 2020-11-16T20:14:44.429091+0000 30'100 62:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-11-16T20:14:44.429032+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.733 INFO:tasks.workunit.client.0.smithi189.stdout:1.14 3 0 0 0 0 1851 0 0 10 10 active+clean 2020-11-16T20:14:38.137377+0000 30'100 59:156 [1,0,2] 1 [1,0,2] 1 30'100 2020-11-16T20:14:38.137233+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.733 INFO:tasks.workunit.client.0.smithi189.stdout:1.13 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-11-16T20:14:39.167884+0000 30'100 59:159 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.733 INFO:tasks.workunit.client.0.smithi189.stdout:1.12 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:37.027837+0000 30'100 60:145 [0,1,2] 0 [0,1,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.733 INFO:tasks.workunit.client.0.smithi189.stdout:1.11 3 0 0 0 0 1851 0 0 16 16 active+clean 2020-11-16T20:14:40.323399+0000 30'100 62:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-11-16T20:14:40.323277+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.733 INFO:tasks.workunit.client.0.smithi189.stdout:1.10 3 0 0 0 0 1851 0 0 10 10 active+clean 2020-11-16T20:14:37.203834+0000 30'100 59:156 [1,0,2] 1 [1,0,2] 1 30'100 2020-11-16T20:14:37.203644+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.733 INFO:tasks.workunit.client.0.smithi189.stdout:1.5 6 0 0 0 0 3702 0 0 16 16 active+clean 2020-11-16T20:14:45.322683+0000 30'100 63:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-11-16T20:14:45.322554+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.734 INFO:tasks.workunit.client.0.smithi189.stdout:1.4 2 0 0 0 0 1234 0 0 16 16 active+clean 2020-11-16T20:14:36.121889+0000 30'100 59:155 [1,0,2] 1 [1,0,2] 1 30'100 2020-11-16T20:14:36.121754+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.734 INFO:tasks.workunit.client.0.smithi189.stdout:1.3 4 0 0 0 0 1929 0 0 10 10 active+clean 2020-11-16T20:14:39.170431+0000 30'100 59:159 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.734 INFO:tasks.workunit.client.0.smithi189.stdout:1.2 1 0 0 0 0 617 0 0 16 16 active+clean 2020-11-16T20:14:38.716338+0000 30'100 60:148 [0,1,2] 0 [0,1,2] 0 30'100 2020-11-16T20:14:38.716202+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.734 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-11-16T20:14:28.272767+0000 30'100 59:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.734 INFO:tasks.workunit.client.0.smithi189.stdout:1.1 4 0 0 0 0 1929 0 0 16 16 active+clean 2020-11-16T20:14:34.993190+0000 30'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.734 INFO:tasks.workunit.client.0.smithi189.stdout:1.6 4 0 0 0 0 2545 0 0 17 17 active+clean 2020-11-16T20:14:37.031238+0000 50'101 60:153 [0,1,2] 0 [0,1,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.734 INFO:tasks.workunit.client.0.smithi189.stdout:1.7 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:39.168878+0000 30'100 59:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.735 INFO:tasks.workunit.client.0.smithi189.stdout:1.8 3 0 0 0 0 1851 0 0 10 10 active+clean 2020-11-16T20:14:40.151790+0000 30'100 59:156 [1,0,2] 1 [1,0,2] 1 30'100 2020-11-16T20:14:40.151655+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.735 INFO:tasks.workunit.client.0.smithi189.stdout:1.9 5 0 0 0 0 3085 0 0 16 16 active+clean 2020-11-16T20:14:39.345879+0000 30'100 62:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-11-16T20:14:39.345749+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.735 INFO:tasks.workunit.client.0.smithi189.stdout:1.a 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:37.028803+0000 30'100 60:145 [0,1,2] 0 [0,1,2] 0 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.735 INFO:tasks.workunit.client.0.smithi189.stdout:1.b 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-11-16T20:14:39.170580+0000 30'100 59:159 [1,2,0] 1 [1,2,0] 1 30'100 2020-11-16T20:14:37.430179+0000 30'100 2020-11-16T20:14:37.430179+0000 0 2020-11-16T20:14:47.735 INFO:tasks.workunit.client.0.smithi189.stdout:1.c 3 0 0 0 0 1928 0 0 14 14 active+clean 2020-11-16T20:14:28.269110+0000 30'100 59:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.735 INFO:tasks.workunit.client.0.smithi189.stdout:1.d 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:34.987792+0000 30'100 62:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.735 INFO:tasks.workunit.client.0.smithi189.stdout:1.e 4 0 0 0 0 2622 0 0 17 17 active+clean 2020-11-16T20:14:41.701970+0000 45'101 60:149 [0,1,2] 0 [0,1,2] 0 45'101 2020-11-16T20:14:41.701890+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.736 INFO:tasks.workunit.client.0.smithi189.stdout:1.f 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-11-16T20:14:39.061198+0000 30'100 59:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-11-16T20:11:58.802132+0000 0'0 2020-11-16T20:11:58.802132+0000 0 2020-11-16T20:14:47.736 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:14:47.736 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:14:47.736 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:14:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2020-11-16T20:14:47.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2020-11-16T20:14:47.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:14:47.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2020-11-16T20:14:47.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:14:47.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2020-11-16T20:14:47.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:14:47.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2020-11-16T20:14:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:14:47.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2020-11-16T20:14:47.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2020-11-16T20:14:47.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-11-16T20:14:47.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2020-11-16T20:14:48.011 INFO:tasks.workunit.client.0.smithi189.stderr:error getting test/newobject_1: (2) No such file or directory 2020-11-16T20:14:48.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-11-16T20:14:48.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2020-11-16T20:14:48.048 INFO:tasks.workunit.client.0.smithi189.stderr:error getting test/newobject_2: (2) No such file or directory 2020-11-16T20:14:48.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2020-11-16T20:14:48.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-11-16T20:14:48.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2020-11-16T20:14:48.075 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.196+0000 7f4aae2ba700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 30'102 (30'2) modify 1:97db2a0f:::existing_2:head by client.4537.0:1 2020-11-16T20:12:32.797948+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:14:48.075 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.196+0000 7f4aae2ba700 20 _merge_object_divergent_entries: keeping 30'102 (30'2) modify 1:97db2a0f:::existing_2:head by client.4537.0:1 2020-11-16T20:12:32.797948+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:14:48.076 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.196+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:14:48.076 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.196+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 30'2 first_divergent_update: 30'102 last_divergent_update: 30'102 2020-11-16T20:14:48.076 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.196+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-11-16T20:14:48.076 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.196+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-11-16T20:14:48.076 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.196+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 30'106 2020-11-16T20:14:48.077 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.196+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 30'102 (30'2) modify 1:97db2a0f:::existing_2:head by client.4537.0:1 2020-11-16T20:12:32.797948+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:14:48.077 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.196+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-11-16T20:14:48.077 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.196+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 30'2 <= info.log_tail 30'89 2020-11-16T20:14:48.077 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-11-16T20:12:32.803316+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-11-16T20:14:48.077 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 20 _merge_object_divergent_entries: keeping 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-11-16T20:12:32.803316+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-11-16T20:14:48.077 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-11-16T20:14:48.077 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 30'3 first_divergent_update: 30'106 last_divergent_update: 30'106 2020-11-16T20:14:48.078 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-11-16T20:14:48.078 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-11-16T20:14:48.078 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 30'106 2020-11-16T20:14:48.078 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-11-16T20:12:32.803316+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-11-16T20:14:48.078 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-11-16T20:14:48.078 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 30'3 <= info.log_tail 30'84 2020-11-16T20:14:48.079 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aae2ba700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 30'104 (30'4) delete 1:70b3bf12:::existing_4:head by client.4541.0:1 2020-11-16T20:12:32.801808+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-11-16T20:14:48.079 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aae2ba700 20 _merge_object_divergent_entries: keeping 30'104 (30'4) delete 1:70b3bf12:::existing_4:head by client.4541.0:1 2020-11-16T20:12:32.801808+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-11-16T20:14:48.079 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:14:48.079 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 30'4 first_divergent_update: 30'104 last_divergent_update: 30'104 2020-11-16T20:14:48.079 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aae2ba700 10 _merge_object_divergent_entries: more recent entry found: 45'101 (30'4) modify 1:70b3bf12:::existing_4:head by client.4650.0:1 2020-11-16T20:13:27.835548+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0, already merged 2020-11-16T20:14:48.079 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aae2ba700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 30'105 (0'0) modify 1:75ac5091:::newobject_2:head by client.4542.0:1 2020-11-16T20:12:32.802487+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-11-16T20:14:48.079 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aae2ba700 20 _merge_object_divergent_entries: keeping 30'105 (0'0) modify 1:75ac5091:::newobject_2:head by client.4542.0:1 2020-11-16T20:12:32.802487+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-11-16T20:14:48.079 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:14:48.080 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 30'105 last_divergent_update: 30'105 2020-11-16T20:14:48.080 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-11-16T20:14:48.080 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aae2ba700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-11-16T20:14:48.080 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 30'103 (30'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-11-16T20:12:32.800284+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:14:48.080 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 20 _merge_object_divergent_entries: keeping 30'103 (30'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-11-16T20:12:32.800284+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:14:48.080 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:14:48.081 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 30'1 first_divergent_update: 30'103 last_divergent_update: 30'103 2020-11-16T20:14:48.081 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-11-16T20:14:48.081 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-11-16T20:14:48.081 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 30'106 2020-11-16T20:14:48.082 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 30'103 (30'1) modify 1:d858c035:::existing_1:head by client.4539.0:1 2020-11-16T20:12:32.800284+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:14:48.082 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-11-16T20:14:48.082 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 30'1 <= info.log_tail 30'90 2020-11-16T20:14:48.082 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 30'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4530.0:1 2020-11-16T20:12:32.790990+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-11-16T20:14:48.082 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 20 _merge_object_divergent_entries: keeping 30'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4530.0:1 2020-11-16T20:12:32.790990+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-11-16T20:14:48.082 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:14:48.082 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 30'101 last_divergent_update: 30'101 2020-11-16T20:14:48.083 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-11-16T20:14:48.083 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:14:13.304+0000 7f4aaf2bc700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-11-16T20:14:48.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-11-16T20:14:48.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2020-11-16T20:14:48.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-11-16T20:14:48.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2020-11-16T20:14:48.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2020-11-16T20:14:48.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.12850.out 2020-11-16T20:14:48.118 INFO:tasks.workunit.client.0.smithi189.stdout:success 2020-11-16T20:14:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.12850.out': No such file or directory 2020-11-16T20:14:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2020-11-16T20:14:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T20:14:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T20:14:48.892 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T20:14:48.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2020-11-16T20:14:48.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:14:48.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:14:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:14:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:14:48.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:14:49.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:14:49.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-11-16T20:14:49.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-11-16T20:14:49.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:14:49.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-11-16T20:14:49.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:14:49.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:14:49.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:14:49.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:14:49.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:14:49.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:14:49.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:14:49.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:14:49.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:14:49.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:14:49.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:14:49.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:14:49.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:14:49.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:14:49.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:14:49.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:14:49.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:14:49.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:14:49.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-11-16T20:14:49.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:14:49.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:49.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:49.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12850 2020-11-16T20:14:49.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:14:49.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:14:49.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-11-16T20:14:49.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-11-16T20:14:49.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-11-16T20:14:49.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-11-16T20:14:49.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-11-16T20:14:49.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:14:49.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-11-16T20:14:49.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:14:49.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:14:49.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:14:49.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:14:49.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:14:49.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:14:49.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:14:49.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:14:49.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:14:49.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:14:49.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:14:49.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:14:49.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:14:49.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:14:49.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:14:49.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:14:49.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:14:49.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:14:49.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-11-16T20:14:49.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:14:49.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:49.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:49.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12850 2020-11-16T20:14:49.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:14:49.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:14:49.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-11-16T20:14:49.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:14:49.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:49.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:49.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12850 2020-11-16T20:14:49.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:14:49.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T20:14:49.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:14:49.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-11-16T20:14:49.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2020-11-16T20:14:49.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2020-11-16T20:14:49.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2020-11-16T20:14:49.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2020-11-16T20:14:49.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2020-11-16T20:14:49.301 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2020-11-16T20:14:49.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2020-11-16T20:14:49.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2020-11-16T20:14:49.305 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:14:49.305 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:14:49.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2020-11-16T20:14:49.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-11-16T20:14:49.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:14:49.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:14:49.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:14:49.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-11-16T20:14:49.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-11-16T20:14:49.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:14:49.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:14:49.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:14:49.597 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:14:49.597 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:49.597 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:49.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:14:49.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:14:49.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:14:49.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:14:49.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:14:49.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:14:49.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:14:49.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:14:49.811 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:14:49.811 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:14:49.811 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:14:49.812 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:14:49.812 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:49.812 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:49.812 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-mon.a.asok 2020-11-16T20:14:49.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:14:49.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.12850/ceph-mon.a.asok config get fsid 2020-11-16T20:14:49.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:14:49.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:14:49.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:14:49.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:14:49.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:14:49.923 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:14:49.923 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:14:49.923 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:14:49.923 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:14:49.924 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:49.924 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:49.924 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-mon.a.asok 2020-11-16T20:14:49.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:14:49.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.12850/ceph-mon.a.asok config get mon_host 2020-11-16T20:14:50.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2020-11-16T20:14:50.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2020-11-16T20:14:50.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:14:50.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:14:50.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:14:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2020-11-16T20:14:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:14:50.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:14:50.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:14:50.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:14:50.547 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:14:50.547 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:50.547 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:50.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:14:50.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:14:50.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:14:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-11-16T20:14:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2020-11-16T20:14:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:14:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:14:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:14:50.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:14:50.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2020-11-16T20:14:50.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:14:50.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:14:50.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:14:50.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:14:50.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-11-16T20:14:50.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-11-16T20:14:50.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:14:50.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:14:50.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:14:50.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:14:50.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:14:50.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:14:50.581 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:14:50.581 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:14:50.581 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:14:50.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:14:50.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:14:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:14:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:14:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:14:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:14:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:14:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:14:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:14:50.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:14:50.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:14:50.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:14:50.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2020-11-16T20:14:50.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:14:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8e23841d-98e5-472c-91fe-6a992d391d13 2020-11-16T20:14:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 8e23841d-98e5-472c-91fe-6a992d391d13' 2020-11-16T20:14:50.584 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 8e23841d-98e5-472c-91fe-6a992d391d13 2020-11-16T20:14:50.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:14:50.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC63bJfhVn5IxAASDM97fRltyek0cbmg7DEdQ== 2020-11-16T20:14:50.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC63bJfhVn5IxAASDM97fRltyek0cbmg7DEdQ=="}' 2020-11-16T20:14:50.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8e23841d-98e5-472c-91fe-6a992d391d13 -i td/divergent-priors/0/new.json 2020-11-16T20:14:50.963 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:14:50.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2020-11-16T20:14:50.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC63bJfhVn5IxAASDM97fRltyek0cbmg7DEdQ== --osd-uuid 8e23841d-98e5-472c-91fe-6a992d391d13 2020-11-16T20:14:51.012 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:51.008+0000 7f3e071a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:51.015 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:51.008+0000 7f3e071a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:51.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:51.012+0000 7f3e071a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:51.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:51.012+0000 7f3e071a3d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:14:51.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:51.012+0000 7f3e071a3d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:14:51.018 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:51.012+0000 7f3e071a3d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-11-16T20:14:51.069 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:51.064+0000 7f3e071a3d80 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-11-16T20:14:55.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-11-16T20:14:55.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:14:55.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:14:55.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:14:55.599 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:14:56.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:14:56.100 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:14:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:14:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:14:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:14:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:14:56.134 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:56.128+0000 7f47d17bdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:56.137 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:56.132+0000 7f47d17bdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:56.138 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:56.132+0000 7f47d17bdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:14:56.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:14:56.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:14:56.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:14:56.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:14:56.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:14:56.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:14:56.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:14:56.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:14:56.513 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:14:56.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:14:56.674 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:56.668+0000 7f47d17bdd80 -1 Falling back to public interface 2020-11-16T20:14:56.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:14:57.548 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:57.531+0000 7f47d17bdd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:14:57.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:14:57.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:14:57.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:14:57.896 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:14:57.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:14:57.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:14:58.276 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/46814,v1:127.0.0.1:6803/46814] [v2:127.0.0.1:6804/46814,v1:127.0.0.1:6805/46814] exists,up 8e23841d-98e5-472c-91fe-6a992d391d13 2020-11-16T20:14:58.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:14:58.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:15:02.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:15:02.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-11-16T20:15:02.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2020-11-16T20:15:02.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:15:02.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:15:02.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:15:02.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:15:02.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:15:02.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:15:02.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:15:02.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:15:02.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:15:02.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:15:02.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:15:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:15:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:15:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:15:02.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:15:02.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:15:02.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:15:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:15:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:15:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:15:02.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:15:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:15:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:15:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:15:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:15:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:15:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:15:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:15:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:15:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:15:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:15:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:15:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:15:02.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:15:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fada656c-964b-4065-ada2-6f88fccdceb7 2020-11-16T20:15:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 fada656c-964b-4065-ada2-6f88fccdceb7' 2020-11-16T20:15:02.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:15:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDC3bJfVsA7EhAAWNR9Ee0yCs0LVOIeCwRYOw== 2020-11-16T20:15:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDC3bJfVsA7EhAAWNR9Ee0yCs0LVOIeCwRYOw=="}' 2020-11-16T20:15:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fada656c-964b-4065-ada2-6f88fccdceb7 -i td/divergent-priors/1/new.json 2020-11-16T20:15:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2020-11-16T20:15:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDC3bJfVsA7EhAAWNR9Ee0yCs0LVOIeCwRYOw== --osd-uuid fada656c-964b-4065-ada2-6f88fccdceb7 2020-11-16T20:15:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:58.971+0000 7f1065ff7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:58.975+0000 7f1065ff7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:58.975+0000 7f1065ff7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:58.975+0000 7f1065ff7d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:15:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:58.975+0000 7f1065ff7d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:15:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:58.975+0000 7f1065ff7d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-11-16T20:15:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:14:59.043+0000 7f1065ff7d80 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-11-16T20:15:02.538 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 fada656c-964b-4065-ada2-6f88fccdceb7 2020-11-16T20:15:02.538 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:15:03.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-11-16T20:15:03.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:15:03.677 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:15:03.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:15:03.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:15:04.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:15:04.179 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:15:04.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:15:04.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:15:04.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:15:04.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:15:04.214 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:04.211+0000 7fe7d56ebd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:04.217 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:04.215+0000 7fe7d56ebd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:04.218 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:04.215+0000 7fe7d56ebd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:04.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:15:04.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:15:04.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:15:04.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:15:04.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:15:04.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:15:04.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:15:04.578 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:15:04.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:15:04.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:15:04.762 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:04.759+0000 7fe7d56ebd80 -1 Falling back to public interface 2020-11-16T20:15:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:15:05.584 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:05.575+0000 7fe7d56ebd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:15:05.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:15:05.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:15:05.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:15:05.968 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:15:05.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:15:05.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:15:06.425 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/47811,v1:127.0.0.1:6811/47811] [v2:127.0.0.1:6812/47811,v1:127.0.0.1:6813/47811] exists,up fada656c-964b-4065-ada2-6f88fccdceb7 2020-11-16T20:15:06.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:15:06.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:15:06.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:15:06.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-11-16T20:15:06.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2020-11-16T20:15:06.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2020-11-16T20:15:06.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:15:06.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:15:06.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:15:06.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2020-11-16T20:15:06.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:15:06.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:15:06.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:15:06.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:15:06.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-11-16T20:15:06.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-11-16T20:15:06.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:15:06.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:15:06.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:15:06.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:15:06.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:15:06.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:15:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:15:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:15:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:15:06.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:15:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:15:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:15:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:15:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:15:06.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:15:06.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:15:06.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:15:06.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:15:06.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:15:06.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:15:06.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:15:06.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2020-11-16T20:15:06.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:15:06.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=41ef07c7-ba20-4c7c-b7b2-5bc264c4a07a 2020-11-16T20:15:06.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 41ef07c7-ba20-4c7c-b7b2-5bc264c4a07a' 2020-11-16T20:15:06.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:15:06.434 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 41ef07c7-ba20-4c7c-b7b2-5bc264c4a07a 2020-11-16T20:15:06.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDK3bJfUWz0GhAAM+8TNIkl9AK+ikoPrTWulA== 2020-11-16T20:15:06.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDK3bJfUWz0GhAAM+8TNIkl9AK+ikoPrTWulA=="}' 2020-11-16T20:15:06.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 41ef07c7-ba20-4c7c-b7b2-5bc264c4a07a -i td/divergent-priors/2/new.json 2020-11-16T20:15:07.039 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:15:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2020-11-16T20:15:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDK3bJfUWz0GhAAM+8TNIkl9AK+ikoPrTWulA== --osd-uuid 41ef07c7-ba20-4c7c-b7b2-5bc264c4a07a 2020-11-16T20:15:07.085 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:07.083+0000 7f8ca08e1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:07.088 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:07.087+0000 7f8ca08e1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:07.090 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:07.087+0000 7f8ca08e1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:07.090 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:07.087+0000 7f8ca08e1d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:15:07.091 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:07.087+0000 7f8ca08e1d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:15:07.091 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:07.087+0000 7f8ca08e1d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-11-16T20:15:07.181 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:07.179+0000 7f8ca08e1d80 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-11-16T20:15:11.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-11-16T20:15:11.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:15:11.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:15:11.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:15:11.544 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:15:12.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:15:12.056 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:15:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:15:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:15:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:15:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:15:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:12.087+0000 7ffbb320ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:12.091+0000 7ffbb320ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:12.091+0000 7ffbb320ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:15:12.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:15:12.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:15:12.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:15:12.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:15:12.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:15:12.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:15:12.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:15:12.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:15:12.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:15:12.461 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:15:12.642 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:12.639+0000 7ffbb320ad80 -1 Falling back to public interface 2020-11-16T20:15:12.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:15:13.518 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:15:13.515+0000 7ffbb320ad80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:15:13.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:15:13.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:15:13.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:15:13.891 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:15:13.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:15:13.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:15:14.285 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/48805,v1:127.0.0.1:6819/48805] [v2:127.0.0.1:6820/48805,v1:127.0.0.1:6821/48805] exists,up 41ef07c7-ba20-4c7c-b7b2-5bc264c4a07a 2020-11-16T20:15:14.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:15:14.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:15:14.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:15:14.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2020-11-16T20:15:15.108 INFO:tasks.workunit.client.0.smithi189.stderr:noout is set 2020-11-16T20:15:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2020-11-16T20:15:15.859 INFO:tasks.workunit.client.0.smithi189.stderr:noin is set 2020-11-16T20:15:15.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2020-11-16T20:15:16.704 INFO:tasks.workunit.client.0.smithi189.stderr:nodown is set 2020-11-16T20:15:16.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2020-11-16T20:15:16.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: create_ec_pool: local pool_name=test 2020-11-16T20:15:16.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: create_ec_pool: shift 2020-11-16T20:15:16.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: local allow_overwrites=true 2020-11-16T20:15:16.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: create_ec_pool: shift 2020-11-16T20:15:16.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2020-11-16T20:15:17.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: create_pool test 1 1 erasure myprofile 2020-11-16T20:15:17.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-11-16T20:15:18.066 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T20:15:18.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:15:19.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: '[' true = true ']' 2020-11-16T20:15:19.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2020-11-16T20:15:19.921 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 allow_ec_overwrites to true 2020-11-16T20:15:19.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: wait_for_clean 2020-11-16T20:15:19.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:15:19.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:15:19.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:15:19.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:15:19.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:15:19.938 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:15:19.938 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:15:19.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:15:19.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:15:19.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:15:20.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:15:20.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:15:20.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:15:20.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:15:20.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:15:20.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:15:20.567 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:15:20.567 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:15:20.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:15:20.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:20.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:15:20.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2020-11-16T20:15:20.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2020-11-16T20:15:20.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2020-11-16T20:15:20.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:20.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:15:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2020-11-16T20:15:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2020-11-16T20:15:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2020-11-16T20:15:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:20.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:15:21.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2020-11-16T20:15:21.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2020-11-16T20:15:21.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509444' 2020-11-16T20:15:21.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:21.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2020-11-16T20:15:21.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:21.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:15:21.098 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2020-11-16T20:15:21.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:21.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2020-11-16T20:15:21.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-11-16T20:15:21.100 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836487 2020-11-16T20:15:21.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:15:21.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2020-11-16T20:15:21.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:15:22.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:15:22.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:15:22.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2020-11-16T20:15:22.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:22.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2020-11-16T20:15:22.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:22.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:15:22.926 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2020-11-16T20:15:22.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2020-11-16T20:15:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-11-16T20:15:22.928 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672965 2020-11-16T20:15:22.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:15:23.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672965 2020-11-16T20:15:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:23.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2020-11-16T20:15:23.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:15:23.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:23.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2020-11-16T20:15:23.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2020-11-16T20:15:23.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-11-16T20:15:23.300 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509444 2020-11-16T20:15:23.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:15:23.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2020-11-16T20:15:23.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:15:23.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:15:23.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:15:24.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:15:24.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:15:24.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:15:24.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:15:24.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:15:24.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:15:24.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:15:24.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:15:24.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:15:24.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:15:24.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:15:24.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:15:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:15:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:15:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:15:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: return 0 2020-11-16T20:15:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2020-11-16T20:15:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:15:25.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:15:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:15:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:15:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:15:25.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:15:25.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:25.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:15:25.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2020-11-16T20:15:25.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2020-11-16T20:15:25.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2020-11-16T20:15:25.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:25.679 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:15:25.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2020-11-16T20:15:25.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2020-11-16T20:15:25.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2020-11-16T20:15:25.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:25.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:15:26.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509447 2020-11-16T20:15:26.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509447 2020-11-16T20:15:26.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509447' 2020-11-16T20:15:26.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:26.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2020-11-16T20:15:26.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:26.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:15:26.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2020-11-16T20:15:26.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:26.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2020-11-16T20:15:26.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-11-16T20:15:26.035 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836490 2020-11-16T20:15:26.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:15:26.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2020-11-16T20:15:26.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:26.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2020-11-16T20:15:26.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:26.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:15:26.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2020-11-16T20:15:26.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:26.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2020-11-16T20:15:26.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-11-16T20:15:26.457 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672968 2020-11-16T20:15:26.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:15:26.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2020-11-16T20:15:26.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:26.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509447 2020-11-16T20:15:26.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:26.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:15:26.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509447 2020-11-16T20:15:26.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:26.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509447 2020-11-16T20:15:26.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2020-11-16T20:15:26.868 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509447 2020-11-16T20:15:26.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:15:27.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509447 -lt 64424509447 2020-11-16T20:15:27.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2020-11-16T20:15:27.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:15:27.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:15:27.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:15:27.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:15:27.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:15:27.274 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:15:27.274 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:15:27.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:15:27.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:15:27.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:15:27.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:15:27.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:15:27.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:15:27.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:15:27.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:15:27.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:15:27.871 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:15:27.872 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:15:27.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:15:27.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:27.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:15:28.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836492 2020-11-16T20:15:28.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836492 2020-11-16T20:15:28.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492' 2020-11-16T20:15:28.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:28.078 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:15:28.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672971 2020-11-16T20:15:28.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672971 2020-11-16T20:15:28.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971' 2020-11-16T20:15:28.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:28.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:15:28.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509449 2020-11-16T20:15:28.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509449 2020-11-16T20:15:28.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449' 2020-11-16T20:15:28.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:28.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836492 2020-11-16T20:15:28.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:28.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:15:28.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836492 2020-11-16T20:15:28.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:28.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836492 2020-11-16T20:15:28.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-11-16T20:15:28.430 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836492 2020-11-16T20:15:28.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:15:28.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836492 2020-11-16T20:15:28.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:28.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672971 2020-11-16T20:15:28.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:28.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:15:28.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2020-11-16T20:15:28.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:28.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672971 2020-11-16T20:15:28.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-11-16T20:15:28.837 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672971 2020-11-16T20:15:28.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:15:29.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672969 -lt 42949672971 2020-11-16T20:15:29.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:15:30.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:15:30.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:15:30.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672971 2020-11-16T20:15:30.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:30.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509449 2020-11-16T20:15:30.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:30.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:15:30.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:30.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509449 2020-11-16T20:15:30.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509449 2020-11-16T20:15:30.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2020-11-16T20:15:30.641 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509449 2020-11-16T20:15:30.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:15:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509450 -lt 64424509449 2020-11-16T20:15:31.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:15:31.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:15:31.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:15:31.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:15:31.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:15:31.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:15:31.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:15:31.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:15:31.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:15:31.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:15:31.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:15:31.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:15:31.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:15:31.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:15:31.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:15:32.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:15:32.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:15:32.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:15:32.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2020-11-16T20:15:32.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2020-11-16T20:15:32.723 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:15:32.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2020-11-16T20:15:32.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2020-11-16T20:15:32.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:15:32.734 INFO:tasks.workunit.client.0.smithi189.stdout:primary and soon to be divergent is 1 2020-11-16T20:15:33.102 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:15:33.103 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:15:19.187780+0000 0'0 28:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:15:33.103 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:15:33.104 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:15:33.105 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:15:33.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2020-11-16T20:15:33.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-11-16T20:15:33.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2020-11-16T20:15:33.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2020-11-16T20:15:33.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-11-16T20:15:33.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2020-11-16T20:15:33.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2020-11-16T20:15:33.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-11-16T20:15:33.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2020-11-16T20:15:33.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2020-11-16T20:15:33.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2020-11-16T20:15:33.117 INFO:tasks.workunit.client.0.smithi189.stdout:writing initial objects 2020-11-16T20:15:33.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2020-11-16T20:15:33.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:33.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2020-11-16T20:15:33.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:33.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2020-11-16T20:15:33.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:33.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2020-11-16T20:15:33.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:33.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2020-11-16T20:15:33.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:33.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2020-11-16T20:15:34.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:34.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2020-11-16T20:15:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2020-11-16T20:15:34.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:34.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2020-11-16T20:15:34.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:34.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2020-11-16T20:15:34.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:34.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2020-11-16T20:15:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2020-11-16T20:15:35.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:35.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2020-11-16T20:15:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2020-11-16T20:15:36.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:36.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2020-11-16T20:15:36.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:36.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2020-11-16T20:15:36.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:36.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2020-11-16T20:15:36.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:36.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2020-11-16T20:15:36.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:36.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2020-11-16T20:15:37.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:37.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2020-11-16T20:15:37.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:37.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2020-11-16T20:15:37.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:37.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2020-11-16T20:15:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2020-11-16T20:15:37.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:37.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2020-11-16T20:15:38.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:38.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2020-11-16T20:15:38.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:38.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2020-11-16T20:15:38.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:38.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2020-11-16T20:15:38.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:38.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2020-11-16T20:15:38.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:38.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2020-11-16T20:15:39.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:39.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2020-11-16T20:15:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2020-11-16T20:15:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2020-11-16T20:15:39.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:39.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2020-11-16T20:15:39.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:39.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2020-11-16T20:15:40.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:40.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2020-11-16T20:15:40.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:40.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2020-11-16T20:15:40.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:40.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2020-11-16T20:15:40.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:40.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2020-11-16T20:15:41.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:41.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2020-11-16T20:15:41.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:41.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2020-11-16T20:15:41.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:41.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2020-11-16T20:15:41.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:41.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2020-11-16T20:15:45.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2020-11-16T20:15:45.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2020-11-16T20:15:45.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2020-11-16T20:15:45.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2020-11-16T20:15:45.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2020-11-16T20:15:45.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2020-11-16T20:15:45.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2020-11-16T20:15:45.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2020-11-16T20:15:45.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2020-11-16T20:15:45.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2020-11-16T20:15:45.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2020-11-16T20:15:45.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2020-11-16T20:15:45.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2020-11-16T20:15:45.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2020-11-16T20:15:45.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2020-11-16T20:15:45.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2020-11-16T20:15:45.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2020-11-16T20:15:45.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2020-11-16T20:15:45.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2020-11-16T20:15:45.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:45.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2020-11-16T20:15:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:46.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2020-11-16T20:15:46.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:46.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2020-11-16T20:15:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2020-11-16T20:15:46.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:46.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2020-11-16T20:15:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2020-11-16T20:15:47.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:47.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2020-11-16T20:15:47.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:47.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2020-11-16T20:15:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2020-11-16T20:15:47.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:47.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2020-11-16T20:15:47.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:47.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2020-11-16T20:15:48.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:48.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2020-11-16T20:15:48.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:48.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2020-11-16T20:15:48.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:48.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2020-11-16T20:15:48.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:48.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2020-11-16T20:15:48.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:48.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2020-11-16T20:15:49.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:49.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2020-11-16T20:15:49.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:49.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2020-11-16T20:15:49.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:49.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2020-11-16T20:15:49.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:49.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2020-11-16T20:15:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:49.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2020-11-16T20:15:50.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:50.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2020-11-16T20:15:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2020-11-16T20:15:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2020-11-16T20:15:50.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:50.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2020-11-16T20:15:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:50.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2020-11-16T20:15:51.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:51.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2020-11-16T20:15:51.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:51.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2020-11-16T20:15:51.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:51.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2020-11-16T20:15:51.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:51.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2020-11-16T20:15:52.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:52.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2020-11-16T20:15:52.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:52.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2020-11-16T20:15:52.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:52.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2020-11-16T20:15:52.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:52.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2020-11-16T20:15:52.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:52.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2020-11-16T20:15:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:53.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2020-11-16T20:15:53.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:53.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2020-11-16T20:15:53.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:53.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2020-11-16T20:15:53.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:53.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-11-16T20:15:53.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-11-16T20:15:53.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-11-16T20:15:54.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-11-16T20:15:54.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2020-11-16T20:15:54.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:15:54.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:15:54.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:15:54.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:15:54.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2020-11-16T20:15:54.155 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:15:54.155 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:15:54.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:15:54.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:15:54.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:15:54.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:15:54.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:15:54.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:15:54.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:15:54.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:15:54.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:15:54.683 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:15:54.683 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:15:54.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:15:54.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:54.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:15:54.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836500 2020-11-16T20:15:54.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836500 2020-11-16T20:15:54.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500' 2020-11-16T20:15:54.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:54.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:15:55.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672978 2020-11-16T20:15:55.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672978 2020-11-16T20:15:55.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978' 2020-11-16T20:15:55.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:15:55.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:15:55.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509457 2020-11-16T20:15:55.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509457 2020-11-16T20:15:55.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-64424509457' 2020-11-16T20:15:55.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:55.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836500 2020-11-16T20:15:55.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:55.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:15:55.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836500 2020-11-16T20:15:55.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:55.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836500 2020-11-16T20:15:55.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2020-11-16T20:15:55.227 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836500 2020-11-16T20:15:55.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:15:55.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836498 -lt 21474836500 2020-11-16T20:15:55.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:15:59.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:15:59.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:15:59.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836500 2020-11-16T20:15:59.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:59.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672978 2020-11-16T20:15:59.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:59.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:15:59.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672978 2020-11-16T20:15:59.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:59.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672978 2020-11-16T20:15:59.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2020-11-16T20:15:59.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:15:59.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672978 -lt 42949672978 2020-11-16T20:15:59.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:15:59.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:15:59.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509457 2020-11-16T20:15:59.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:15:59.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509457 2020-11-16T20:15:59.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:15:59.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509457 2020-11-16T20:15:59.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2020-11-16T20:15:59.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:15:59.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509457 -lt 64424509457 2020-11-16T20:15:59.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:15:59.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:15:59.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:15:59.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:15:59.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:15:59.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:15:59.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:15:59.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:15:59.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:15:59.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:15:59.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:15:59.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:15:59.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:15:59.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:15:59.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:15:59.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:15:59.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:15:59.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:15:59.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2020-11-16T20:15:59.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T20:15:59.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2020-11-16T20:15:59.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T20:15:59.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2020-11-16T20:15:59.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2020-11-16T20:15:59.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2020-11-16T20:15:59.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:15:59.732 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672978 2020-11-16T20:15:59.733 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509457 2020-11-16T20:15:59.733 INFO:tasks.workunit.client.0.smithi189.stdout:blackholing osds 0 2 2020-11-16T20:15:59.973 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:15:59.973 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2020-11-16T20:15:19.187780+0000 29'100 29:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:15:59.973 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:15:59.973 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:15:59.975 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:15:59.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:15:59.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2020-11-16T20:15:59.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T20:15:59.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T20:15:59.986 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:15:59.986 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:15:59.986 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:15:59.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.0.asok 2020-11-16T20:15:59.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-11-16T20:15:59.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.0.asok config set objectstore_blackhole 1 2020-11-16T20:16:00.082 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:16:00.082 INFO:tasks.workunit.client.0.smithi189.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-11-16T20:16:00.082 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:16:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:16:00.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2020-11-16T20:16:00.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-11-16T20:16:00.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-11-16T20:16:00.091 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:16:00.091 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:16:00.092 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:16:00.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.2.asok 2020-11-16T20:16:00.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-11-16T20:16:00.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.2.asok config set objectstore_blackhole 1 2020-11-16T20:16:00.179 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:16:00.179 INFO:tasks.workunit.client.0.smithi189.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-11-16T20:16:00.179 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:16:00.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2020-11-16T20:16:00.187 INFO:tasks.workunit.client.0.smithi189.stdout:writing divergent object 2020-11-16T20:16:00.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2020-11-16T20:16:00.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2020-11-16T20:16:01.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2020-11-16T20:16:01.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-11-16T20:16:01.466 INFO:tasks.workunit.client.0.smithi189.stdout:created pool test snap snap1 2020-11-16T20:16:01.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2020-11-16T20:16:01.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2020-11-16T20:16:01.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-11-16T20:16:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2020-11-16T20:16:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:rados: no process found 2020-11-16T20:16:11.477 INFO:tasks.workunit.client.0.smithi189.stdout:killing all the osds 2020-11-16T20:16:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2020-11-16T20:16:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:16:11.867 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:16:11.867 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 0 0 0 0 62317 0 0 14 14 active+clean 2020-11-16T20:15:19.187780+0000 30'104 30:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:16:11.867 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:16:11.868 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:16:11.869 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:16:11.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2020-11-16T20:16:11.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:16:11.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:16:11.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:16:11.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:16:11.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:16:11.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:16:11.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-11-16T20:16:11.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2020-11-16T20:16:12.573 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already down. 2020-11-16T20:16:12.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-11-16T20:16:12.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2020-11-16T20:16:13.170 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already down. 2020-11-16T20:16:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-11-16T20:16:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2020-11-16T20:16:13.751 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already down. 2020-11-16T20:16:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:16:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2020-11-16T20:16:14.330 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T20:16:14.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:16:14.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2020-11-16T20:16:14.943 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already out. 2020-11-16T20:16:14.957 INFO:tasks.workunit.client.0.smithi189.stdout:bringing up non_divergent 0 2 2020-11-16T20:16:14.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2020-11-16T20:16:14.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:16:15.331 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:16:15.331 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 0 0 0 0 62317 0 0 14 14 stale+active+clean 2020-11-16T20:15:19.187780+0000 30'104 30:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:16:15.332 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:16:15.332 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:16:15.333 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:16:15.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:16:15.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-11-16T20:16:15.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:16:15.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:16:15.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T20:16:15.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:16:15.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2020-11-16T20:16:15.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:16:15.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:16:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:16:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:16:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-11-16T20:16:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-11-16T20:16:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:16:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:16:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:16:15.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:16:15.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:16:15.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:16:15.348 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:16:15.348 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:16:15.348 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:16:15.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:16:15.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:16:15.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:16:15.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:16:15.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:16:15.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:16:15.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:16:15.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:16:15.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:16:15.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:16:15.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2020-11-16T20:16:15.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T20:16:15.350 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:16:15.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:16:15.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2020-11-16T20:16:15.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T20:16:15.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:16:15.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:16:15.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:16:15.377 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:16:15.374+0000 7f16d6dd4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:16:15.380 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:16:15.374+0000 7f16d6dd4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:16:15.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:16:15.378+0000 7f16d6dd4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:16:15.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T20:16:15.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:16:15.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:16:15.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:16:15.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:16:15.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:16:15.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:16:15.728 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:16:15.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:16:15.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:16:15.914 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:16:15.910+0000 7f16d6dd4d80 -1 Falling back to public interface 2020-11-16T20:16:16.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:16:17.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:16:17.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:16:17.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:16:17.121 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:16:17.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:16:17.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:16:17.250 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:16:17.246+0000 7f16d6dd4d80 -1 osd.0 29 log_to_monitors {default=true} 2020-11-16T20:16:17.553 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/53056,v1:127.0.0.1:6803/53056] [v2:127.0.0.1:6804/53056,v1:127.0.0.1:6805/53056] exists,up 8e23841d-98e5-472c-91fe-6a992d391d13 2020-11-16T20:16:17.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:16:17.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:16:17.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:16:17.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:16:17.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-11-16T20:16:17.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:16:17.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:16:17.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-11-16T20:16:17.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:16:17.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2020-11-16T20:16:17.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:16:17.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:16:17.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:16:17.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:16:17.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-11-16T20:16:17.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-11-16T20:16:17.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:16:17.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:16:17.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:16:17.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:16:17.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:16:17.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:16:17.559 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:16:17.559 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:16:17.559 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:16:17.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:16:17.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:16:17.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:16:17.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:16:17.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:16:17.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:16:17.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:16:17.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:16:17.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:16:17.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:16:17.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2020-11-16T20:16:17.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-11-16T20:16:17.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:16:17.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2020-11-16T20:16:17.561 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:16:17.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-11-16T20:16:17.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:16:17.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:16:17.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:16:17.588 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:16:17.586+0000 7f2fb299cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:16:17.590 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:16:17.586+0000 7f2fb299cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:16:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:16:17.586+0000 7f2fb299cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:16:17.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-11-16T20:16:17.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:16:17.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:16:17.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:16:17.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:16:17.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:16:17.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:16:17.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:16:17.960 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:16:17.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:16:18.134 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:16:18.130+0000 7f2fb299cd80 -1 Falling back to public interface 2020-11-16T20:16:18.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:16:19.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:16:19.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:16:19.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:16:19.373 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:16:19.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:16:19.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:16:19.408 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:16:19.406+0000 7f2fb299cd80 -1 osd.2 29 log_to_monitors {default=true} 2020-11-16T20:16:19.770 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/53455,v1:127.0.0.1:6811/53455] [v2:127.0.0.1:6812/53455,v1:127.0.0.1:6813/53455] exists,up 41ef07c7-ba20-4c7c-b7b2-5bc264c4a07a 2020-11-16T20:16:19.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:16:19.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:16:19.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:16:19.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:16:19.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2020-11-16T20:16:20.358 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already in. 2020-11-16T20:16:20.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:16:20.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2020-11-16T20:16:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already in. 2020-11-16T20:16:20.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2020-11-16T20:16:25.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2020-11-16T20:16:25.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2020-11-16T20:16:25.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2020-11-16T20:16:25.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:16:25.992 INFO:tasks.workunit.client.0.smithi189.stdout:writing non-divergent object existing_4 2020-11-16T20:16:26.382 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:16:26.382 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2020-11-16T20:16:21.401605+0000 29'100 41:124 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:16:26.382 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:16:26.382 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:16:26.384 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:16:26.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2020-11-16T20:16:26.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-11-16T20:16:26.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2020-11-16T20:16:26.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:16:26.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:16:26.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:16:26.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:16:26.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2020-11-16T20:16:26.567 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:16:26.567 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:16:26.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:16:26.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:16:26.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:16:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:16:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:16:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:16:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:16:26.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:16:27.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:16:27.092 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:16:27.092 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:16:27.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:16:27.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:16:27.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:16:27.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822660 2020-11-16T20:16:27.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822660 2020-11-16T20:16:27.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822660' 2020-11-16T20:16:27.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:16:27.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:16:27.393 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-11-16T20:16:27.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T20:16:27.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T20:16:27.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T20:16:27.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:16:27.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:16:27.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789956 2020-11-16T20:16:27.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789956 2020-11-16T20:16:27.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822660 2-158913789956' 2020-11-16T20:16:27.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:16:27.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-154618822660 2020-11-16T20:16:27.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:16:27.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:16:27.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-154618822660 2020-11-16T20:16:27.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:16:27.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822660 2020-11-16T20:16:27.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2020-11-16T20:16:27.579 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 154618822660 2020-11-16T20:16:27.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:16:27.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822658 -lt 154618822660 2020-11-16T20:16:27.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:16:28.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:16:28.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:16:29.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822661 -lt 154618822660 2020-11-16T20:16:29.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:16:29.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-158913789956 2020-11-16T20:16:29.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:16:29.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:16:29.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-158913789956 2020-11-16T20:16:29.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:16:29.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789956 2020-11-16T20:16:29.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 158913789956' 2020-11-16T20:16:29.391 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 158913789956 2020-11-16T20:16:29.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:16:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789956 -lt 158913789956 2020-11-16T20:16:29.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:16:29.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:16:29.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:16:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:16:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:16:30.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:16:30.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:16:30.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:16:30.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:16:30.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:16:30.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:16:30.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:16:30.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:16:30.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:16:30.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:16:31.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:16:31.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:16:31.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:16:31.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:16:31.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:16:31.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:16:31.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:16:31.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:16:31.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:16:31.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:16:31.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:16:31.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:16:31.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:16:31.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:16:31.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:16:31.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:16:31.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:16:31.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:16:32.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:16:32.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:16:32.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:16:32.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:16:32.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:16:32.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:16:32.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:16:32.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:16:32.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:16:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:16:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:16:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2020-11-16T20:16:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:16:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-11-16T20:16:32.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:16:32.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:16:32.822 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:16:32.822 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:16:32.822 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:16:32.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:16:32.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:16:32.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:16:33.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:16:33.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:16:33.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:16:33.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:16:33.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:16:33.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:16:33.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:16:33.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:16:33.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:16:33.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:16:33.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:16:33.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:16:33.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:16:34.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:16:34.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:16:34.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2020-11-16T20:16:34.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:16:34.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2020-11-16T20:16:34.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:16:34.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:16:34.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:16:34.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:16:34.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:16:34.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:16:34.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:16:34.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:16:34.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:16:34.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:16:34.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:16:34.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:16:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:16:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:16:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:16:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:16:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:16:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:16:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:16:35.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:16:35.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:16:35.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:16:35.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:16:35.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2020-11-16T20:16:35.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:16:35.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2020-11-16T20:16:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:16:36.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:16:36.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:16:36.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:16:36.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:16:36.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:16:36.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:16:36.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:16:37.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:16:37.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:16:37.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:16:37.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:16:37.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:16:37.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:16:37.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:16:37.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:16:37.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:16:37.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:16:37.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:16:37.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:16:37.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:16:38.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:16:38.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:16:38.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2020-11-16T20:16:38.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:16:38.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2020-11-16T20:16:39.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:16:39.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:16:39.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:16:39.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:16:39.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:16:39.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:16:39.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:16:39.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:16:40.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:16:40.084 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:16:40.084 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:16:40.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:16:40.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:16:40.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:16:40.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:16:40.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:16:40.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:16:40.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:16:40.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:16:40.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:16:40.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:16:41.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:16:41.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:16:41.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2020-11-16T20:16:41.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:16:41.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2020-11-16T20:16:44.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:16:44.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:16:44.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:16:44.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:16:44.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:16:44.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:16:44.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:16:44.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:16:44.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:16:44.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:16:44.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:16:44.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:16:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:16:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:16:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:16:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:16:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:16:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:16:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:16:45.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:16:45.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:16:45.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:16:45.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:16:45.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2020-11-16T20:16:45.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:16:45.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2020-11-16T20:16:52.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:16:52.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:16:52.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:16:52.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:16:52.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:16:52.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:16:52.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:16:52.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:16:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:16:52.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:16:52.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:16:52.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:16:52.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:16:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:16:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:16:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:16:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:16:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:16:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:16:52.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:16:52.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:16:53.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:16:53.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:16:53.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2020-11-16T20:16:53.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:16:53.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2020-11-16T20:17:00.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:17:00.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:17:00.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:17:00.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:17:00.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:17:00.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:17:00.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:17:00.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:17:01.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:17:01.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:17:01.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:17:01.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:17:01.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:17:01.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:17:01.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:17:01.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:17:01.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:17:01.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:17:01.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:17:01.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:17:01.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:17:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:17:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:17:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2020-11-16T20:17:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2020-11-16T20:17:02.490 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:17:02.490 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_fingerprint": "f0ffaefd-8deb-4050-8593-5c7677a2934c", 2020-11-16T20:17:02.491 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "16.0.0-7219-g911cc357", 2020-11-16T20:17:02.491 INFO:tasks.workunit.client.0.smithi189.stdout: "commit": "911cc35794d871071a8bebfb0919482263b1d18b", 2020-11-16T20:17:02.491 INFO:tasks.workunit.client.0.smithi189.stdout: "timestamp": "2020-11-16T20:17:02.487202+0000", 2020-11-16T20:17:02.491 INFO:tasks.workunit.client.0.smithi189.stdout: "tag": "", 2020-11-16T20:17:02.491 INFO:tasks.workunit.client.0.smithi189.stdout: "health": { 2020-11-16T20:17:02.491 INFO:tasks.workunit.client.0.smithi189.stdout: "status": "HEALTH_WARN", 2020-11-16T20:17:02.492 INFO:tasks.workunit.client.0.smithi189.stdout: "checks": { 2020-11-16T20:17:02.492 INFO:tasks.workunit.client.0.smithi189.stdout: "OSDMAP_FLAGS": { 2020-11-16T20:17:02.492 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:17:02.492 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:17:02.492 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "nodown,noout,noin flag(s) set", 2020-11-16T20:17:02.492 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 17 2020-11-16T20:17:02.492 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.493 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [], 2020-11-16T20:17:02.493 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:17:02.493 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.493 INFO:tasks.workunit.client.0.smithi189.stdout: "OSD_DOWN": { 2020-11-16T20:17:02.493 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:17:02.493 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:17:02.493 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 osds down", 2020-11-16T20:17:02.494 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:17:02.494 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.494 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:17:02.494 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.494 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "osd.1 (root=default,host=smithi189) is down" 2020-11-16T20:17:02.494 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.494 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.495 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:17:02.495 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.495 INFO:tasks.workunit.client.0.smithi189.stdout: "PG_DEGRADED": { 2020-11-16T20:17:02.495 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:17:02.495 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:17:02.495 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-11-16T20:17:02.495 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:17:02.496 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.496 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:17:02.496 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.496 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-11-16T20:17:02.496 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.496 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.496 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:17:02.497 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.497 INFO:tasks.workunit.client.0.smithi189.stdout: "POOL_APP_NOT_ENABLED": { 2020-11-16T20:17:02.497 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:17:02.497 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:17:02.497 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 pool(s) do not have an application enabled", 2020-11-16T20:17:02.497 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:17:02.497 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.498 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:17:02.498 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.498 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "application not enabled on pool 'test'" 2020-11-16T20:17:02.498 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.498 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.498 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-11-16T20:17:02.499 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.499 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.499 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:17:02.499 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.499 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.499 INFO:tasks.workunit.client.0.smithi189.stdout: "mutes": [] 2020-11-16T20:17:02.499 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.500 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_first_committed": 1, 2020-11-16T20:17:02.500 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_last_committed": 1, 2020-11-16T20:17:02.500 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap": { 2020-11-16T20:17:02.500 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:17:02.500 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:17:02.500 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:14:49.332386Z", 2020-11-16T20:17:02.500 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:14:49.332386Z", 2020-11-16T20:17:02.501 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release": 16, 2020-11-16T20:17:02.501 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release_name": "pacific", 2020-11-16T20:17:02.501 INFO:tasks.workunit.client.0.smithi189.stdout: "election_strategy": 1, 2020-11-16T20:17:02.501 INFO:tasks.workunit.client.0.smithi189.stdout: "disallowed_leaders: ": "", 2020-11-16T20:17:02.501 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": false, 2020-11-16T20:17:02.501 INFO:tasks.workunit.client.0.smithi189.stdout: "features": { 2020-11-16T20:17:02.501 INFO:tasks.workunit.client.0.smithi189.stdout: "persistent": [ 2020-11-16T20:17:02.502 INFO:tasks.workunit.client.0.smithi189.stdout: "kraken", 2020-11-16T20:17:02.502 INFO:tasks.workunit.client.0.smithi189.stdout: "luminous", 2020-11-16T20:17:02.502 INFO:tasks.workunit.client.0.smithi189.stdout: "mimic", 2020-11-16T20:17:02.502 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap-prune", 2020-11-16T20:17:02.502 INFO:tasks.workunit.client.0.smithi189.stdout: "nautilus", 2020-11-16T20:17:02.502 INFO:tasks.workunit.client.0.smithi189.stdout: "octopus", 2020-11-16T20:17:02.502 INFO:tasks.workunit.client.0.smithi189.stdout: "pacific", 2020-11-16T20:17:02.503 INFO:tasks.workunit.client.0.smithi189.stdout: "elector-pinging" 2020-11-16T20:17:02.503 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.503 INFO:tasks.workunit.client.0.smithi189.stdout: "optional": [] 2020-11-16T20:17:02.503 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.503 INFO:tasks.workunit.client.0.smithi189.stdout: "mons": [ 2020-11-16T20:17:02.503 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.504 INFO:tasks.workunit.client.0.smithi189.stdout: "rank": 0, 2020-11-16T20:17:02.504 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "a", 2020-11-16T20:17:02.504 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:17:02.504 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.504 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.504 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.505 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115", 2020-11-16T20:17:02.505 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 0 2020-11-16T20:17:02.505 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.505 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.505 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.505 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115/0", 2020-11-16T20:17:02.505 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:7115/0", 2020-11-16T20:17:02.506 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 0, 2020-11-16T20:17:02.506 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 0, 2020-11-16T20:17:02.506 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_location": "{}" 2020-11-16T20:17:02.506 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.506 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.506 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.506 INFO:tasks.workunit.client.0.smithi189.stdout: "quorum": [ 2020-11-16T20:17:02.507 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2020-11-16T20:17:02.507 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.507 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap": { 2020-11-16T20:17:02.507 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 41, 2020-11-16T20:17:02.507 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:17:02.507 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:14:50.055857+0000", 2020-11-16T20:17:02.507 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:16:21.074583+0000", 2020-11-16T20:17:02.508 INFO:tasks.workunit.client.0.smithi189.stdout: "last_up_change": "2020-11-16T20:16:19.485803+0000", 2020-11-16T20:17:02.508 INFO:tasks.workunit.client.0.smithi189.stdout: "last_in_change": "2020-11-16T20:16:20.781603+0000", 2020-11-16T20:17:02.508 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-11-16T20:17:02.508 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_num": 5800384, 2020-11-16T20:17:02.508 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_set": [ 2020-11-16T20:17:02.508 INFO:tasks.workunit.client.0.smithi189.stdout: "nodown", 2020-11-16T20:17:02.508 INFO:tasks.workunit.client.0.smithi189.stdout: "noin", 2020-11-16T20:17:02.509 INFO:tasks.workunit.client.0.smithi189.stdout: "noout", 2020-11-16T20:17:02.509 INFO:tasks.workunit.client.0.smithi189.stdout: "pglog_hardlimit", 2020-11-16T20:17:02.509 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snapdirs", 2020-11-16T20:17:02.509 INFO:tasks.workunit.client.0.smithi189.stdout: "recovery_deletes", 2020-11-16T20:17:02.509 INFO:tasks.workunit.client.0.smithi189.stdout: "sortbitwise" 2020-11-16T20:17:02.509 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.509 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_version": 8, 2020-11-16T20:17:02.510 INFO:tasks.workunit.client.0.smithi189.stdout: "full_ratio": 0.99000000953674316, 2020-11-16T20:17:02.510 INFO:tasks.workunit.client.0.smithi189.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-11-16T20:17:02.510 INFO:tasks.workunit.client.0.smithi189.stdout: "nearfull_ratio": 0.85000002384185791, 2020-11-16T20:17:02.510 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_snapshot": "", 2020-11-16T20:17:02.510 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_max": 1, 2020-11-16T20:17:02.510 INFO:tasks.workunit.client.0.smithi189.stdout: "max_osd": 3, 2020-11-16T20:17:02.510 INFO:tasks.workunit.client.0.smithi189.stdout: "require_min_compat_client": "luminous", 2020-11-16T20:17:02.511 INFO:tasks.workunit.client.0.smithi189.stdout: "min_compat_client": "jewel", 2020-11-16T20:17:02.511 INFO:tasks.workunit.client.0.smithi189.stdout: "require_osd_release": "pacific", 2020-11-16T20:17:02.511 INFO:tasks.workunit.client.0.smithi189.stdout: "pools": [ 2020-11-16T20:17:02.511 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.511 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:17:02.511 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_name": "test", 2020-11-16T20:17:02.511 INFO:tasks.workunit.client.0.smithi189.stdout: "create_time": "2020-11-16T20:15:17.753606+0000", 2020-11-16T20:17:02.511 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": 16389, 2020-11-16T20:17:02.512 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-11-16T20:17:02.512 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 3, 2020-11-16T20:17:02.512 INFO:tasks.workunit.client.0.smithi189.stdout: "size": 3, 2020-11-16T20:17:02.512 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 2, 2020-11-16T20:17:02.512 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_rule": 1, 2020-11-16T20:17:02.512 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_count": 0, 2020-11-16T20:17:02.513 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_target": 0, 2020-11-16T20:17:02.513 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_barrier": 0, 2020-11-16T20:17:02.513 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2020-11-16T20:17:02.513 INFO:tasks.workunit.client.0.smithi189.stdout: "object_hash": 2, 2020-11-16T20:17:02.513 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_autoscale_mode": "off", 2020-11-16T20:17:02.513 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num": 1, 2020-11-16T20:17:02.513 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num": 1, 2020-11-16T20:17:02.514 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num_target": 1, 2020-11-16T20:17:02.514 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_target": 1, 2020-11-16T20:17:02.514 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_pending": 1, 2020-11-16T20:17:02.514 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pg_merge_meta": { 2020-11-16T20:17:02.514 INFO:tasks.workunit.client.0.smithi189.stdout: "source_pgid": "0.0", 2020-11-16T20:17:02.514 INFO:tasks.workunit.client.0.smithi189.stdout: "ready_epoch": 0, 2020-11-16T20:17:02.514 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_started": 0, 2020-11-16T20:17:02.515 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": 0, 2020-11-16T20:17:02.515 INFO:tasks.workunit.client.0.smithi189.stdout: "source_version": "0'0", 2020-11-16T20:17:02.515 INFO:tasks.workunit.client.0.smithi189.stdout: "target_version": "0'0" 2020-11-16T20:17:02.515 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.515 INFO:tasks.workunit.client.0.smithi189.stdout: "last_change": "30", 2020-11-16T20:17:02.515 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend": "0", 2020-11-16T20:17:02.515 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_prenautilus": "0", 2020-11-16T20:17:02.516 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_preluminous": "0", 2020-11-16T20:17:02.516 INFO:tasks.workunit.client.0.smithi189.stdout: "auid": 0, 2020-11-16T20:17:02.516 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_mode": "pool", 2020-11-16T20:17:02.516 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_seq": 1, 2020-11-16T20:17:02.516 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_epoch": 30, 2020-11-16T20:17:02.516 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_snaps": [ 2020-11-16T20:17:02.516 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.517 INFO:tasks.workunit.client.0.smithi189.stdout: "snapid": 1, 2020-11-16T20:17:02.517 INFO:tasks.workunit.client.0.smithi189.stdout: "stamp": "2020-11-16T20:16:01.228891+0000", 2020-11-16T20:17:02.517 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "snap1" 2020-11-16T20:17:02.517 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.517 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.517 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps": "[]", 2020-11-16T20:17:02.517 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_bytes": 0, 2020-11-16T20:17:02.518 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_objects": 0, 2020-11-16T20:17:02.518 INFO:tasks.workunit.client.0.smithi189.stdout: "tiers": [], 2020-11-16T20:17:02.518 INFO:tasks.workunit.client.0.smithi189.stdout: "tier_of": -1, 2020-11-16T20:17:02.518 INFO:tasks.workunit.client.0.smithi189.stdout: "read_tier": -1, 2020-11-16T20:17:02.518 INFO:tasks.workunit.client.0.smithi189.stdout: "write_tier": -1, 2020-11-16T20:17:02.518 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_mode": "none", 2020-11-16T20:17:02.518 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_bytes": 0, 2020-11-16T20:17:02.519 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_objects": 0, 2020-11-16T20:17:02.519 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-11-16T20:17:02.519 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-11-16T20:17:02.519 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_full_ratio_micro": 800000, 2020-11-16T20:17:02.519 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_flush_age": 0, 2020-11-16T20:17:02.519 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_evict_age": 0, 2020-11-16T20:17:02.519 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profile": "myprofile", 2020-11-16T20:17:02.519 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_params": { 2020-11-16T20:17:02.520 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "none" 2020-11-16T20:17:02.520 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.520 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_period": 0, 2020-11-16T20:17:02.520 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_count": 0, 2020-11-16T20:17:02.520 INFO:tasks.workunit.client.0.smithi189.stdout: "use_gmt_hitset": true, 2020-11-16T20:17:02.521 INFO:tasks.workunit.client.0.smithi189.stdout: "min_read_recency_for_promote": 0, 2020-11-16T20:17:02.521 INFO:tasks.workunit.client.0.smithi189.stdout: "min_write_recency_for_promote": 0, 2020-11-16T20:17:02.521 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_grade_decay_rate": 0, 2020-11-16T20:17:02.521 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_search_last_n": 0, 2020-11-16T20:17:02.521 INFO:tasks.workunit.client.0.smithi189.stdout: "grade_table": [], 2020-11-16T20:17:02.521 INFO:tasks.workunit.client.0.smithi189.stdout: "stripe_width": 8192, 2020-11-16T20:17:02.522 INFO:tasks.workunit.client.0.smithi189.stdout: "expected_num_objects": 0, 2020-11-16T20:17:02.522 INFO:tasks.workunit.client.0.smithi189.stdout: "fast_read": false, 2020-11-16T20:17:02.522 INFO:tasks.workunit.client.0.smithi189.stdout: "options": {}, 2020-11-16T20:17:02.522 INFO:tasks.workunit.client.0.smithi189.stdout: "application_metadata": {} 2020-11-16T20:17:02.522 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.522 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.523 INFO:tasks.workunit.client.0.smithi189.stdout: "osds": [ 2020-11-16T20:17:02.523 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.523 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:17:02.523 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "8e23841d-98e5-472c-91fe-6a992d391d13", 2020-11-16T20:17:02.523 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:17:02.523 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:17:02.523 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:17:02.523 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:17:02.524 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 5, 2020-11-16T20:17:02.524 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 29, 2020-11-16T20:17:02.524 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 36, 2020-11-16T20:17:02.524 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 40, 2020-11-16T20:17:02.524 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 31, 2020-11-16T20:17:02.524 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:17:02.524 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:17:02.525 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.525 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.525 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.525 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6802", 2020-11-16T20:17:02.525 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53056 2020-11-16T20:17:02.525 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.525 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.526 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.526 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6803", 2020-11-16T20:17:02.526 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53056 2020-11-16T20:17:02.526 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.526 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.526 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.526 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:17:02.527 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.527 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.527 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.527 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6804", 2020-11-16T20:17:02.527 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53056 2020-11-16T20:17:02.527 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.528 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.528 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.528 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6805", 2020-11-16T20:17:02.528 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53056 2020-11-16T20:17:02.528 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.528 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.528 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.529 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:17:02.529 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.529 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.529 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.529 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6808", 2020-11-16T20:17:02.529 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53056 2020-11-16T20:17:02.529 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.530 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.530 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.530 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6809", 2020-11-16T20:17:02.530 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53056 2020-11-16T20:17:02.530 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.530 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.530 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.531 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:17:02.531 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.531 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.531 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.531 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6806", 2020-11-16T20:17:02.531 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53056 2020-11-16T20:17:02.531 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.532 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.532 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.532 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6807", 2020-11-16T20:17:02.532 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53056 2020-11-16T20:17:02.532 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.532 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.532 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.533 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6803/53056", 2020-11-16T20:17:02.533 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6805/53056", 2020-11-16T20:17:02.533 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6809/53056", 2020-11-16T20:17:02.533 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6807/53056", 2020-11-16T20:17:02.533 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:17:02.533 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:17:02.533 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:17:02.533 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.534 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.534 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.534 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:17:02.534 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "fada656c-964b-4065-ada2-6f88fccdceb7", 2020-11-16T20:17:02.534 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 0, 2020-11-16T20:17:02.534 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:17:02.534 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:17:02.535 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:17:02.535 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 0, 2020-11-16T20:17:02.535 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 0, 2020-11-16T20:17:02.535 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 10, 2020-11-16T20:17:02.535 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 25, 2020-11-16T20:17:02.535 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 32, 2020-11-16T20:17:02.535 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:17:02.536 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:17:02.536 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.536 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.536 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.536 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:17:02.536 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:02.536 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.537 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.537 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.537 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:17:02.537 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:02.537 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.537 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.537 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.538 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:17:02.538 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.538 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.538 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.538 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:17:02.538 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:02.538 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.539 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.539 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.539 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:17:02.539 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:02.539 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.539 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.539 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.540 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:17:02.540 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.540 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.540 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.540 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:17:02.540 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:02.540 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.541 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.541 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.541 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:17:02.541 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:02.541 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.541 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.541 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.542 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:17:02.542 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.542 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.543 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.543 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:17:02.543 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:02.543 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.543 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.543 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.543 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:17:02.544 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:02.544 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.544 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.544 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.544 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/47811", 2020-11-16T20:17:02.544 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/47811", 2020-11-16T20:17:02.544 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/47811", 2020-11-16T20:17:02.545 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/47811", 2020-11-16T20:17:02.545 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:17:02.545 INFO:tasks.workunit.client.0.smithi189.stdout: "exists" 2020-11-16T20:17:02.545 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.545 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.545 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.545 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:17:02.545 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "41ef07c7-ba20-4c7c-b7b2-5bc264c4a07a", 2020-11-16T20:17:02.546 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:17:02.546 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:17:02.546 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:17:02.546 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:17:02.546 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 15, 2020-11-16T20:17:02.547 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 29, 2020-11-16T20:17:02.547 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 37, 2020-11-16T20:17:02.547 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 0, 2020-11-16T20:17:02.547 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 33, 2020-11-16T20:17:02.547 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:17:02.547 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:17:02.547 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.548 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.548 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.548 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:17:02.548 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53455 2020-11-16T20:17:02.548 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.548 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.548 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.549 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:17:02.549 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53455 2020-11-16T20:17:02.549 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.549 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.549 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.549 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:17:02.549 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.550 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.550 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.550 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:17:02.550 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53455 2020-11-16T20:17:02.550 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.550 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.550 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.551 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:17:02.551 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53455 2020-11-16T20:17:02.551 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.551 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.551 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.551 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:17:02.551 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.551 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.552 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.552 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:17:02.552 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53455 2020-11-16T20:17:02.552 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.552 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.552 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.552 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:17:02.552 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53455 2020-11-16T20:17:02.553 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.553 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.553 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.553 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:17:02.553 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:02.553 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.553 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:02.554 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:17:02.554 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53455 2020-11-16T20:17:02.554 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.554 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.554 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:02.554 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:17:02.554 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 53455 2020-11-16T20:17:02.555 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.555 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.555 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.555 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/53455", 2020-11-16T20:17:02.555 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/53455", 2020-11-16T20:17:02.555 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/53455", 2020-11-16T20:17:02.555 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/53455", 2020-11-16T20:17:02.555 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:17:02.556 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:17:02.556 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:17:02.556 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.556 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.556 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.556 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_xinfo": [ 2020-11-16T20:17:02.556 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.557 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:17:02.557 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:16:12.348105+0000", 2020-11-16T20:17:02.557 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:17:02.557 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:17:02.557 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:17:02.557 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:17:02.557 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:14:58.521932+0000", 2020-11-16T20:17:02.557 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:17:02.558 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.558 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.558 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:17:02.558 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:16:12.931373+0000", 2020-11-16T20:17:02.558 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:17:02.558 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:17:02.558 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:17:02.559 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:17:02.559 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:15:06.606321+0000", 2020-11-16T20:17:02.559 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:17:02.559 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.559 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.559 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:17:02.559 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:16:13.519146+0000", 2020-11-16T20:17:02.560 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:17:02.560 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:17:02.560 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:17:02.560 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:17:02.560 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:15:14.477925+0000", 2020-11-16T20:17:02.560 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:17:02.560 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.560 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.561 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap": [], 2020-11-16T20:17:02.561 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap_items": [], 2020-11-16T20:17:02.561 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_temp": [], 2020-11-16T20:17:02.561 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_temp": [], 2020-11-16T20:17:02.561 INFO:tasks.workunit.client.0.smithi189.stdout: "blocklist": {}, 2020-11-16T20:17:02.561 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profiles": { 2020-11-16T20:17:02.561 INFO:tasks.workunit.client.0.smithi189.stdout: "default": { 2020-11-16T20:17:02.562 INFO:tasks.workunit.client.0.smithi189.stdout: "k": "2", 2020-11-16T20:17:02.562 INFO:tasks.workunit.client.0.smithi189.stdout: "m": "2", 2020-11-16T20:17:02.562 INFO:tasks.workunit.client.0.smithi189.stdout: "plugin": "jerasure", 2020-11-16T20:17:02.562 INFO:tasks.workunit.client.0.smithi189.stdout: "technique": "reed_sol_van" 2020-11-16T20:17:02.562 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.562 INFO:tasks.workunit.client.0.smithi189.stdout: "myprofile": { 2020-11-16T20:17:02.562 INFO:tasks.workunit.client.0.smithi189.stdout: "crush-device-class": "", 2020-11-16T20:17:02.563 INFO:tasks.workunit.client.0.smithi189.stdout: "crush-failure-domain": "osd", 2020-11-16T20:17:02.563 INFO:tasks.workunit.client.0.smithi189.stdout: "crush-root": "default", 2020-11-16T20:17:02.563 INFO:tasks.workunit.client.0.smithi189.stdout: "jerasure-per-chunk-alignment": "false", 2020-11-16T20:17:02.563 INFO:tasks.workunit.client.0.smithi189.stdout: "k": "2", 2020-11-16T20:17:02.563 INFO:tasks.workunit.client.0.smithi189.stdout: "m": "1", 2020-11-16T20:17:02.563 INFO:tasks.workunit.client.0.smithi189.stdout: "plugin": "jerasure", 2020-11-16T20:17:02.563 INFO:tasks.workunit.client.0.smithi189.stdout: "technique": "reed_sol_van", 2020-11-16T20:17:02.563 INFO:tasks.workunit.client.0.smithi189.stdout: "w": "8" 2020-11-16T20:17:02.564 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.564 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.564 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps_queue": [], 2020-11-16T20:17:02.564 INFO:tasks.workunit.client.0.smithi189.stdout: "new_removed_snaps": [], 2020-11-16T20:17:02.564 INFO:tasks.workunit.client.0.smithi189.stdout: "new_purged_snaps": [], 2020-11-16T20:17:02.564 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_node_flags": {}, 2020-11-16T20:17:02.564 INFO:tasks.workunit.client.0.smithi189.stdout: "device_class_flags": {}, 2020-11-16T20:17:02.565 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": { 2020-11-16T20:17:02.565 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_enabled": false, 2020-11-16T20:17:02.565 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_bucket_count": 0, 2020-11-16T20:17:02.565 INFO:tasks.workunit.client.0.smithi189.stdout: "degraded_stretch_mode": 0, 2020-11-16T20:17:02.565 INFO:tasks.workunit.client.0.smithi189.stdout: "recovering_stretch_mode": 0, 2020-11-16T20:17:02.565 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_bucket": 0 2020-11-16T20:17:02.565 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.565 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.566 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_metadata": [ 2020-11-16T20:17:02.566 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.566 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:17:02.566 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:17:02.566 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6804/53056,v1:127.0.0.1:6805/53056]", 2020-11-16T20:17:02.566 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:17:02.566 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:17:02.567 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:17:02.567 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:17:02.567 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:17:02.567 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:17:02.567 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:17:02.567 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:17:02.567 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:17:02.568 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-11-16T20:17:02.568 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:17:02.568 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:17:02.568 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:17:02.568 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:17:02.568 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:17:02.568 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:17:02.568 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:17:02.569 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:17:02.569 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:17:02.569 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:17:02.569 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:17:02.569 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:17:02.569 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:17:02.569 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:17:02.569 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:17:02.570 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6802/53056,v1:127.0.0.1:6803/53056]", 2020-11-16T20:17:02.570 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:17:02.570 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/53056,v1:127.0.0.1:6809/53056]", 2020-11-16T20:17:02.570 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/53056,v1:127.0.0.1:6807/53056]", 2020-11-16T20:17:02.570 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:17:02.570 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:17:02.570 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:17:02.571 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:17:02.571 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:17:02.571 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:17:02.571 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:17:02.571 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:17:02.571 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:17:02.571 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/0", 2020-11-16T20:17:02.572 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:17:02.572 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:17:02.572 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:17:02.572 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.572 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.572 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:17:02.572 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:17:02.572 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/47811,v1:127.0.0.1:6813/47811]", 2020-11-16T20:17:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:17:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:17:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:17:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:17:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:17:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:17:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:17:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:17:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:17:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-11-16T20:17:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:17:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:17:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:17:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:17:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:17:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:17:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:17:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:17:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:17:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:17:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:17:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:17:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:17:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:17:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:17:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/47811,v1:127.0.0.1:6811/47811]", 2020-11-16T20:17:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:17:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/47811,v1:127.0.0.1:6817/47811]", 2020-11-16T20:17:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/47811,v1:127.0.0.1:6815/47811]", 2020-11-16T20:17:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:17:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:17:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:17:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:17:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:17:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:17:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:17:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:17:02.578 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:17:02.578 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/1", 2020-11-16T20:17:02.578 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:17:02.578 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:17:02.578 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:17:02.578 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.578 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.579 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:17:02.579 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:17:02.579 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/53455,v1:127.0.0.1:6813/53455]", 2020-11-16T20:17:02.579 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:17:02.579 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:17:02.579 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:17:02.579 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:17:02.579 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:17:02.580 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:17:02.580 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:17:02.580 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:17:02.580 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:17:02.580 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-11-16T20:17:02.580 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:17:02.580 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:17:02.581 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:17:02.581 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:17:02.581 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:17:02.581 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:17:02.581 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:17:02.581 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:17:02.581 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:17:02.581 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:17:02.582 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:17:02.582 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:17:02.582 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:17:02.582 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:17:02.582 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:17:02.582 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/53455,v1:127.0.0.1:6811/53455]", 2020-11-16T20:17:02.582 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:17:02.583 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/53455,v1:127.0.0.1:6817/53455]", 2020-11-16T20:17:02.583 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/53455,v1:127.0.0.1:6815/53455]", 2020-11-16T20:17:02.583 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:17:02.583 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:17:02.583 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:17:02.583 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:17:02.583 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:17:02.584 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:17:02.584 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:17:02.584 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:17:02.584 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:17:02.584 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/2", 2020-11-16T20:17:02.584 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:17:02.584 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:17:02.584 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:17:02.585 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.585 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.585 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_clean_epochs": { 2020-11-16T20:17:02.585 INFO:tasks.workunit.client.0.smithi189.stdout: "min_last_epoch_clean": 0, 2020-11-16T20:17:02.585 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": { 2020-11-16T20:17:02.585 INFO:tasks.workunit.client.0.smithi189.stdout: "per_pool": [] 2020-11-16T20:17:02.585 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.586 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_epochs": [ 2020-11-16T20:17:02.586 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.586 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:17:02.586 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 36 2020-11-16T20:17:02.586 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.586 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.586 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:17:02.586 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 10 2020-11-16T20:17:02.587 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.587 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.587 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:17:02.587 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 38 2020-11-16T20:17:02.587 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.587 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.587 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.588 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_first_committed": 1, 2020-11-16T20:17:02.588 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_last_committed": 41, 2020-11-16T20:17:02.588 INFO:tasks.workunit.client.0.smithi189.stdout: "crushmap": { 2020-11-16T20:17:02.588 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": [ 2020-11-16T20:17:02.588 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.588 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:17:02.588 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.0", 2020-11-16T20:17:02.588 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:17:02.589 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.589 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.589 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:17:02.589 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.1", 2020-11-16T20:17:02.589 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:17:02.589 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.589 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.590 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:17:02.590 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.2", 2020-11-16T20:17:02.590 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:17:02.590 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.590 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.590 INFO:tasks.workunit.client.0.smithi189.stdout: "types": [ 2020-11-16T20:17:02.590 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.590 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 0, 2020-11-16T20:17:02.591 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd" 2020-11-16T20:17:02.591 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.591 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.591 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:17:02.591 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "host" 2020-11-16T20:17:02.591 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.591 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.592 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 2, 2020-11-16T20:17:02.592 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "chassis" 2020-11-16T20:17:02.592 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.592 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.592 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 3, 2020-11-16T20:17:02.592 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "rack" 2020-11-16T20:17:02.592 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.592 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.593 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 4, 2020-11-16T20:17:02.593 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "row" 2020-11-16T20:17:02.593 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.593 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.593 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 5, 2020-11-16T20:17:02.593 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pdu" 2020-11-16T20:17:02.593 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.594 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.594 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 6, 2020-11-16T20:17:02.594 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pod" 2020-11-16T20:17:02.594 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.594 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.594 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 7, 2020-11-16T20:17:02.594 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "room" 2020-11-16T20:17:02.595 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.595 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.595 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 8, 2020-11-16T20:17:02.595 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "datacenter" 2020-11-16T20:17:02.595 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.595 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.595 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 9, 2020-11-16T20:17:02.595 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "zone" 2020-11-16T20:17:02.596 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.596 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.596 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 10, 2020-11-16T20:17:02.596 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "region" 2020-11-16T20:17:02.596 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.596 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.596 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:17:02.596 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "root" 2020-11-16T20:17:02.597 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.597 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.597 INFO:tasks.workunit.client.0.smithi189.stdout: "buckets": [ 2020-11-16T20:17:02.597 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.597 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -1, 2020-11-16T20:17:02.597 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default", 2020-11-16T20:17:02.597 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:17:02.598 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:17:02.598 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:02.598 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:17:02.598 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:17:02.598 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:17:02.598 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.598 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:17:02.599 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:02.599 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:17:02.599 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.599 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.599 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.599 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.599 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -2, 2020-11-16T20:17:02.599 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default~hdd", 2020-11-16T20:17:02.600 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:17:02.600 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:17:02.600 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:02.600 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:17:02.600 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:17:02.600 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:17:02.600 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.601 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:17:02.601 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:02.601 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:17:02.601 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.601 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.601 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.601 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.601 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:17:02.602 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189", 2020-11-16T20:17:02.602 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:17:02.602 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:17:02.602 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:02.602 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:17:02.602 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:17:02.602 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:17:02.603 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.603 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:17:02.603 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:02.603 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:17:02.603 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.603 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.603 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:17:02.603 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:02.604 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:17:02.604 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.604 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.604 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:17:02.604 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:02.604 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:17:02.604 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.604 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.605 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.605 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.605 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:17:02.605 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189~hdd", 2020-11-16T20:17:02.605 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:17:02.605 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:17:02.605 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:02.606 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:17:02.606 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:17:02.606 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:17:02.606 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.606 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:17:02.606 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:02.606 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:17:02.606 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.607 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.607 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:17:02.607 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:02.607 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:17:02.607 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.607 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.607 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:17:02.608 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:02.608 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:17:02.608 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.608 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.608 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.608 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.608 INFO:tasks.workunit.client.0.smithi189.stdout: "rules": [ 2020-11-16T20:17:02.608 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.609 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 0, 2020-11-16T20:17:02.609 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "replicated_rule", 2020-11-16T20:17:02.609 INFO:tasks.workunit.client.0.smithi189.stdout: "ruleset": 0, 2020-11-16T20:17:02.609 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:17:02.609 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 1, 2020-11-16T20:17:02.609 INFO:tasks.workunit.client.0.smithi189.stdout: "max_size": 10, 2020-11-16T20:17:02.609 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2020-11-16T20:17:02.610 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.610 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2020-11-16T20:17:02.610 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2020-11-16T20:17:02.610 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2020-11-16T20:17:02.610 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.610 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.610 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_firstn", 2020-11-16T20:17:02.610 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2020-11-16T20:17:02.611 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2020-11-16T20:17:02.611 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.611 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.611 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2020-11-16T20:17:02.611 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.611 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.611 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.612 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.612 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 1, 2020-11-16T20:17:02.612 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "test", 2020-11-16T20:17:02.612 INFO:tasks.workunit.client.0.smithi189.stdout: "ruleset": 1, 2020-11-16T20:17:02.612 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 3, 2020-11-16T20:17:02.612 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 3, 2020-11-16T20:17:02.612 INFO:tasks.workunit.client.0.smithi189.stdout: "max_size": 3, 2020-11-16T20:17:02.612 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2020-11-16T20:17:02.613 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.613 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_chooseleaf_tries", 2020-11-16T20:17:02.613 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 5 2020-11-16T20:17:02.613 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.613 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.613 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_choose_tries", 2020-11-16T20:17:02.613 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 100 2020-11-16T20:17:02.614 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.614 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.614 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2020-11-16T20:17:02.614 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2020-11-16T20:17:02.614 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2020-11-16T20:17:02.614 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.614 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.614 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_indep", 2020-11-16T20:17:02.615 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2020-11-16T20:17:02.615 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2020-11-16T20:17:02.615 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.615 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.615 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2020-11-16T20:17:02.615 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.615 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:02.616 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.616 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.616 INFO:tasks.workunit.client.0.smithi189.stdout: "tunables": { 2020-11-16T20:17:02.616 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_tries": 0, 2020-11-16T20:17:02.616 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_fallback_tries": 0, 2020-11-16T20:17:02.616 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_total_tries": 50, 2020-11-16T20:17:02.616 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_descend_once": 1, 2020-11-16T20:17:02.616 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_vary_r": 1, 2020-11-16T20:17:02.617 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_stable": 1, 2020-11-16T20:17:02.617 INFO:tasks.workunit.client.0.smithi189.stdout: "straw_calc_version": 1, 2020-11-16T20:17:02.617 INFO:tasks.workunit.client.0.smithi189.stdout: "allowed_bucket_algs": 54, 2020-11-16T20:17:02.617 INFO:tasks.workunit.client.0.smithi189.stdout: "profile": "jewel", 2020-11-16T20:17:02.617 INFO:tasks.workunit.client.0.smithi189.stdout: "optimal_tunables": 1, 2020-11-16T20:17:02.618 INFO:tasks.workunit.client.0.smithi189.stdout: "legacy_tunables": 0, 2020-11-16T20:17:02.618 INFO:tasks.workunit.client.0.smithi189.stdout: "minimum_required_version": "jewel", 2020-11-16T20:17:02.618 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables": 1, 2020-11-16T20:17:02.618 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables2": 1, 2020-11-16T20:17:02.618 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v2_rules": 1, 2020-11-16T20:17:02.618 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables3": 1, 2020-11-16T20:17:02.619 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v3_rules": 0, 2020-11-16T20:17:02.619 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v4_buckets": 1, 2020-11-16T20:17:02.619 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables5": 1, 2020-11-16T20:17:02.619 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v5_rules": 0 2020-11-16T20:17:02.619 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.619 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_args": {} 2020-11-16T20:17:02.619 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.619 INFO:tasks.workunit.client.0.smithi189.stdout: "fsmap": { 2020-11-16T20:17:02.620 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:17:02.620 INFO:tasks.workunit.client.0.smithi189.stdout: "default_fscid": -1, 2020-11-16T20:17:02.620 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": { 2020-11-16T20:17:02.620 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": {}, 2020-11-16T20:17:02.620 INFO:tasks.workunit.client.0.smithi189.stdout: "ro_compat": {}, 2020-11-16T20:17:02.620 INFO:tasks.workunit.client.0.smithi189.stdout: "incompat": { 2020-11-16T20:17:02.620 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_1": "base v0.20", 2020-11-16T20:17:02.621 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_2": "client writeable ranges", 2020-11-16T20:17:02.621 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_3": "default file layouts on dirs", 2020-11-16T20:17:02.621 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_4": "dir inode in separate object", 2020-11-16T20:17:02.621 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_5": "mds uses versioned encoding", 2020-11-16T20:17:02.621 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_6": "dirfrag is stored in omap", 2020-11-16T20:17:02.621 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_8": "no anchor table", 2020-11-16T20:17:02.621 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_9": "file layout v2", 2020-11-16T20:17:02.621 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_10": "snaprealm v2" 2020-11-16T20:17:02.622 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.622 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.622 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_flags": { 2020-11-16T20:17:02.622 INFO:tasks.workunit.client.0.smithi189.stdout: "enable_multiple": false, 2020-11-16T20:17:02.622 INFO:tasks.workunit.client.0.smithi189.stdout: "ever_enabled_multiple": false 2020-11-16T20:17:02.622 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.622 INFO:tasks.workunit.client.0.smithi189.stdout: "standbys": [], 2020-11-16T20:17:02.623 INFO:tasks.workunit.client.0.smithi189.stdout: "filesystems": [] 2020-11-16T20:17:02.623 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.623 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_first_committed": 1, 2020-11-16T20:17:02.623 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_last_committed": 1, 2020-11-16T20:17:02.623 INFO:tasks.workunit.client.0.smithi189.stdout: "auth": { 2020-11-16T20:17:02.623 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:17:02.623 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 5, 2020-11-16T20:17:02.624 INFO:tasks.workunit.client.0.smithi189.stdout: "num_secrets": 3 2020-11-16T20:17:02.624 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.624 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 1, 2020-11-16T20:17:02.624 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_active": 1, 2020-11-16T20:17:02.624 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_unknown": 0, 2020-11-16T20:17:02.624 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osd": 3, 2020-11-16T20:17:02.624 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_sum": { 2020-11-16T20:17:02.624 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:17:02.625 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 62471, 2020-11-16T20:17:02.625 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 101, 2020-11-16T20:17:02.625 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 1, 2020-11-16T20:17:02.625 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 303, 2020-11-16T20:17:02.625 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:17:02.625 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:17:02.625 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 101, 2020-11-16T20:17:02.626 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:17:02.626 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:17:02.626 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 101, 2020-11-16T20:17:02.626 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:17:02.626 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:17:02.626 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:17:02.626 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 101, 2020-11-16T20:17:02.626 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 101, 2020-11-16T20:17:02.627 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:17:02.627 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:17:02.627 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:17:02.627 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:17:02.627 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:17:02.627 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:17:02.627 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:17:02.628 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:17:02.628 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:17:02.628 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:17:02.628 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:17:02.628 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:17:02.628 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:17:02.628 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:17:02.628 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:17:02.629 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:17:02.629 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:17:02.629 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:17:02.629 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:17:02.629 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:17:02.629 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:17:02.629 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:17:02.630 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:17:02.630 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:17:02.630 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:17:02.630 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.630 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:17:02.630 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:17:02.630 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:17:02.630 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:17:02.631 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 0, 2020-11-16T20:17:02.631 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 0, 2020-11-16T20:17:02.631 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:17:02.631 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:17:02.631 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:17:02.631 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:17:02.631 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:17:02.631 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.632 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 18, 2020-11-16T20:17:02.632 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 18, 2020-11-16T20:17:02.632 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 3, 2020-11-16T20:17:02.632 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 3, 2020-11-16T20:17:02.632 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 0 2020-11-16T20:17:02.632 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.632 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum": { 2020-11-16T20:17:02.633 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:17:02.633 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:17:02.633 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 3, 2020-11-16T20:17:02.633 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:17:02.633 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:17:02.633 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:17:02.633 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:17:02.633 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 18420, 2020-11-16T20:17:02.634 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2132, 2020-11-16T20:17:02.634 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:17:02.634 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 16254, 2020-11-16T20:17:02.634 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314554380, 2020-11-16T20:17:02.634 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:17:02.634 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:17:02.634 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322103685120, 2020-11-16T20:17:02.635 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:17:02.635 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2183168, 2020-11-16T20:17:02.635 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 1530053, 2020-11-16T20:17:02.635 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:17:02.635 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:17:02.635 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:17:02.635 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:17:02.635 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 16644216 2020-11-16T20:17:02.636 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.636 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:17:02.636 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:17:02.636 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:17:02.636 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:17:02.636 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:17:02.636 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:17:02.637 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:17:02.637 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.637 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:17:02.637 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 0, 2020-11-16T20:17:02.637 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 0, 2020-11-16T20:17:02.637 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 0, 2020-11-16T20:17:02.637 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 0 2020-11-16T20:17:02.637 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.638 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:17:02.638 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:17:02.638 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.638 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum_by_class": { 2020-11-16T20:17:02.638 INFO:tasks.workunit.client.0.smithi189.stdout: "hdd": { 2020-11-16T20:17:02.639 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:17:02.639 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:17:02.639 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 3, 2020-11-16T20:17:02.639 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:17:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:17:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:17:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:17:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 18420, 2020-11-16T20:17:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2132, 2020-11-16T20:17:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 1, 2020-11-16T20:17:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 16254, 2020-11-16T20:17:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314554380, 2020-11-16T20:17:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:17:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:17:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322103685120, 2020-11-16T20:17:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:17:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2183168, 2020-11-16T20:17:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 1530053, 2020-11-16T20:17:02.642 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:17:02.642 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:17:02.642 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:17:02.642 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 1928, 2020-11-16T20:17:02.642 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 16644216 2020-11-16T20:17:02.642 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.643 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:17:02.643 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:17:02.643 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:17:02.643 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:17:02.643 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:17:02.643 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:17:02.643 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:17:02.644 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.644 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:17:02.644 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 0, 2020-11-16T20:17:02.644 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 0, 2020-11-16T20:17:02.644 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 0, 2020-11-16T20:17:02.644 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 0 2020-11-16T20:17:02.644 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.644 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:17:02.645 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:17:02.645 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.645 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.645 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_stats": [ 2020-11-16T20:17:02.645 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.645 INFO:tasks.workunit.client.0.smithi189.stdout: "poolid": 1, 2020-11-16T20:17:02.645 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 1, 2020-11-16T20:17:02.646 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:17:02.646 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 62471, 2020-11-16T20:17:02.646 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 101, 2020-11-16T20:17:02.646 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 1, 2020-11-16T20:17:02.646 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 303, 2020-11-16T20:17:02.646 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:17:02.646 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:17:02.646 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 101, 2020-11-16T20:17:02.647 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:17:02.647 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:17:02.647 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 101, 2020-11-16T20:17:02.647 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:17:02.647 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:17:02.647 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:17:02.647 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 101, 2020-11-16T20:17:02.648 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 101, 2020-11-16T20:17:02.648 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:17:02.648 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:17:02.648 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:17:02.648 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:17:02.648 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:17:02.648 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:17:02.648 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:17:02.649 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:17:02.649 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:17:02.649 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:17:02.649 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:17:02.649 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:17:02.649 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:17:02.649 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:17:02.649 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:17:02.650 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:17:02.650 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:17:02.650 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:17:02.650 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:17:02.650 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:17:02.650 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:17:02.650 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:17:02.651 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:17:02.651 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:17:02.651 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:17:02.651 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.651 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:17:02.651 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:17:02.651 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:17:02.652 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:17:02.652 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 1241088, 2020-11-16T20:17:02.652 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 1241088, 2020-11-16T20:17:02.652 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:17:02.652 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:17:02.652 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:17:02.652 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:17:02.652 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:17:02.653 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.653 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 18, 2020-11-16T20:17:02.653 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 18, 2020-11-16T20:17:02.653 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 3, 2020-11-16T20:17:02.653 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 3, 2020-11-16T20:17:02.653 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 3 2020-11-16T20:17:02.653 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.654 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.654 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_stats": [ 2020-11-16T20:17:02.654 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.654 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:17:02.654 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 154618822667 2020-11-16T20:17:02.654 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.654 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.654 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:17:02.655 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 42949672981 2020-11-16T20:17:02.655 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.655 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.655 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:17:02.655 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 158913789963 2020-11-16T20:17:02.655 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.655 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.655 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_state": [ 2020-11-16T20:17:02.656 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.656 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "active+undersized+degraded", 2020-11-16T20:17:02.656 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 1 2020-11-16T20:17:02.656 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.656 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.656 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_osd": [ 2020-11-16T20:17:02.656 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.657 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:17:02.657 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:17:02.657 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:17:02.657 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:17:02.657 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.657 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.657 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2147483647, 2020-11-16T20:17:02.657 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:17:02.658 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:17:02.658 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:17:02.658 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.658 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.658 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:17:02.658 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:17:02.658 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 0, 2020-11-16T20:17:02.659 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:17:02.659 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.659 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.659 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:17:02.659 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 1, 2020-11-16T20:17:02.659 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:17:02.659 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:17:02.659 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.660 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.660 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": [ 2020-11-16T20:17:02.660 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:02.660 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:17:02.660 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": {} 2020-11-16T20:17:02.660 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.660 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:02.661 INFO:tasks.workunit.client.0.smithi189.stdout: "servicemap": { 2020-11-16T20:17:02.661 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 5, 2020-11-16T20:17:02.661 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:16:16.217809+0000", 2020-11-16T20:17:02.661 INFO:tasks.workunit.client.0.smithi189.stdout: "services": {} 2020-11-16T20:17:02.661 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:02.661 INFO:tasks.workunit.client.0.smithi189.stdout: "paxos": { 2020-11-16T20:17:02.661 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:17:02.661 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 390, 2020-11-16T20:17:02.662 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pn": 0, 2020-11-16T20:17:02.662 INFO:tasks.workunit.client.0.smithi189.stdout: "accepted_pn": 0 2020-11-16T20:17:02.662 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:02.662 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:17:02.663 INFO:tasks.workunit.client.0.smithi189.stderr:report 1760149682 2020-11-16T20:17:02.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2020-11-16T20:17:02.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:17:02.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2020-11-16T20:17:02.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:17:02.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:17:02.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:17:02.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:17:02.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:17:02.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:17:02.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2020-11-16T20:17:02.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-11-16T20:17:02.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T20:17:02.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=0 2020-11-16T20:17:02.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T20:17:02.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2020-11-16T20:17:02.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2020-11-16T20:17:03.611 INFO:tasks.workunit.client.0.smithi189.stderr:PG '1.0' not found 2020-11-16T20:17:04.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: return 1 2020-11-16T20:17:04.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-11-16T20:17:04.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:17:04.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:17:04.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T20:17:04.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:17:04.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2020-11-16T20:17:04.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:17:04.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:17:04.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:17:04.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:17:04.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-11-16T20:17:04.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-11-16T20:17:04.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:17:04.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:17:04.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:17:04.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:17:04.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:17:04.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:17:04.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:17:04.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:17:04.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:17:04.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:17:04.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:17:04.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:17:04.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:17:04.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:17:04.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:17:04.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:17:04.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:17:04.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:17:04.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:17:04.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2020-11-16T20:17:04.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T20:17:04.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:17:04.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2020-11-16T20:17:04.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T20:17:04.154 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:17:04.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:17:04.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:17:04.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:17:04.186 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:04.181+0000 7fdfc3a6dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:17:04.190 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:04.185+0000 7fdfc3a6dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:17:04.191 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:04.185+0000 7fdfc3a6dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:17:04.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T20:17:04.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:17:04.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:17:04.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:17:04.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:17:04.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:17:04.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:17:04.594 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:17:04.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:17:04.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:17:04.729 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:04.725+0000 7fdfc3a6dd80 -1 Falling back to public interface 2020-11-16T20:17:05.010 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/53056,v1:127.0.0.1:6803/53056] [v2:127.0.0.1:6804/53056,v1:127.0.0.1:6805/53056] exists,up 8e23841d-98e5-472c-91fe-6a992d391d13 2020-11-16T20:17:05.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:17:05.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:17:05.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:17:05.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:17:05.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2020-11-16T20:17:05.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:17:05.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:17:05.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:17:05.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:17:05.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:17:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:17:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2020-11-16T20:17:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-11-16T20:17:05.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T20:17:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=2 2020-11-16T20:17:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T20:17:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2020-11-16T20:17:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2020-11-16T20:17:05.814 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:05.809+0000 7fdfc3a6dd80 -1 osd.0 41 log_to_monitors {default=true} 2020-11-16T20:17:06.398 INFO:tasks.workunit.client.0.smithi189.stderr:PG '1.0' not found 2020-11-16T20:17:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: return 1 2020-11-16T20:17:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-11-16T20:17:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:17:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:17:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-11-16T20:17:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:17:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2020-11-16T20:17:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:17:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:17:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:17:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:17:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-11-16T20:17:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-11-16T20:17:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:17:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:17:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:17:06.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:17:06.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:17:06.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:17:06.709 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:17:06.709 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:17:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:17:06.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:17:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:17:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:17:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:17:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:17:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:17:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:17:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:17:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:17:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:17:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2020-11-16T20:17:06.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-11-16T20:17:06.712 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:17:06.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:17:06.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2020-11-16T20:17:06.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-11-16T20:17:06.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:17:06.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:17:06.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:17:06.747 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:06.741+0000 7f189ad11d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:17:06.750 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:06.745+0000 7f189ad11d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:17:06.752 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:06.745+0000 7f189ad11d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:17:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-11-16T20:17:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:17:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:17:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:17:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:17:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:17:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:17:07.120 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:17:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:17:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:17:07.301 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:07.297+0000 7f189ad11d80 -1 Falling back to public interface 2020-11-16T20:17:07.566 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/53455,v1:127.0.0.1:6811/53455] [v2:127.0.0.1:6812/53455,v1:127.0.0.1:6813/53455] exists,up 41ef07c7-ba20-4c7c-b7b2-5bc264c4a07a 2020-11-16T20:17:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:17:07.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:17:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:17:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2020-11-16T20:17:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-11-16T20:17:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T20:17:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-11-16T20:17:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T20:17:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-11-16T20:17:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2020-11-16T20:17:08.764 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:08.757+0000 7f189ad11d80 -1 osd.2 41 log_to_monitors {default=true} 2020-11-16T20:17:08.997 INFO:tasks.workunit.client.0.smithi189.stderr:PG '1.0' not found 2020-11-16T20:17:09.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: return 1 2020-11-16T20:17:09.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-11-16T20:17:09.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2020-11-16T20:17:09.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:17:09.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:17:09.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:17:09.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:17:09.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2020-11-16T20:17:09.294 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:17:09.295 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:17:09.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:17:09.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:17:09.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:17:09.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:17:09.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:17:09.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:17:09.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:17:09.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:17:09.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:17:09.875 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:17:09.875 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:17:09.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:17:09.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:17:09.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:17:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=184683593731 2020-11-16T20:17:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 184683593731 2020-11-16T20:17:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-184683593731' 2020-11-16T20:17:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:17:10.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:17:10.172 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-11-16T20:17:10.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T20:17:10.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T20:17:10.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T20:17:10.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:17:10.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:17:10.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495619 2020-11-16T20:17:10.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495619 2020-11-16T20:17:10.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-184683593731 2-197568495619' 2020-11-16T20:17:10.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:17:10.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-184683593731 2020-11-16T20:17:10.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:17:10.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:17:10.361 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-184683593731 2020-11-16T20:17:10.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:17:10.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=184683593731 2020-11-16T20:17:10.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 184683593731' 2020-11-16T20:17:10.364 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 184683593731 2020-11-16T20:17:10.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:17:10.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 184683593731 -lt 184683593731 2020-11-16T20:17:10.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:17:10.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-197568495619 2020-11-16T20:17:10.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:17:10.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:17:10.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-197568495619 2020-11-16T20:17:10.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:17:10.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495619 2020-11-16T20:17:10.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2020-11-16T20:17:10.763 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 197568495619 2020-11-16T20:17:10.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:17:11.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 197568495619 2020-11-16T20:17:11.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:17:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:17:12.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:17:12.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495619 -lt 197568495619 2020-11-16T20:17:12.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:17:12.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:17:12.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:17:13.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:17:13.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:17:13.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:17:13.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:17:13.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:17:13.098 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:17:13.098 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:17:13.098 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:17:13.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:17:13.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:17:13.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:17:13.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:17:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:17:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:17:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:17:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:17:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:17:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:17:14.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:17:14.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:17:14.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:17:14.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:17:14.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:17:14.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:17:14.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:17:14.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:17:14.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:17:14.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:17:14.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:17:14.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:17:14.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:17:14.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:17:14.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:17:14.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:17:14.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:17:14.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:17:14.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:17:14.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:17:14.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:17:15.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:17:15.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:17:15.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2020-11-16T20:17:15.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:17:15.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-11-16T20:17:15.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:17:15.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:17:15.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:17:15.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:17:15.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:17:15.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:17:15.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:17:15.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:17:15.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:17:15.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:17:15.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:17:15.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:17:16.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:17:16.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:17:16.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:17:16.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:17:16.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:17:16.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:17:16.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:17:16.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:17:16.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:17:16.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:17:16.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:17:16.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2020-11-16T20:17:16.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:17:16.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2020-11-16T20:17:17.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:17:17.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:17:17.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:17:17.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:17:17.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:17:17.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:17:17.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:17:17.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:17:17.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:17:17.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:17:17.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:17:17.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:17:18.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:17:18.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:17:18.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:17:18.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:17:18.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:17:18.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:17:18.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:17:18.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:17:18.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:17:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:17:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:17:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2020-11-16T20:17:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:17:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2020-11-16T20:17:19.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:17:19.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:17:19.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:17:19.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:17:19.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:17:19.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:17:19.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:17:19.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:17:19.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:17:19.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:17:19.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:17:19.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:17:20.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:17:20.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:17:20.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:17:20.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:17:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:17:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:17:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:17:20.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:17:20.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:17:20.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:17:20.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:17:20.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2020-11-16T20:17:20.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:17:20.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2020-11-16T20:17:22.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:17:22.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:17:22.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:17:22.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:17:22.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:17:22.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:17:22.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:17:22.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:17:22.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:17:22.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:17:22.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:17:22.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:17:23.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:17:23.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:17:23.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:17:23.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:17:23.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:17:23.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:17:23.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:17:23.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:17:23.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:17:23.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:17:23.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:17:23.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2020-11-16T20:17:23.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:17:23.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2020-11-16T20:17:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:17:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:17:27.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:17:27.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:17:27.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:17:27.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:17:27.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:17:27.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:17:27.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:17:27.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:17:27.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:17:27.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:17:27.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:17:27.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:17:27.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:17:27.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:17:27.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:17:27.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:17:27.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:17:27.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:17:27.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:17:28.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:17:28.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:17:28.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2020-11-16T20:17:28.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:17:28.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2020-11-16T20:17:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:17:34.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:17:34.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:17:34.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:17:34.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:17:34.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:17:34.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:17:34.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:17:35.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:17:35.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:17:35.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:17:35.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:17:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:17:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:17:35.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:17:35.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:17:35.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:17:35.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:17:35.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:17:35.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:17:35.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:17:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:17:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:17:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2020-11-16T20:17:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:17:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2020-11-16T20:17:43.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:17:43.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:17:43.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:17:43.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:17:43.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:17:43.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:17:43.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:17:43.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:17:43.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:17:43.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:17:43.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:17:43.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:17:44.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:17:44.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:17:44.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:17:44.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:17:44.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:17:44.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:17:44.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:17:44.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:17:44.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:17:44.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T20:17:44.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T20:17:44.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2020-11-16T20:17:44.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2020-11-16T20:17:45.325 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:17:45.325 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_fingerprint": "f0ffaefd-8deb-4050-8593-5c7677a2934c", 2020-11-16T20:17:45.325 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "16.0.0-7219-g911cc357", 2020-11-16T20:17:45.325 INFO:tasks.workunit.client.0.smithi189.stdout: "commit": "911cc35794d871071a8bebfb0919482263b1d18b", 2020-11-16T20:17:45.326 INFO:tasks.workunit.client.0.smithi189.stdout: "timestamp": "2020-11-16T20:17:45.321815+0000", 2020-11-16T20:17:45.326 INFO:tasks.workunit.client.0.smithi189.stdout: "tag": "", 2020-11-16T20:17:45.326 INFO:tasks.workunit.client.0.smithi189.stdout: "health": { 2020-11-16T20:17:45.326 INFO:tasks.workunit.client.0.smithi189.stdout: "status": "HEALTH_WARN", 2020-11-16T20:17:45.326 INFO:tasks.workunit.client.0.smithi189.stdout: "checks": { 2020-11-16T20:17:45.326 INFO:tasks.workunit.client.0.smithi189.stdout: "OSDMAP_FLAGS": { 2020-11-16T20:17:45.326 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:17:45.327 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:17:45.327 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "nodown,noout,noin flag(s) set", 2020-11-16T20:17:45.327 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 17 2020-11-16T20:17:45.327 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.327 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [], 2020-11-16T20:17:45.327 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:17:45.327 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.328 INFO:tasks.workunit.client.0.smithi189.stdout: "OSD_DOWN": { 2020-11-16T20:17:45.328 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:17:45.328 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:17:45.328 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 osds down", 2020-11-16T20:17:45.328 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:17:45.328 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.328 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:17:45.329 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.329 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "osd.1 (root=default,host=smithi189) is down" 2020-11-16T20:17:45.329 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.329 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.329 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:17:45.329 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.329 INFO:tasks.workunit.client.0.smithi189.stdout: "PG_DEGRADED": { 2020-11-16T20:17:45.330 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:17:45.330 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:17:45.330 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-11-16T20:17:45.330 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:17:45.330 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.330 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:17:45.330 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.330 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-11-16T20:17:45.331 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.331 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.331 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:17:45.331 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.331 INFO:tasks.workunit.client.0.smithi189.stdout: "POOL_APP_NOT_ENABLED": { 2020-11-16T20:17:45.331 INFO:tasks.workunit.client.0.smithi189.stdout: "severity": "HEALTH_WARN", 2020-11-16T20:17:45.331 INFO:tasks.workunit.client.0.smithi189.stdout: "summary": { 2020-11-16T20:17:45.332 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "1 pool(s) do not have an application enabled", 2020-11-16T20:17:45.332 INFO:tasks.workunit.client.0.smithi189.stdout: "count": 1 2020-11-16T20:17:45.332 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.332 INFO:tasks.workunit.client.0.smithi189.stdout: "detail": [ 2020-11-16T20:17:45.332 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.332 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "application not enabled on pool 'test'" 2020-11-16T20:17:45.332 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.333 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.333 INFO:tasks.workunit.client.0.smithi189.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-11-16T20:17:45.333 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.333 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.333 INFO:tasks.workunit.client.0.smithi189.stdout: "muted": false 2020-11-16T20:17:45.333 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.333 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.333 INFO:tasks.workunit.client.0.smithi189.stdout: "mutes": [] 2020-11-16T20:17:45.334 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.334 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_first_committed": 1, 2020-11-16T20:17:45.334 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap_last_committed": 1, 2020-11-16T20:17:45.334 INFO:tasks.workunit.client.0.smithi189.stdout: "monmap": { 2020-11-16T20:17:45.334 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:17:45.334 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:17:45.334 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:14:49.332386Z", 2020-11-16T20:17:45.335 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:14:49.332386Z", 2020-11-16T20:17:45.335 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release": 16, 2020-11-16T20:17:45.335 INFO:tasks.workunit.client.0.smithi189.stdout: "min_mon_release_name": "pacific", 2020-11-16T20:17:45.335 INFO:tasks.workunit.client.0.smithi189.stdout: "election_strategy": 1, 2020-11-16T20:17:45.335 INFO:tasks.workunit.client.0.smithi189.stdout: "disallowed_leaders: ": "", 2020-11-16T20:17:45.335 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": false, 2020-11-16T20:17:45.335 INFO:tasks.workunit.client.0.smithi189.stdout: "features": { 2020-11-16T20:17:45.336 INFO:tasks.workunit.client.0.smithi189.stdout: "persistent": [ 2020-11-16T20:17:45.336 INFO:tasks.workunit.client.0.smithi189.stdout: "kraken", 2020-11-16T20:17:45.336 INFO:tasks.workunit.client.0.smithi189.stdout: "luminous", 2020-11-16T20:17:45.336 INFO:tasks.workunit.client.0.smithi189.stdout: "mimic", 2020-11-16T20:17:45.336 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap-prune", 2020-11-16T20:17:45.336 INFO:tasks.workunit.client.0.smithi189.stdout: "nautilus", 2020-11-16T20:17:45.336 INFO:tasks.workunit.client.0.smithi189.stdout: "octopus", 2020-11-16T20:17:45.336 INFO:tasks.workunit.client.0.smithi189.stdout: "pacific", 2020-11-16T20:17:45.337 INFO:tasks.workunit.client.0.smithi189.stdout: "elector-pinging" 2020-11-16T20:17:45.337 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.337 INFO:tasks.workunit.client.0.smithi189.stdout: "optional": [] 2020-11-16T20:17:45.337 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.337 INFO:tasks.workunit.client.0.smithi189.stdout: "mons": [ 2020-11-16T20:17:45.337 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.337 INFO:tasks.workunit.client.0.smithi189.stdout: "rank": 0, 2020-11-16T20:17:45.338 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "a", 2020-11-16T20:17:45.338 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:17:45.338 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.338 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.338 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.338 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115", 2020-11-16T20:17:45.338 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 0 2020-11-16T20:17:45.338 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.339 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.339 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.339 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:7115/0", 2020-11-16T20:17:45.339 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:7115/0", 2020-11-16T20:17:45.339 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 0, 2020-11-16T20:17:45.339 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 0, 2020-11-16T20:17:45.339 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_location": "{}" 2020-11-16T20:17:45.340 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.340 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.340 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.340 INFO:tasks.workunit.client.0.smithi189.stdout: "quorum": [ 2020-11-16T20:17:45.340 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2020-11-16T20:17:45.340 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.340 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap": { 2020-11-16T20:17:45.340 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 47, 2020-11-16T20:17:45.341 INFO:tasks.workunit.client.0.smithi189.stdout: "fsid": "37a1fb3b-5cce-4fc2-9886-c734e76404a1", 2020-11-16T20:17:45.341 INFO:tasks.workunit.client.0.smithi189.stdout: "created": "2020-11-16T20:14:50.055857+0000", 2020-11-16T20:17:45.341 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:17:09.415173+0000", 2020-11-16T20:17:45.341 INFO:tasks.workunit.client.0.smithi189.stdout: "last_up_change": "2020-11-16T20:17:09.122430+0000", 2020-11-16T20:17:45.341 INFO:tasks.workunit.client.0.smithi189.stdout: "last_in_change": "2020-11-16T20:16:20.781603+0000", 2020-11-16T20:17:45.341 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-11-16T20:17:45.341 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_num": 5800384, 2020-11-16T20:17:45.342 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_set": [ 2020-11-16T20:17:45.342 INFO:tasks.workunit.client.0.smithi189.stdout: "nodown", 2020-11-16T20:17:45.342 INFO:tasks.workunit.client.0.smithi189.stdout: "noin", 2020-11-16T20:17:45.342 INFO:tasks.workunit.client.0.smithi189.stdout: "noout", 2020-11-16T20:17:45.342 INFO:tasks.workunit.client.0.smithi189.stdout: "pglog_hardlimit", 2020-11-16T20:17:45.342 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snapdirs", 2020-11-16T20:17:45.342 INFO:tasks.workunit.client.0.smithi189.stdout: "recovery_deletes", 2020-11-16T20:17:45.342 INFO:tasks.workunit.client.0.smithi189.stdout: "sortbitwise" 2020-11-16T20:17:45.343 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.343 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_version": 8, 2020-11-16T20:17:45.343 INFO:tasks.workunit.client.0.smithi189.stdout: "full_ratio": 0.99000000953674316, 2020-11-16T20:17:45.343 INFO:tasks.workunit.client.0.smithi189.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-11-16T20:17:45.343 INFO:tasks.workunit.client.0.smithi189.stdout: "nearfull_ratio": 0.85000002384185791, 2020-11-16T20:17:45.343 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_snapshot": "", 2020-11-16T20:17:45.343 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_max": 1, 2020-11-16T20:17:45.344 INFO:tasks.workunit.client.0.smithi189.stdout: "max_osd": 3, 2020-11-16T20:17:45.344 INFO:tasks.workunit.client.0.smithi189.stdout: "require_min_compat_client": "luminous", 2020-11-16T20:17:45.344 INFO:tasks.workunit.client.0.smithi189.stdout: "min_compat_client": "jewel", 2020-11-16T20:17:45.344 INFO:tasks.workunit.client.0.smithi189.stdout: "require_osd_release": "pacific", 2020-11-16T20:17:45.344 INFO:tasks.workunit.client.0.smithi189.stdout: "pools": [ 2020-11-16T20:17:45.344 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.344 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:17:45.344 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_name": "test", 2020-11-16T20:17:45.345 INFO:tasks.workunit.client.0.smithi189.stdout: "create_time": "2020-11-16T20:15:17.753606+0000", 2020-11-16T20:17:45.345 INFO:tasks.workunit.client.0.smithi189.stdout: "flags": 16389, 2020-11-16T20:17:45.345 INFO:tasks.workunit.client.0.smithi189.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-11-16T20:17:45.345 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 3, 2020-11-16T20:17:45.345 INFO:tasks.workunit.client.0.smithi189.stdout: "size": 3, 2020-11-16T20:17:45.345 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 2, 2020-11-16T20:17:45.345 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_rule": 1, 2020-11-16T20:17:45.346 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_count": 0, 2020-11-16T20:17:45.346 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_target": 0, 2020-11-16T20:17:45.346 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_barrier": 0, 2020-11-16T20:17:45.346 INFO:tasks.workunit.client.0.smithi189.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2020-11-16T20:17:45.346 INFO:tasks.workunit.client.0.smithi189.stdout: "object_hash": 2, 2020-11-16T20:17:45.346 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_autoscale_mode": "off", 2020-11-16T20:17:45.346 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num": 1, 2020-11-16T20:17:45.347 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num": 1, 2020-11-16T20:17:45.347 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_placement_num_target": 1, 2020-11-16T20:17:45.347 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_target": 1, 2020-11-16T20:17:45.347 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_num_pending": 1, 2020-11-16T20:17:45.347 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pg_merge_meta": { 2020-11-16T20:17:45.347 INFO:tasks.workunit.client.0.smithi189.stdout: "source_pgid": "0.0", 2020-11-16T20:17:45.347 INFO:tasks.workunit.client.0.smithi189.stdout: "ready_epoch": 0, 2020-11-16T20:17:45.347 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_started": 0, 2020-11-16T20:17:45.348 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": 0, 2020-11-16T20:17:45.348 INFO:tasks.workunit.client.0.smithi189.stdout: "source_version": "0'0", 2020-11-16T20:17:45.348 INFO:tasks.workunit.client.0.smithi189.stdout: "target_version": "0'0" 2020-11-16T20:17:45.348 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.348 INFO:tasks.workunit.client.0.smithi189.stdout: "last_change": "30", 2020-11-16T20:17:45.348 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend": "0", 2020-11-16T20:17:45.348 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_prenautilus": "0", 2020-11-16T20:17:45.348 INFO:tasks.workunit.client.0.smithi189.stdout: "last_force_op_resend_preluminous": "0", 2020-11-16T20:17:45.349 INFO:tasks.workunit.client.0.smithi189.stdout: "auid": 0, 2020-11-16T20:17:45.349 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_mode": "pool", 2020-11-16T20:17:45.349 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_seq": 1, 2020-11-16T20:17:45.349 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_epoch": 30, 2020-11-16T20:17:45.349 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_snaps": [ 2020-11-16T20:17:45.349 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.349 INFO:tasks.workunit.client.0.smithi189.stdout: "snapid": 1, 2020-11-16T20:17:45.350 INFO:tasks.workunit.client.0.smithi189.stdout: "stamp": "2020-11-16T20:16:01.228891+0000", 2020-11-16T20:17:45.350 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "snap1" 2020-11-16T20:17:45.350 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.350 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.350 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps": "[]", 2020-11-16T20:17:45.350 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_bytes": 0, 2020-11-16T20:17:45.350 INFO:tasks.workunit.client.0.smithi189.stdout: "quota_max_objects": 0, 2020-11-16T20:17:45.351 INFO:tasks.workunit.client.0.smithi189.stdout: "tiers": [], 2020-11-16T20:17:45.351 INFO:tasks.workunit.client.0.smithi189.stdout: "tier_of": -1, 2020-11-16T20:17:45.351 INFO:tasks.workunit.client.0.smithi189.stdout: "read_tier": -1, 2020-11-16T20:17:45.351 INFO:tasks.workunit.client.0.smithi189.stdout: "write_tier": -1, 2020-11-16T20:17:45.351 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_mode": "none", 2020-11-16T20:17:45.351 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_bytes": 0, 2020-11-16T20:17:45.351 INFO:tasks.workunit.client.0.smithi189.stdout: "target_max_objects": 0, 2020-11-16T20:17:45.351 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-11-16T20:17:45.352 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-11-16T20:17:45.352 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_target_full_ratio_micro": 800000, 2020-11-16T20:17:45.352 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_flush_age": 0, 2020-11-16T20:17:45.352 INFO:tasks.workunit.client.0.smithi189.stdout: "cache_min_evict_age": 0, 2020-11-16T20:17:45.352 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profile": "myprofile", 2020-11-16T20:17:45.352 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_params": { 2020-11-16T20:17:45.352 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "none" 2020-11-16T20:17:45.352 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.353 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_period": 0, 2020-11-16T20:17:45.353 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_count": 0, 2020-11-16T20:17:45.353 INFO:tasks.workunit.client.0.smithi189.stdout: "use_gmt_hitset": true, 2020-11-16T20:17:45.353 INFO:tasks.workunit.client.0.smithi189.stdout: "min_read_recency_for_promote": 0, 2020-11-16T20:17:45.353 INFO:tasks.workunit.client.0.smithi189.stdout: "min_write_recency_for_promote": 0, 2020-11-16T20:17:45.353 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_grade_decay_rate": 0, 2020-11-16T20:17:45.353 INFO:tasks.workunit.client.0.smithi189.stdout: "hit_set_search_last_n": 0, 2020-11-16T20:17:45.354 INFO:tasks.workunit.client.0.smithi189.stdout: "grade_table": [], 2020-11-16T20:17:45.354 INFO:tasks.workunit.client.0.smithi189.stdout: "stripe_width": 8192, 2020-11-16T20:17:45.354 INFO:tasks.workunit.client.0.smithi189.stdout: "expected_num_objects": 0, 2020-11-16T20:17:45.354 INFO:tasks.workunit.client.0.smithi189.stdout: "fast_read": false, 2020-11-16T20:17:45.354 INFO:tasks.workunit.client.0.smithi189.stdout: "options": {}, 2020-11-16T20:17:45.354 INFO:tasks.workunit.client.0.smithi189.stdout: "application_metadata": {} 2020-11-16T20:17:45.354 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.354 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.355 INFO:tasks.workunit.client.0.smithi189.stdout: "osds": [ 2020-11-16T20:17:45.355 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.355 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:17:45.355 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "8e23841d-98e5-472c-91fe-6a992d391d13", 2020-11-16T20:17:45.355 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:17:45.355 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:17:45.355 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:17:45.356 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:17:45.356 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 36, 2020-11-16T20:17:45.356 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 41, 2020-11-16T20:17:45.356 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 43, 2020-11-16T20:17:45.356 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 46, 2020-11-16T20:17:45.356 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 42, 2020-11-16T20:17:45.356 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:17:45.356 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:17:45.357 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.357 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.357 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.357 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6802", 2020-11-16T20:17:45.357 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55203 2020-11-16T20:17:45.357 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.357 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.358 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.358 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6803", 2020-11-16T20:17:45.358 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55203 2020-11-16T20:17:45.358 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.358 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.358 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.358 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:17:45.358 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.359 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.359 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.359 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6804", 2020-11-16T20:17:45.359 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55203 2020-11-16T20:17:45.359 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.359 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.359 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.360 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6805", 2020-11-16T20:17:45.360 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55203 2020-11-16T20:17:45.360 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.360 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.360 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.360 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:17:45.360 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.360 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.361 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.361 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6808", 2020-11-16T20:17:45.361 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55203 2020-11-16T20:17:45.361 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.361 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.361 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.361 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6809", 2020-11-16T20:17:45.362 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55203 2020-11-16T20:17:45.362 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.362 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.362 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.362 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:17:45.362 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.362 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.362 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.363 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6806", 2020-11-16T20:17:45.363 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55203 2020-11-16T20:17:45.363 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.363 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.363 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.363 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6807", 2020-11-16T20:17:45.363 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55203 2020-11-16T20:17:45.363 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.364 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.364 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.364 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6803/55203", 2020-11-16T20:17:45.364 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6805/55203", 2020-11-16T20:17:45.364 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6809/55203", 2020-11-16T20:17:45.364 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6807/55203", 2020-11-16T20:17:45.364 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:17:45.365 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:17:45.365 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:17:45.365 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.365 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.365 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.365 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:17:45.365 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "fada656c-964b-4065-ada2-6f88fccdceb7", 2020-11-16T20:17:45.366 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 0, 2020-11-16T20:17:45.366 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:17:45.366 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:17:45.366 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:17:45.366 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 0, 2020-11-16T20:17:45.366 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 0, 2020-11-16T20:17:45.366 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 10, 2020-11-16T20:17:45.367 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 25, 2020-11-16T20:17:45.367 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 32, 2020-11-16T20:17:45.367 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:17:45.367 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:17:45.367 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.367 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.367 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.368 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:17:45.368 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:45.368 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.368 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.368 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.368 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:17:45.368 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:45.369 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.369 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.369 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.369 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:17:45.369 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.369 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.369 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.369 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:17:45.370 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:45.370 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.370 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.370 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.370 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:17:45.370 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:45.370 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.371 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.371 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.371 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:17:45.371 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.371 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.371 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.371 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:17:45.372 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:45.372 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.372 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.372 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.372 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:17:45.372 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:45.372 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.373 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.373 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.373 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:17:45.373 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.373 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.373 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.373 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:17:45.374 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:45.374 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.374 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.374 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.374 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:17:45.374 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 47811 2020-11-16T20:17:45.374 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.374 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.375 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.375 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/47811", 2020-11-16T20:17:45.375 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/47811", 2020-11-16T20:17:45.375 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/47811", 2020-11-16T20:17:45.375 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/47811", 2020-11-16T20:17:45.375 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:17:45.375 INFO:tasks.workunit.client.0.smithi189.stdout: "exists" 2020-11-16T20:17:45.376 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.376 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.376 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.376 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:17:45.376 INFO:tasks.workunit.client.0.smithi189.stdout: "uuid": "41ef07c7-ba20-4c7c-b7b2-5bc264c4a07a", 2020-11-16T20:17:45.376 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 1, 2020-11-16T20:17:45.376 INFO:tasks.workunit.client.0.smithi189.stdout: "in": 1, 2020-11-16T20:17:45.377 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 1, 2020-11-16T20:17:45.377 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_affinity": 1, 2020-11-16T20:17:45.377 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_begin": 37, 2020-11-16T20:17:45.377 INFO:tasks.workunit.client.0.smithi189.stdout: "last_clean_end": 41, 2020-11-16T20:17:45.377 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 46, 2020-11-16T20:17:45.377 INFO:tasks.workunit.client.0.smithi189.stdout: "up_thru": 0, 2020-11-16T20:17:45.377 INFO:tasks.workunit.client.0.smithi189.stdout: "down_at": 45, 2020-11-16T20:17:45.377 INFO:tasks.workunit.client.0.smithi189.stdout: "lost_at": 0, 2020-11-16T20:17:45.378 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addrs": { 2020-11-16T20:17:45.378 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.378 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.378 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.378 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6810", 2020-11-16T20:17:45.378 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55852 2020-11-16T20:17:45.378 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.379 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.379 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.379 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6811", 2020-11-16T20:17:45.379 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55852 2020-11-16T20:17:45.379 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.379 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.379 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.379 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addrs": { 2020-11-16T20:17:45.380 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.380 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.380 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.380 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6812", 2020-11-16T20:17:45.380 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55852 2020-11-16T20:17:45.380 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.380 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.381 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.381 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6813", 2020-11-16T20:17:45.381 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55852 2020-11-16T20:17:45.381 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.381 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.381 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.382 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addrs": { 2020-11-16T20:17:45.382 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.382 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.382 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.382 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6816", 2020-11-16T20:17:45.382 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55852 2020-11-16T20:17:45.382 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.382 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.383 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.383 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6817", 2020-11-16T20:17:45.383 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55852 2020-11-16T20:17:45.383 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.383 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.383 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.383 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addrs": { 2020-11-16T20:17:45.384 INFO:tasks.workunit.client.0.smithi189.stdout: "addrvec": [ 2020-11-16T20:17:45.384 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.384 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v2", 2020-11-16T20:17:45.384 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6814", 2020-11-16T20:17:45.384 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55852 2020-11-16T20:17:45.384 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.384 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.384 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "v1", 2020-11-16T20:17:45.385 INFO:tasks.workunit.client.0.smithi189.stdout: "addr": "127.0.0.1:6815", 2020-11-16T20:17:45.385 INFO:tasks.workunit.client.0.smithi189.stdout: "nonce": 55852 2020-11-16T20:17:45.385 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.385 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.385 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.385 INFO:tasks.workunit.client.0.smithi189.stdout: "public_addr": "127.0.0.1:6811/55852", 2020-11-16T20:17:45.385 INFO:tasks.workunit.client.0.smithi189.stdout: "cluster_addr": "127.0.0.1:6813/55852", 2020-11-16T20:17:45.385 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_back_addr": "127.0.0.1:6817/55852", 2020-11-16T20:17:45.386 INFO:tasks.workunit.client.0.smithi189.stdout: "heartbeat_front_addr": "127.0.0.1:6815/55852", 2020-11-16T20:17:45.386 INFO:tasks.workunit.client.0.smithi189.stdout: "state": [ 2020-11-16T20:17:45.386 INFO:tasks.workunit.client.0.smithi189.stdout: "exists", 2020-11-16T20:17:45.386 INFO:tasks.workunit.client.0.smithi189.stdout: "up" 2020-11-16T20:17:45.386 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.386 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.386 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.387 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_xinfo": [ 2020-11-16T20:17:45.387 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.387 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:17:45.387 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:17:05.896529+0000", 2020-11-16T20:17:45.387 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:17:45.387 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:17:45.387 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:17:45.387 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:17:45.388 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:14:58.521932+0000", 2020-11-16T20:17:45.388 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:17:45.388 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.388 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.388 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:17:45.388 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:16:12.931373+0000", 2020-11-16T20:17:45.388 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:17:45.389 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:17:45.389 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:17:45.389 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:17:45.389 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:15:06.606321+0000", 2020-11-16T20:17:45.389 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:17:45.389 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.389 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.389 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:17:45.390 INFO:tasks.workunit.client.0.smithi189.stdout: "down_stamp": "2020-11-16T20:17:08.905179+0000", 2020-11-16T20:17:45.390 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_probability": 0, 2020-11-16T20:17:45.390 INFO:tasks.workunit.client.0.smithi189.stdout: "laggy_interval": 0, 2020-11-16T20:17:45.390 INFO:tasks.workunit.client.0.smithi189.stdout: "features": 4540138297136906239, 2020-11-16T20:17:45.390 INFO:tasks.workunit.client.0.smithi189.stdout: "old_weight": 0, 2020-11-16T20:17:45.390 INFO:tasks.workunit.client.0.smithi189.stdout: "last_purged_snaps_scrub": "2020-11-16T20:15:14.477925+0000", 2020-11-16T20:17:45.390 INFO:tasks.workunit.client.0.smithi189.stdout: "dead_epoch": 0 2020-11-16T20:17:45.391 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.391 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.391 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap": [], 2020-11-16T20:17:45.391 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_upmap_items": [], 2020-11-16T20:17:45.391 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_temp": [], 2020-11-16T20:17:45.391 INFO:tasks.workunit.client.0.smithi189.stdout: "primary_temp": [], 2020-11-16T20:17:45.391 INFO:tasks.workunit.client.0.smithi189.stdout: "blocklist": {}, 2020-11-16T20:17:45.391 INFO:tasks.workunit.client.0.smithi189.stdout: "erasure_code_profiles": { 2020-11-16T20:17:45.392 INFO:tasks.workunit.client.0.smithi189.stdout: "default": { 2020-11-16T20:17:45.392 INFO:tasks.workunit.client.0.smithi189.stdout: "k": "2", 2020-11-16T20:17:45.392 INFO:tasks.workunit.client.0.smithi189.stdout: "m": "2", 2020-11-16T20:17:45.392 INFO:tasks.workunit.client.0.smithi189.stdout: "plugin": "jerasure", 2020-11-16T20:17:45.392 INFO:tasks.workunit.client.0.smithi189.stdout: "technique": "reed_sol_van" 2020-11-16T20:17:45.392 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.392 INFO:tasks.workunit.client.0.smithi189.stdout: "myprofile": { 2020-11-16T20:17:45.393 INFO:tasks.workunit.client.0.smithi189.stdout: "crush-device-class": "", 2020-11-16T20:17:45.393 INFO:tasks.workunit.client.0.smithi189.stdout: "crush-failure-domain": "osd", 2020-11-16T20:17:45.393 INFO:tasks.workunit.client.0.smithi189.stdout: "crush-root": "default", 2020-11-16T20:17:45.393 INFO:tasks.workunit.client.0.smithi189.stdout: "jerasure-per-chunk-alignment": "false", 2020-11-16T20:17:45.393 INFO:tasks.workunit.client.0.smithi189.stdout: "k": "2", 2020-11-16T20:17:45.393 INFO:tasks.workunit.client.0.smithi189.stdout: "m": "1", 2020-11-16T20:17:45.393 INFO:tasks.workunit.client.0.smithi189.stdout: "plugin": "jerasure", 2020-11-16T20:17:45.394 INFO:tasks.workunit.client.0.smithi189.stdout: "technique": "reed_sol_van", 2020-11-16T20:17:45.394 INFO:tasks.workunit.client.0.smithi189.stdout: "w": "8" 2020-11-16T20:17:45.394 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.394 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.394 INFO:tasks.workunit.client.0.smithi189.stdout: "removed_snaps_queue": [], 2020-11-16T20:17:45.394 INFO:tasks.workunit.client.0.smithi189.stdout: "new_removed_snaps": [], 2020-11-16T20:17:45.394 INFO:tasks.workunit.client.0.smithi189.stdout: "new_purged_snaps": [], 2020-11-16T20:17:45.394 INFO:tasks.workunit.client.0.smithi189.stdout: "crush_node_flags": {}, 2020-11-16T20:17:45.395 INFO:tasks.workunit.client.0.smithi189.stdout: "device_class_flags": {}, 2020-11-16T20:17:45.395 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode": { 2020-11-16T20:17:45.395 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_enabled": false, 2020-11-16T20:17:45.395 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_bucket_count": 0, 2020-11-16T20:17:45.395 INFO:tasks.workunit.client.0.smithi189.stdout: "degraded_stretch_mode": 0, 2020-11-16T20:17:45.395 INFO:tasks.workunit.client.0.smithi189.stdout: "recovering_stretch_mode": 0, 2020-11-16T20:17:45.395 INFO:tasks.workunit.client.0.smithi189.stdout: "stretch_mode_bucket": 0 2020-11-16T20:17:45.396 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.396 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.396 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_metadata": [ 2020-11-16T20:17:45.396 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.396 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:17:45.396 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:17:45.396 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6804/55203,v1:127.0.0.1:6805/55203]", 2020-11-16T20:17:45.396 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:17:45.397 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:17:45.397 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:17:45.397 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:17:45.397 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:17:45.397 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:17:45.397 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:17:45.397 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:17:45.398 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:17:45.398 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-11-16T20:17:45.398 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:17:45.398 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:17:45.398 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:17:45.398 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:17:45.398 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:17:45.399 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:17:45.399 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:17:45.399 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:17:45.399 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:17:45.399 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:17:45.399 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:17:45.399 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:17:45.400 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:17:45.400 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:17:45.400 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:17:45.400 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6802/55203,v1:127.0.0.1:6803/55203]", 2020-11-16T20:17:45.400 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:17:45.400 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/55203,v1:127.0.0.1:6809/55203]", 2020-11-16T20:17:45.400 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/55203,v1:127.0.0.1:6807/55203]", 2020-11-16T20:17:45.401 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:17:45.401 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:17:45.401 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:17:45.401 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:17:45.401 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:17:45.401 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:17:45.401 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:17:45.401 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:17:45.402 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:17:45.402 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/0", 2020-11-16T20:17:45.402 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:17:45.402 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:17:45.402 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:17:45.402 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.402 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.403 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:17:45.403 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:17:45.403 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/47811,v1:127.0.0.1:6813/47811]", 2020-11-16T20:17:45.403 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:17:45.403 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:17:45.403 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:17:45.403 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:17:45.403 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:17:45.404 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:17:45.404 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:17:45.404 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:17:45.404 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:17:45.404 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-11-16T20:17:45.404 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:17:45.404 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:17:45.405 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:17:45.405 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:17:45.405 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:17:45.405 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:17:45.405 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:17:45.405 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:17:45.405 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:17:45.405 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:17:45.406 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:17:45.406 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:17:45.406 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:17:45.406 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:17:45.406 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:17:45.406 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/47811,v1:127.0.0.1:6811/47811]", 2020-11-16T20:17:45.406 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:17:45.407 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/47811,v1:127.0.0.1:6817/47811]", 2020-11-16T20:17:45.407 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/47811,v1:127.0.0.1:6815/47811]", 2020-11-16T20:17:45.407 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:17:45.407 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:17:45.407 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:17:45.407 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:17:45.407 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:17:45.408 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:17:45.408 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:17:45.408 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:17:45.408 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:17:45.408 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/1", 2020-11-16T20:17:45.408 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:17:45.408 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:17:45.408 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:17:45.409 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.409 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.409 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:17:45.409 INFO:tasks.workunit.client.0.smithi189.stdout: "arch": "x86_64", 2020-11-16T20:17:45.409 INFO:tasks.workunit.client.0.smithi189.stdout: "back_addr": "[v2:127.0.0.1:6812/55852,v1:127.0.0.1:6813/55852]", 2020-11-16T20:17:45.409 INFO:tasks.workunit.client.0.smithi189.stdout: "back_iface": "", 2020-11-16T20:17:45.409 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs": "1", 2020-11-16T20:17:45.410 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_db": "0", 2020-11-16T20:17:45.410 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_dedicated_wal": "0", 2020-11-16T20:17:45.410 INFO:tasks.workunit.client.0.smithi189.stdout: "bluefs_single_shared_device": "1", 2020-11-16T20:17:45.410 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T20:17:45.410 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T20:17:45.410 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T20:17:45.410 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T20:17:45.410 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-11-16T20:17:45.411 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T20:17:45.411 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T20:17:45.411 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T20:17:45.411 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T20:17:45.411 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_release": "pacific", 2020-11-16T20:17:45.411 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7219-g911cc357 (911cc35794d871071a8bebfb0919482263b1d18b) pacific (dev)", 2020-11-16T20:17:45.411 INFO:tasks.workunit.client.0.smithi189.stdout: "ceph_version_short": "16.0.0-7219-g911cc357", 2020-11-16T20:17:45.412 INFO:tasks.workunit.client.0.smithi189.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-11-16T20:17:45.412 INFO:tasks.workunit.client.0.smithi189.stdout: "default_device_class": "hdd", 2020-11-16T20:17:45.412 INFO:tasks.workunit.client.0.smithi189.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XKLY", 2020-11-16T20:17:45.412 INFO:tasks.workunit.client.0.smithi189.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-11-16T20:17:45.412 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": "sda", 2020-11-16T20:17:45.412 INFO:tasks.workunit.client.0.smithi189.stdout: "distro": "ubuntu", 2020-11-16T20:17:45.412 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-11-16T20:17:45.412 INFO:tasks.workunit.client.0.smithi189.stdout: "distro_version": "18.04", 2020-11-16T20:17:45.413 INFO:tasks.workunit.client.0.smithi189.stdout: "front_addr": "[v2:127.0.0.1:6810/55852,v1:127.0.0.1:6811/55852]", 2020-11-16T20:17:45.413 INFO:tasks.workunit.client.0.smithi189.stdout: "front_iface": "", 2020-11-16T20:17:45.413 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/55852,v1:127.0.0.1:6817/55852]", 2020-11-16T20:17:45.413 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/55852,v1:127.0.0.1:6815/55852]", 2020-11-16T20:17:45.413 INFO:tasks.workunit.client.0.smithi189.stdout: "hostname": "smithi189", 2020-11-16T20:17:45.413 INFO:tasks.workunit.client.0.smithi189.stdout: "journal_rotational": "1", 2020-11-16T20:17:45.413 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_description": "#127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020", 2020-11-16T20:17:45.414 INFO:tasks.workunit.client.0.smithi189.stdout: "kernel_version": "4.15.0-124-generic", 2020-11-16T20:17:45.414 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_swap_kb": "2097148", 2020-11-16T20:17:45.414 INFO:tasks.workunit.client.0.smithi189.stdout: "mem_total_kb": "32824080", 2020-11-16T20:17:45.414 INFO:tasks.workunit.client.0.smithi189.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-11-16T20:17:45.414 INFO:tasks.workunit.client.0.smithi189.stdout: "objectstore_numa_unknown_devices": "sda", 2020-11-16T20:17:45.414 INFO:tasks.workunit.client.0.smithi189.stdout: "os": "Linux", 2020-11-16T20:17:45.414 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_data": "td/divergent-priors/2", 2020-11-16T20:17:45.415 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T20:17:45.415 INFO:tasks.workunit.client.0.smithi189.stdout: "osdspec_affinity": "", 2020-11-16T20:17:45.415 INFO:tasks.workunit.client.0.smithi189.stdout: "rotational": "1" 2020-11-16T20:17:45.415 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.415 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.415 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_clean_epochs": { 2020-11-16T20:17:45.415 INFO:tasks.workunit.client.0.smithi189.stdout: "min_last_epoch_clean": 0, 2020-11-16T20:17:45.415 INFO:tasks.workunit.client.0.smithi189.stdout: "last_epoch_clean": { 2020-11-16T20:17:45.416 INFO:tasks.workunit.client.0.smithi189.stdout: "per_pool": [] 2020-11-16T20:17:45.416 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.416 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_epochs": [ 2020-11-16T20:17:45.416 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.416 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:17:45.416 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 44 2020-11-16T20:17:45.416 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.417 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.417 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:17:45.417 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 10 2020-11-16T20:17:45.417 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.417 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.417 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:17:45.417 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 47 2020-11-16T20:17:45.417 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.418 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.418 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.418 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_first_committed": 1, 2020-11-16T20:17:45.418 INFO:tasks.workunit.client.0.smithi189.stdout: "osdmap_last_committed": 47, 2020-11-16T20:17:45.418 INFO:tasks.workunit.client.0.smithi189.stdout: "crushmap": { 2020-11-16T20:17:45.418 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": [ 2020-11-16T20:17:45.418 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.419 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:17:45.419 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.0", 2020-11-16T20:17:45.419 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:17:45.419 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.419 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.419 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:17:45.419 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.1", 2020-11-16T20:17:45.420 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:17:45.420 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.420 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.420 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:17:45.420 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.2", 2020-11-16T20:17:45.420 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "hdd" 2020-11-16T20:17:45.420 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.420 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.421 INFO:tasks.workunit.client.0.smithi189.stdout: "types": [ 2020-11-16T20:17:45.421 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.421 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 0, 2020-11-16T20:17:45.421 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd" 2020-11-16T20:17:45.421 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.421 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.421 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:17:45.422 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "host" 2020-11-16T20:17:45.422 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.422 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.422 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 2, 2020-11-16T20:17:45.422 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "chassis" 2020-11-16T20:17:45.422 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.422 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.422 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 3, 2020-11-16T20:17:45.423 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "rack" 2020-11-16T20:17:45.423 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.423 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.423 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 4, 2020-11-16T20:17:45.423 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "row" 2020-11-16T20:17:45.423 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.423 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.423 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 5, 2020-11-16T20:17:45.424 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pdu" 2020-11-16T20:17:45.424 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.424 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.424 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 6, 2020-11-16T20:17:45.424 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pod" 2020-11-16T20:17:45.424 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.424 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.425 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 7, 2020-11-16T20:17:45.425 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "room" 2020-11-16T20:17:45.425 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.425 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.425 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 8, 2020-11-16T20:17:45.425 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "datacenter" 2020-11-16T20:17:45.425 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.425 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.426 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 9, 2020-11-16T20:17:45.426 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "zone" 2020-11-16T20:17:45.426 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.426 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.426 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 10, 2020-11-16T20:17:45.426 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "region" 2020-11-16T20:17:45.426 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.427 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.427 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:17:45.427 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "root" 2020-11-16T20:17:45.427 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.427 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.427 INFO:tasks.workunit.client.0.smithi189.stdout: "buckets": [ 2020-11-16T20:17:45.427 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.427 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -1, 2020-11-16T20:17:45.428 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default", 2020-11-16T20:17:45.428 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:17:45.428 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:17:45.428 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:45.428 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:17:45.428 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:17:45.428 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:17:45.428 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.429 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:17:45.429 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:45.429 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:17:45.429 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.429 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.429 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.429 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.430 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -2, 2020-11-16T20:17:45.430 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default~hdd", 2020-11-16T20:17:45.430 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2020-11-16T20:17:45.430 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2020-11-16T20:17:45.430 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:45.430 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:17:45.430 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:17:45.430 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:17:45.431 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.431 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:17:45.431 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:45.431 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:17:45.431 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.431 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.431 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.431 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.432 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2020-11-16T20:17:45.432 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189", 2020-11-16T20:17:45.432 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:17:45.432 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:17:45.432 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:45.432 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:17:45.432 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:17:45.433 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:17:45.433 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.433 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:17:45.433 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:45.433 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:17:45.433 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.433 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.433 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:17:45.434 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:45.434 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:17:45.434 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.434 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.434 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:17:45.434 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:45.434 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:17:45.435 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.435 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.435 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.435 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.435 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2020-11-16T20:17:45.435 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189~hdd", 2020-11-16T20:17:45.435 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2020-11-16T20:17:45.435 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2020-11-16T20:17:45.436 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 19206, 2020-11-16T20:17:45.436 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2020-11-16T20:17:45.436 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2020-11-16T20:17:45.436 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:17:45.436 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.436 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2020-11-16T20:17:45.436 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:45.436 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2020-11-16T20:17:45.437 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.437 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.437 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2020-11-16T20:17:45.437 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:45.437 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2020-11-16T20:17:45.437 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.437 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.438 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2020-11-16T20:17:45.438 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 6402, 2020-11-16T20:17:45.438 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2020-11-16T20:17:45.438 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.438 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.438 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.438 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.438 INFO:tasks.workunit.client.0.smithi189.stdout: "rules": [ 2020-11-16T20:17:45.439 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.439 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 0, 2020-11-16T20:17:45.439 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "replicated_rule", 2020-11-16T20:17:45.439 INFO:tasks.workunit.client.0.smithi189.stdout: "ruleset": 0, 2020-11-16T20:17:45.439 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2020-11-16T20:17:45.439 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 1, 2020-11-16T20:17:45.439 INFO:tasks.workunit.client.0.smithi189.stdout: "max_size": 10, 2020-11-16T20:17:45.440 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2020-11-16T20:17:45.440 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.440 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2020-11-16T20:17:45.440 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2020-11-16T20:17:45.440 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2020-11-16T20:17:45.440 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.440 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.440 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_firstn", 2020-11-16T20:17:45.441 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2020-11-16T20:17:45.441 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2020-11-16T20:17:45.441 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.441 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.441 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2020-11-16T20:17:45.441 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.441 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.442 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.442 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.442 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 1, 2020-11-16T20:17:45.442 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "test", 2020-11-16T20:17:45.442 INFO:tasks.workunit.client.0.smithi189.stdout: "ruleset": 1, 2020-11-16T20:17:45.442 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 3, 2020-11-16T20:17:45.442 INFO:tasks.workunit.client.0.smithi189.stdout: "min_size": 3, 2020-11-16T20:17:45.442 INFO:tasks.workunit.client.0.smithi189.stdout: "max_size": 3, 2020-11-16T20:17:45.443 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2020-11-16T20:17:45.443 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.443 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_chooseleaf_tries", 2020-11-16T20:17:45.443 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 5 2020-11-16T20:17:45.443 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.443 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.443 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_choose_tries", 2020-11-16T20:17:45.443 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 100 2020-11-16T20:17:45.444 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.444 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.444 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2020-11-16T20:17:45.444 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2020-11-16T20:17:45.444 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2020-11-16T20:17:45.444 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.444 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.445 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_indep", 2020-11-16T20:17:45.445 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2020-11-16T20:17:45.445 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2020-11-16T20:17:45.445 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.445 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.445 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2020-11-16T20:17:45.445 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.445 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:17:45.446 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.446 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.446 INFO:tasks.workunit.client.0.smithi189.stdout: "tunables": { 2020-11-16T20:17:45.446 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_tries": 0, 2020-11-16T20:17:45.446 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_fallback_tries": 0, 2020-11-16T20:17:45.446 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_total_tries": 50, 2020-11-16T20:17:45.446 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_descend_once": 1, 2020-11-16T20:17:45.447 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_vary_r": 1, 2020-11-16T20:17:45.447 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_stable": 1, 2020-11-16T20:17:45.447 INFO:tasks.workunit.client.0.smithi189.stdout: "straw_calc_version": 1, 2020-11-16T20:17:45.447 INFO:tasks.workunit.client.0.smithi189.stdout: "allowed_bucket_algs": 54, 2020-11-16T20:17:45.447 INFO:tasks.workunit.client.0.smithi189.stdout: "profile": "jewel", 2020-11-16T20:17:45.447 INFO:tasks.workunit.client.0.smithi189.stdout: "optimal_tunables": 1, 2020-11-16T20:17:45.447 INFO:tasks.workunit.client.0.smithi189.stdout: "legacy_tunables": 0, 2020-11-16T20:17:45.447 INFO:tasks.workunit.client.0.smithi189.stdout: "minimum_required_version": "jewel", 2020-11-16T20:17:45.448 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables": 1, 2020-11-16T20:17:45.448 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables2": 1, 2020-11-16T20:17:45.448 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v2_rules": 1, 2020-11-16T20:17:45.448 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables3": 1, 2020-11-16T20:17:45.448 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v3_rules": 0, 2020-11-16T20:17:45.448 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v4_buckets": 1, 2020-11-16T20:17:45.448 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables5": 1, 2020-11-16T20:17:45.448 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v5_rules": 0 2020-11-16T20:17:45.449 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.449 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_args": {} 2020-11-16T20:17:45.449 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.449 INFO:tasks.workunit.client.0.smithi189.stdout: "fsmap": { 2020-11-16T20:17:45.449 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 1, 2020-11-16T20:17:45.449 INFO:tasks.workunit.client.0.smithi189.stdout: "default_fscid": -1, 2020-11-16T20:17:45.449 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": { 2020-11-16T20:17:45.450 INFO:tasks.workunit.client.0.smithi189.stdout: "compat": {}, 2020-11-16T20:17:45.450 INFO:tasks.workunit.client.0.smithi189.stdout: "ro_compat": {}, 2020-11-16T20:17:45.450 INFO:tasks.workunit.client.0.smithi189.stdout: "incompat": { 2020-11-16T20:17:45.450 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_1": "base v0.20", 2020-11-16T20:17:45.450 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_2": "client writeable ranges", 2020-11-16T20:17:45.450 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_3": "default file layouts on dirs", 2020-11-16T20:17:45.450 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_4": "dir inode in separate object", 2020-11-16T20:17:45.450 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_5": "mds uses versioned encoding", 2020-11-16T20:17:45.451 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_6": "dirfrag is stored in omap", 2020-11-16T20:17:45.451 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_8": "no anchor table", 2020-11-16T20:17:45.451 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_9": "file layout v2", 2020-11-16T20:17:45.451 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_10": "snaprealm v2" 2020-11-16T20:17:45.451 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.451 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.451 INFO:tasks.workunit.client.0.smithi189.stdout: "feature_flags": { 2020-11-16T20:17:45.452 INFO:tasks.workunit.client.0.smithi189.stdout: "enable_multiple": false, 2020-11-16T20:17:45.452 INFO:tasks.workunit.client.0.smithi189.stdout: "ever_enabled_multiple": false 2020-11-16T20:17:45.452 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.452 INFO:tasks.workunit.client.0.smithi189.stdout: "standbys": [], 2020-11-16T20:17:45.452 INFO:tasks.workunit.client.0.smithi189.stdout: "filesystems": [] 2020-11-16T20:17:45.452 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.452 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_first_committed": 1, 2020-11-16T20:17:45.452 INFO:tasks.workunit.client.0.smithi189.stdout: "mdsmap_last_committed": 1, 2020-11-16T20:17:45.453 INFO:tasks.workunit.client.0.smithi189.stdout: "auth": { 2020-11-16T20:17:45.453 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:17:45.453 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 5, 2020-11-16T20:17:45.453 INFO:tasks.workunit.client.0.smithi189.stdout: "num_secrets": 3 2020-11-16T20:17:45.453 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.453 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 1, 2020-11-16T20:17:45.453 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_active": 1, 2020-11-16T20:17:45.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_unknown": 0, 2020-11-16T20:17:45.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osd": 3, 2020-11-16T20:17:45.454 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_sum": { 2020-11-16T20:17:45.454 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:17:45.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 62471, 2020-11-16T20:17:45.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 101, 2020-11-16T20:17:45.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 1, 2020-11-16T20:17:45.454 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 303, 2020-11-16T20:17:45.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:17:45.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:17:45.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 101, 2020-11-16T20:17:45.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:17:45.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:17:45.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 101, 2020-11-16T20:17:45.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:17:45.455 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:17:45.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:17:45.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 101, 2020-11-16T20:17:45.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 101, 2020-11-16T20:17:45.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:17:45.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:17:45.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:17:45.456 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:17:45.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:17:45.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:17:45.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:17:45.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:17:45.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:17:45.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:17:45.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:17:45.457 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:17:45.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:17:45.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:17:45.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:17:45.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:17:45.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:17:45.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:17:45.458 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:17:45.459 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:17:45.459 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:17:45.459 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:17:45.459 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:17:45.459 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:17:45.459 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:17:45.459 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.459 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:17:45.460 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:17:45.460 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:17:45.460 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:17:45.460 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 0, 2020-11-16T20:17:45.460 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 0, 2020-11-16T20:17:45.460 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:17:45.460 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:17:45.461 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:17:45.461 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:17:45.461 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:17:45.461 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.461 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 18, 2020-11-16T20:17:45.461 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 18, 2020-11-16T20:17:45.461 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 3, 2020-11-16T20:17:45.461 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 3, 2020-11-16T20:17:45.462 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 0 2020-11-16T20:17:45.462 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.462 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum": { 2020-11-16T20:17:45.462 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:17:45.462 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:17:45.462 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 3, 2020-11-16T20:17:45.462 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:17:45.463 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:17:45.463 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:17:45.463 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:17:45.463 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 19220, 2020-11-16T20:17:45.463 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2228, 2020-11-16T20:17:45.463 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 7, 2020-11-16T20:17:45.463 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 16952, 2020-11-16T20:17:45.463 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314553580, 2020-11-16T20:17:45.464 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:17:45.464 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:17:45.464 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322102865920, 2020-11-16T20:17:45.464 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:17:45.464 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2281472, 2020-11-16T20:17:45.464 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 1568461, 2020-11-16T20:17:45.464 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:17:45.464 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:17:45.465 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:17:45.465 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 7442, 2020-11-16T20:17:45.465 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 17359598 2020-11-16T20:17:45.465 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.465 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:17:45.465 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:17:45.465 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:17:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:17:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:17:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:17:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:17:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:17:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 0, 2020-11-16T20:17:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 0, 2020-11-16T20:17:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 0, 2020-11-16T20:17:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 0 2020-11-16T20:17:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:17:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:17:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_sum_by_class": { 2020-11-16T20:17:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "hdd": { 2020-11-16T20:17:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "up_from": 0, 2020-11-16T20:17:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 0, 2020-11-16T20:17:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pgs": 3, 2020-11-16T20:17:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "num_osds": 3, 2020-11-16T20:17:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_osds": 3, 2020-11-16T20:17:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "num_per_pool_omap_osds": 3, 2020-11-16T20:17:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "kb": 314572800, 2020-11-16T20:17:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used": 19220, 2020-11-16T20:17:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_data": 2228, 2020-11-16T20:17:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_omap": 7, 2020-11-16T20:17:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_used_meta": 16952, 2020-11-16T20:17:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "kb_avail": 314553580, 2020-11-16T20:17:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "statfs": { 2020-11-16T20:17:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 322122547200, 2020-11-16T20:17:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 322102865920, 2020-11-16T20:17:45.470 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:17:45.470 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 2281472, 2020-11-16T20:17:45.470 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 1568461, 2020-11-16T20:17:45.470 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:17:45.470 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:17:45.470 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:17:45.470 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 7442, 2020-11-16T20:17:45.471 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 17359598 2020-11-16T20:17:45.471 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.471 INFO:tasks.workunit.client.0.smithi189.stdout: "hb_peers": [], 2020-11-16T20:17:45.471 INFO:tasks.workunit.client.0.smithi189.stdout: "snap_trim_queue_len": 0, 2020-11-16T20:17:45.471 INFO:tasks.workunit.client.0.smithi189.stdout: "num_snap_trimming": 0, 2020-11-16T20:17:45.471 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shards_repaired": 0, 2020-11-16T20:17:45.471 INFO:tasks.workunit.client.0.smithi189.stdout: "op_queue_age_hist": { 2020-11-16T20:17:45.471 INFO:tasks.workunit.client.0.smithi189.stdout: "histogram": [], 2020-11-16T20:17:45.472 INFO:tasks.workunit.client.0.smithi189.stdout: "upper_bound": 1 2020-11-16T20:17:45.472 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.472 INFO:tasks.workunit.client.0.smithi189.stdout: "perf_stat": { 2020-11-16T20:17:45.472 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ms": 0, 2020-11-16T20:17:45.472 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ms": 0, 2020-11-16T20:17:45.472 INFO:tasks.workunit.client.0.smithi189.stdout: "commit_latency_ns": 0, 2020-11-16T20:17:45.472 INFO:tasks.workunit.client.0.smithi189.stdout: "apply_latency_ns": 0 2020-11-16T20:17:45.473 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.473 INFO:tasks.workunit.client.0.smithi189.stdout: "alerts": [], 2020-11-16T20:17:45.473 INFO:tasks.workunit.client.0.smithi189.stdout: "network_ping_times": [] 2020-11-16T20:17:45.473 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.473 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.473 INFO:tasks.workunit.client.0.smithi189.stdout: "pool_stats": [ 2020-11-16T20:17:45.473 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.473 INFO:tasks.workunit.client.0.smithi189.stdout: "poolid": 1, 2020-11-16T20:17:45.474 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg": 1, 2020-11-16T20:17:45.474 INFO:tasks.workunit.client.0.smithi189.stdout: "stat_sum": { 2020-11-16T20:17:45.474 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes": 62471, 2020-11-16T20:17:45.474 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects": 101, 2020-11-16T20:17:45.474 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_clones": 1, 2020-11-16T20:17:45.474 INFO:tasks.workunit.client.0.smithi189.stdout: "num_object_copies": 303, 2020-11-16T20:17:45.474 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing_on_primary": 0, 2020-11-16T20:17:45.474 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_missing": 0, 2020-11-16T20:17:45.475 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_degraded": 101, 2020-11-16T20:17:45.475 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_misplaced": 0, 2020-11-16T20:17:45.475 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_unfound": 0, 2020-11-16T20:17:45.475 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_dirty": 101, 2020-11-16T20:17:45.475 INFO:tasks.workunit.client.0.smithi189.stdout: "num_whiteouts": 0, 2020-11-16T20:17:45.475 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read": 0, 2020-11-16T20:17:45.475 INFO:tasks.workunit.client.0.smithi189.stdout: "num_read_kb": 0, 2020-11-16T20:17:45.476 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write": 101, 2020-11-16T20:17:45.476 INFO:tasks.workunit.client.0.smithi189.stdout: "num_write_kb": 101, 2020-11-16T20:17:45.476 INFO:tasks.workunit.client.0.smithi189.stdout: "num_scrub_errors": 0, 2020-11-16T20:17:45.476 INFO:tasks.workunit.client.0.smithi189.stdout: "num_shallow_scrub_errors": 0, 2020-11-16T20:17:45.476 INFO:tasks.workunit.client.0.smithi189.stdout: "num_deep_scrub_errors": 0, 2020-11-16T20:17:45.476 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_recovered": 0, 2020-11-16T20:17:45.476 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_recovered": 0, 2020-11-16T20:17:45.476 INFO:tasks.workunit.client.0.smithi189.stdout: "num_keys_recovered": 0, 2020-11-16T20:17:45.477 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_omap": 0, 2020-11-16T20:17:45.477 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_hit_set_archive": 0, 2020-11-16T20:17:45.477 INFO:tasks.workunit.client.0.smithi189.stdout: "num_bytes_hit_set_archive": 0, 2020-11-16T20:17:45.477 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush": 0, 2020-11-16T20:17:45.477 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_kb": 0, 2020-11-16T20:17:45.477 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict": 0, 2020-11-16T20:17:45.477 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_kb": 0, 2020-11-16T20:17:45.477 INFO:tasks.workunit.client.0.smithi189.stdout: "num_promote": 0, 2020-11-16T20:17:45.478 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_high": 0, 2020-11-16T20:17:45.478 INFO:tasks.workunit.client.0.smithi189.stdout: "num_flush_mode_low": 0, 2020-11-16T20:17:45.478 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_some": 0, 2020-11-16T20:17:45.478 INFO:tasks.workunit.client.0.smithi189.stdout: "num_evict_mode_full": 0, 2020-11-16T20:17:45.478 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_pinned": 0, 2020-11-16T20:17:45.478 INFO:tasks.workunit.client.0.smithi189.stdout: "num_legacy_snapsets": 0, 2020-11-16T20:17:45.478 INFO:tasks.workunit.client.0.smithi189.stdout: "num_large_omap_objects": 0, 2020-11-16T20:17:45.479 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_manifest": 0, 2020-11-16T20:17:45.479 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_bytes": 0, 2020-11-16T20:17:45.479 INFO:tasks.workunit.client.0.smithi189.stdout: "num_omap_keys": 0, 2020-11-16T20:17:45.479 INFO:tasks.workunit.client.0.smithi189.stdout: "num_objects_repaired": 0 2020-11-16T20:17:45.479 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.479 INFO:tasks.workunit.client.0.smithi189.stdout: "store_stats": { 2020-11-16T20:17:45.479 INFO:tasks.workunit.client.0.smithi189.stdout: "total": 0, 2020-11-16T20:17:45.479 INFO:tasks.workunit.client.0.smithi189.stdout: "available": 0, 2020-11-16T20:17:45.480 INFO:tasks.workunit.client.0.smithi189.stdout: "internally_reserved": 0, 2020-11-16T20:17:45.480 INFO:tasks.workunit.client.0.smithi189.stdout: "allocated": 1241088, 2020-11-16T20:17:45.480 INFO:tasks.workunit.client.0.smithi189.stdout: "data_stored": 1241088, 2020-11-16T20:17:45.480 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed": 0, 2020-11-16T20:17:45.480 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_allocated": 0, 2020-11-16T20:17:45.480 INFO:tasks.workunit.client.0.smithi189.stdout: "data_compressed_original": 0, 2020-11-16T20:17:45.480 INFO:tasks.workunit.client.0.smithi189.stdout: "omap_allocated": 0, 2020-11-16T20:17:45.480 INFO:tasks.workunit.client.0.smithi189.stdout: "internal_metadata": 0 2020-11-16T20:17:45.481 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.481 INFO:tasks.workunit.client.0.smithi189.stdout: "log_size": 18, 2020-11-16T20:17:45.481 INFO:tasks.workunit.client.0.smithi189.stdout: "ondisk_log_size": 18, 2020-11-16T20:17:45.481 INFO:tasks.workunit.client.0.smithi189.stdout: "up": 3, 2020-11-16T20:17:45.481 INFO:tasks.workunit.client.0.smithi189.stdout: "acting": 3, 2020-11-16T20:17:45.481 INFO:tasks.workunit.client.0.smithi189.stdout: "num_store_stats": 3 2020-11-16T20:17:45.481 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.482 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.482 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_stats": [ 2020-11-16T20:17:45.482 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.482 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:17:45.482 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 184683593738 2020-11-16T20:17:45.482 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.482 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.482 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:17:45.483 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 42949672981 2020-11-16T20:17:45.483 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.483 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.483 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:17:45.483 INFO:tasks.workunit.client.0.smithi189.stdout: "seq": 197568495626 2020-11-16T20:17:45.483 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.483 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.484 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_state": [ 2020-11-16T20:17:45.484 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.484 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "active+undersized+degraded", 2020-11-16T20:17:45.484 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 1 2020-11-16T20:17:45.484 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.484 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.484 INFO:tasks.workunit.client.0.smithi189.stdout: "num_pg_by_osd": [ 2020-11-16T20:17:45.484 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.485 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2, 2020-11-16T20:17:45.485 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:17:45.485 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:17:45.485 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:17:45.485 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.485 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.485 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 2147483647, 2020-11-16T20:17:45.486 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:17:45.486 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:17:45.486 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:17:45.486 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.486 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.486 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 1, 2020-11-16T20:17:45.486 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 0, 2020-11-16T20:17:45.487 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 0, 2020-11-16T20:17:45.487 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:17:45.487 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.487 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.487 INFO:tasks.workunit.client.0.smithi189.stdout: "osd": 0, 2020-11-16T20:17:45.487 INFO:tasks.workunit.client.0.smithi189.stdout: "num_primary_pg": 1, 2020-11-16T20:17:45.487 INFO:tasks.workunit.client.0.smithi189.stdout: "num_acting_pg": 1, 2020-11-16T20:17:45.487 INFO:tasks.workunit.client.0.smithi189.stdout: "num_up_not_acting_pg": 0 2020-11-16T20:17:45.488 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.488 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.488 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": [ 2020-11-16T20:17:45.488 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:17:45.488 INFO:tasks.workunit.client.0.smithi189.stdout: "pool": 1, 2020-11-16T20:17:45.488 INFO:tasks.workunit.client.0.smithi189.stdout: "purged_snaps": {} 2020-11-16T20:17:45.488 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.488 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:17:45.489 INFO:tasks.workunit.client.0.smithi189.stdout: "servicemap": { 2020-11-16T20:17:45.489 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": 5, 2020-11-16T20:17:45.489 INFO:tasks.workunit.client.0.smithi189.stdout: "modified": "2020-11-16T20:16:16.217809+0000", 2020-11-16T20:17:45.489 INFO:tasks.workunit.client.0.smithi189.stdout: "services": {} 2020-11-16T20:17:45.489 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:17:45.489 INFO:tasks.workunit.client.0.smithi189.stdout: "paxos": { 2020-11-16T20:17:45.489 INFO:tasks.workunit.client.0.smithi189.stdout: "first_committed": 1, 2020-11-16T20:17:45.490 INFO:tasks.workunit.client.0.smithi189.stdout: "last_committed": 507, 2020-11-16T20:17:45.490 INFO:tasks.workunit.client.0.smithi189.stdout: "last_pn": 0, 2020-11-16T20:17:45.490 INFO:tasks.workunit.client.0.smithi189.stdout: "accepted_pn": 0 2020-11-16T20:17:45.490 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:17:45.490 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:17:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:report 146110740 2020-11-16T20:17:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2020-11-16T20:17:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2020-11-16T20:17:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:17:45.492 INFO:tasks.workunit.client.0.smithi189.stdout:delay recovery 2020-11-16T20:17:45.687 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:17:45.687 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 0 101 0 0 62471 0 0 18 18 active+undersized+degraded 2020-11-16T20:17:09.649926+0000 41'102 47:139 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:17:45.687 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:17:45.688 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:17:45.690 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:17:45.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:17:45.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2020-11-16T20:17:45.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T20:17:45.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T20:17:45.700 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:17:45.701 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:17:45.701 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:17:45.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.0.asok 2020-11-16T20:17:45.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-11-16T20:17:45.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.0.asok set_recovery_delay 100000 2020-11-16T20:17:45.804 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:17:45.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-11-16T20:17:45.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2020-11-16T20:17:45.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-11-16T20:17:45.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-11-16T20:17:45.814 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:17:45.815 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:17:45.815 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:17:45.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.2.asok 2020-11-16T20:17:45.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-11-16T20:17:45.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.2.asok set_recovery_delay 100000 2020-11-16T20:17:45.911 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:17:45.919 INFO:tasks.workunit.client.0.smithi189.stdout:revive divergent 1 2020-11-16T20:17:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2020-11-16T20:17:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:17:46.289 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:17:46.289 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 0 101 0 0 62471 0 0 18 18 active+undersized+degraded 2020-11-16T20:17:09.649926+0000 41'102 47:139 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:17:46.290 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:17:46.290 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:17:46.291 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:17:46.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2020-11-16T20:17:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:noup is set 2020-11-16T20:17:46.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-11-16T20:17:46.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:17:46.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:17:46.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T20:17:46.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:17:46.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:17:46.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:17:46.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:17:46.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:17:46.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:17:46.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:17:46.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:17:46.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:17:46.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:17:46.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:17:46.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:17:46.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:17:46.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:17:46.992 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:17:46.992 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:17:46.992 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:17:46.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:17:46.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:17:46.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:17:46.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:17:46.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:17:46.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:17:46.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:17:46.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:17:46.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:17:46.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:17:46.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:17:46.994 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:17:46.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T20:17:46.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:17:46.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2020-11-16T20:17:46.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T20:17:46.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:17:46.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:17:46.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:17:47.027 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:47.019+0000 7f026278fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:17:47.030 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:47.023+0000 7f026278fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:17:47.032 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:47.023+0000 7f026278fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:17:47.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2020-11-16T20:17:47.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2020-11-16T20:17:47.570 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:47.563+0000 7f026278fd80 -1 Falling back to public interface 2020-11-16T20:17:48.580 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:17:48.579+0000 7f026278fd80 -1 osd.1 30 log_to_monitors {default=true} 2020-11-16T20:17:52.378 INFO:tasks.workunit.client.0.smithi189.stdout:delay recovery divergent 2020-11-16T20:17:52.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2020-11-16T20:17:52.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:17:52.756 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:17:52.756 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 0 101 0 0 62471 0 0 18 18 active+undersized+degraded 2020-11-16T20:17:09.649926+0000 41'102 47:140 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:17:52.756 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:17:52.756 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:17:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:17:52.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2020-11-16T20:17:52.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:17:52.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:17:52.769 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:17:52.769 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:17:52.769 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:17:52.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12850/ceph-osd.1.asok 2020-11-16T20:17:52.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2020-11-16T20:17:52.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.12850/ceph-osd.1.asok set_recovery_delay 100000 2020-11-16T20:17:52.877 INFO:tasks.workunit.client.0.smithi189.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-11-16T20:17:52.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2020-11-16T20:17:53.578 INFO:tasks.workunit.client.0.smithi189.stderr:noup is unset 2020-11-16T20:17:53.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2020-11-16T20:17:53.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:17:53.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:17:53.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:17:53.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:17:53.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:17:53.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:17:53.591 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:17:53.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:17:53.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:17:54.025 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 43 up_thru 46 down_at 42 last_clean_interval [36,41) [v2:127.0.0.1:6802/55203,v1:127.0.0.1:6803/55203] [v2:127.0.0.1:6804/55203,v1:127.0.0.1:6805/55203] exists,up 8e23841d-98e5-472c-91fe-6a992d391d13 2020-11-16T20:17:54.026 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:17:54.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:17:54.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:17:54.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:17:54.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2020-11-16T20:17:54.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:17:54.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:17:54.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:17:54.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:17:54.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:17:54.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:17:54.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:17:54.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:17:54.398 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 51 up_thru 51 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/57548,v1:127.0.0.1:6819/57548] [v2:127.0.0.1:6820/57548,v1:127.0.0.1:6821/57548] exists,up fada656c-964b-4065-ada2-6f88fccdceb7 2020-11-16T20:17:54.398 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:17:54.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:17:54.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:17:54.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:17:54.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2020-11-16T20:17:54.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:17:54.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:17:54.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:17:54.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:17:54.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:17:54.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:17:54.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:17:54.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:17:54.794 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 46 up_thru 0 down_at 45 last_clean_interval [37,41) [v2:127.0.0.1:6810/55852,v1:127.0.0.1:6811/55852] [v2:127.0.0.1:6812/55852,v1:127.0.0.1:6813/55852] exists,up 41ef07c7-ba20-4c7c-b7b2-5bc264c4a07a 2020-11-16T20:17:54.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:17:54.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:17:54.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:17:54.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:17:55.164 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:17:55.164 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 0 101 0 0 62471 0 0 18 18 active+undersized+degraded 2020-11-16T20:17:09.649926+0000 41'102 47:140 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:17:55.164 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:17:55.164 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:17:55.166 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:17:55.176 INFO:tasks.workunit.client.0.smithi189.stdout:wait for peering 2020-11-16T20:17:55.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2020-11-16T20:17:55.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:17:55.549 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:17:55.550 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 0 101 0 0 62471 0 0 18 18 active+undersized+degraded 2020-11-16T20:17:09.649926+0000 41'102 47:140 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:17:55.550 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:17:55.550 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:17:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:17:55.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2020-11-16T20:18:05.747 INFO:tasks.workunit.client.0.smithi189.stdout:killing divergent 1 2020-11-16T20:18:05.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2020-11-16T20:18:05.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:18:06.133 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:18:06.134 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 4 4 0 0 62471 0 0 12 12 active+recovering+degraded+wait 2020-11-16T20:17:54.158829+0000 41'102 52:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:18:06.134 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:18:06.134 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:18:06.136 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:18:06.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2020-11-16T20:18:06.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:18:06.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:18:06.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:18:06.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:18:06.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:18:06.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:18:06.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2020-11-16T20:18:06.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:18:06.257 INFO:tasks.workunit.client.0.smithi189.stdout:reviving divergent 1 2020-11-16T20:18:06.636 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:18:06.636 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 101 4 4 0 0 62471 0 0 12 12 active+recovering+degraded+wait 2020-11-16T20:17:54.158829+0000 41'102 52:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:18:06.636 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:18:06.636 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:18:06.638 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:18:06.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-11-16T20:18:06.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2020-11-16T20:18:06.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:18:06.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T20:18:06.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:18:06.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2020-11-16T20:18:06.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-11-16T20:18:06.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:18:06.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:18:06.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:18:06.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-11-16T20:18:06.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-11-16T20:18:06.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:18:06.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:18:06.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-11-16T20:18:06.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:18:06.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:18:06.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:18:06.652 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:18:06.652 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:18:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:18:06.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:18:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' 2020-11-16T20:18:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:18:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-11-16T20:18:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-11-16T20:18:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:18:06.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:18:06.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:18:06.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:18:06.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:18:06.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2020-11-16T20:18:06.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T20:18:06.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=37a1fb3b-5cce-4fc2-9886-c734e76404a1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.12850/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:18:06.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2020-11-16T20:18:06.655 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:18:06.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T20:18:06.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:18:06.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:18:06.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:18:06.686 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:06.683+0000 7f6975d1dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:18:06.694 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:06.691+0000 7f6975d1dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:18:06.695 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:06.691+0000 7f6975d1dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:18:07.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T20:18:07.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:18:07.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:18:07.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:18:07.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:18:07.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:18:07.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:18:07.018 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:18:07.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:18:07.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:18:07.235 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:07.231+0000 7f6975d1dd80 -1 Falling back to public interface 2020-11-16T20:18:07.419 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 51 up_thru 51 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/57548,v1:127.0.0.1:6819/57548] [v2:127.0.0.1:6820/57548,v1:127.0.0.1:6821/57548] exists,up fada656c-964b-4065-ada2-6f88fccdceb7 2020-11-16T20:18:07.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:18:07.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:18:07.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:18:07.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2020-11-16T20:18:08.382 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:08.379+0000 7f6975d1dd80 -1 osd.1 52 log_to_monitors {default=true} 2020-11-16T20:18:27.422 INFO:tasks.workunit.client.0.smithi189.stdout:allowing recovery 2020-11-16T20:18:27.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2020-11-16T20:18:27.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:18:27.804 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:18:27.805 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 102 0 0 0 0 63088 0 0 13 13 active+clean 2020-11-16T20:18:10.326845+0000 52'103 56:154 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:18:27.805 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:18:27.805 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:18:27.807 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:18:27.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-11-16T20:18:27.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2020-11-16T20:18:27.975 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:18:27.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-11-16T20:18:27.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T20:18:28.146 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:18:28.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-11-16T20:18:28.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2020-11-16T20:18:28.306 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:18:28.315 INFO:tasks.workunit.client.0.smithi189.stdout:reading divergent objects 2020-11-16T20:18:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2020-11-16T20:18:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2020-11-16T20:18:28.667 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:18:28.667 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 102 0 0 0 0 63088 0 0 13 13 active+clean 2020-11-16T20:18:10.326845+0000 52'103 56:154 [1,0,2] 1 [1,0,2] 1 0'0 2020-11-16T20:15:17.806340+0000 0'0 2020-11-16T20:15:17.806340+0000 0 2020-11-16T20:18:28.667 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:18:28.668 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:18:28.669 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:18:28.680 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2020-11-16T20:18:28.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2020-11-16T20:18:28.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:18:28.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2020-11-16T20:18:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:18:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2020-11-16T20:18:28.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:18:28.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2020-11-16T20:18:28.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-11-16T20:18:28.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2020-11-16T20:18:28.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2020-11-16T20:18:28.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-11-16T20:18:28.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2020-11-16T20:18:28.943 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-11-16T20:15:53.871029+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-11-16T20:18:28.943 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-11-16T20:15:53.871029+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-11-16T20:18:28.943 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:18:28.943 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2020-11-16T20:18:28.944 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2020-11-16T20:18:28.944 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2020-11-16T20:18:28.944 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4555.0:1 2020-11-16T20:16:01.509001+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-11-16T20:18:28.944 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4555.0:1 2020-11-16T20:16:01.509001+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-11-16T20:18:28.944 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:18:28.944 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2020-11-16T20:18:28.944 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2020-11-16T20:18:28.945 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2020-11-16T20:18:28.945 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2020-11-16T20:18:28.945 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4555.0:1 2020-11-16T20:16:01.509001+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-11-16T20:18:28.945 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2020-11-16T20:18:28.945 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4546.0:1 2020-11-16T20:16:00.219999+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4552.0:1 2020-11-16T20:16:01.231759+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:18:28.946 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4546.0:1 2020-11-16T20:16:00.219999+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:18:28.946 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4552.0:1 2020-11-16T20:16:01.231759+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:18:28.946 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-11-16T20:18:28.946 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2020-11-16T20:18:28.946 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-11-16T20:18:28.946 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-11-16T20:18:28.947 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2020-11-16T20:18:28.947 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4552.0:1 2020-11-16T20:16:01.231759+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-11-16T20:18:28.947 INFO:tasks.workunit.client.0.smithi189.stdout:td/divergent-priors/osd.1.log:2020-11-16T20:17:54.019+0000 7f0240c1b700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-11-16T20:18:28.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-11-16T20:18:28.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2020-11-16T20:18:28.954 INFO:tasks.workunit.client.0.smithi189.stdout:success 2020-11-16T20:18:28.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2020-11-16T20:18:28.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2020-11-16T20:18:28.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T20:18:28.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T20:18:29.525 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T20:18:29.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2020-11-16T20:18:29.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:18:29.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:18:29.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:18:29.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:18:29.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:18:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:18:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-11-16T20:18:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-11-16T20:18:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:18:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-11-16T20:18:29.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:18:29.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:18:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:18:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:18:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:18:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:18:29.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:18:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:18:29.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:18:29.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:18:29.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:18:29.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:18:29.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:18:29.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:18:29.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:18:29.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:18:29.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:18:29.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:18:29.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-11-16T20:18:29.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:18:29.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:18:29.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:18:29.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12850 2020-11-16T20:18:29.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:18:29.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:18:29.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T20:18:29.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/divergent-priors 0 2020-11-16T20:18:29.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-11-16T20:18:29.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T20:18:29.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-11-16T20:18:29.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:18:29.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:18:29.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:18:29.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:18:29.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:18:29.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:18:29.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:18:29.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:18:29.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:18:29.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:18:29.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:18:29.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:18:29.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:18:29.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:18:29.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:18:29.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:18:29.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:18:29.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T20:18:29.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-11-16T20:18:29.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:18:29.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:18:29.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12850 2020-11-16T20:18:29.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12850 2020-11-16T20:18:29.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:18:29.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:18:29.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T20:18:29.925 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T20:18:29.925 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T20:18:29.978 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2020-11-16T20:18:29.979 INFO:teuthology.orchestra.run.smithi189:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2020-11-16T20:18:30.039 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T20:18:30.052 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T20:18:30.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:18:30.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:18:30.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T20:18:30.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T20:18:30.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T20:18:30.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T20:18:30.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T20:18:30.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T20:18:30.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/ec-error-rollforward 2020-11-16T20:18:30.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2020-11-16T20:18:30.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2020-11-16T20:18:30.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2020-11-16T20:18:30.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2020-11-16T20:18:30.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2020-11-16T20:18:30.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2020-11-16T20:18:30.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none ' 2020-11-16T20:18:30.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2020-11-16T20:18:30.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2020-11-16T20:18:30.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2020-11-16T20:18:30.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2020-11-16T20:18:30.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2020-11-16T20:18:30.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2020-11-16T20:18:30.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2020-11-16T20:18:30.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2020-11-16T20:18:30.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T20:18:30.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2020-11-16T20:18:30.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2020-11-16T20:18:30.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2020-11-16T20:18:30.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2020-11-16T20:18:30.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2020-11-16T20:18:30.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-11-16T20:18:30.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:18:30.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-11-16T20:18:30.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:18:30.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:18:30.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:18:30.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:18:30.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:18:30.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:18:30.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:18:30.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:18:30.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:18:30.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:18:30.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:18:30.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:18:30.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:18:30.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:18:30.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:18:30.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:18:30.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:18:30.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:18:30.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-11-16T20:18:30.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:18:30.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:18:30.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:18:30.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.58817 2020-11-16T20:18:30.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:18:30.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:18:30.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2020-11-16T20:18:30.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:18:30.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:18:30.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:18:30.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.58817 2020-11-16T20:18:30.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:18:30.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T20:18:30.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T20:18:30.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:18:30.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2020-11-16T20:18:30.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2020-11-16T20:18:30.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2020-11-16T20:18:30.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2020-11-16T20:18:30.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2020-11-16T20:18:30.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:18:30.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:18:30.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:18:30.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2020-11-16T20:18:30.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2020-11-16T20:18:30.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:18:30.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:18:30.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:18:30.321 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:18:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:18:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:18:30.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:18:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:18:30.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:18:30.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:18:30.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:18:30.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:18:30.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:18:30.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:18:30.548 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:18:30.548 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:18:30.548 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:18:30.549 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:18:30.549 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:18:30.549 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:18:30.550 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.58817/ceph-mon.a.asok 2020-11-16T20:18:30.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:18:30.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.58817/ceph-mon.a.asok config get fsid 2020-11-16T20:18:30.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:18:30.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:18:30.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:18:30.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:18:30.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:18:30.662 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:18:30.662 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:18:30.662 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:18:30.663 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:18:30.663 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:18:30.663 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:18:30.663 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.58817/ceph-mon.a.asok 2020-11-16T20:18:30.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:18:30.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.58817/ceph-mon.a.asok config get mon_host 2020-11-16T20:18:30.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2020-11-16T20:18:30.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/ec-error-rollforward 2020-11-16T20:18:30.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:18:30.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:18:30.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:18:30.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/ec-error-rollforward/x 2020-11-16T20:18:30.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:18:31.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:18:31.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:18:31.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:18:31.287 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:18:31.287 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:18:31.287 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:18:31.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:18:31.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:18:31.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:18:31.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2020-11-16T20:18:31.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/ec-error-rollforward 2020-11-16T20:18:31.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:18:31.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:18:31.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:18:31.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/ec-error-rollforward/0 2020-11-16T20:18:31.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-11-16T20:18:31.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:18:31.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:18:31.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:18:31.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-11-16T20:18:31.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-11-16T20:18:31.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:18:31.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:18:31.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-11-16T20:18:31.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:18:31.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:18:31.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:18:31.323 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:18:31.323 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:18:31.323 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:18:31.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:18:31.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:18:31.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:18:31.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:18:31.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:18:31.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-11-16T20:18:31.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-11-16T20:18:31.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:18:31.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:18:31.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:18:31.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:18:31.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:18:31.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/ec-error-rollforward/0 2020-11-16T20:18:31.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:18:31.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=717f0e44-cc28-4b90-922e-c09b8573b2fb 2020-11-16T20:18:31.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 717f0e44-cc28-4b90-922e-c09b8573b2fb' 2020-11-16T20:18:31.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:18:31.330 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 717f0e44-cc28-4b90-922e-c09b8573b2fb 2020-11-16T20:18:31.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCX3rJfTdIMFBAAwKIfsDYghGvv/hT4QYicyg== 2020-11-16T20:18:31.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCX3rJfTdIMFBAAwKIfsDYghGvv/hT4QYicyg=="}' 2020-11-16T20:18:31.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 717f0e44-cc28-4b90-922e-c09b8573b2fb -i td/ec-error-rollforward/0/new.json 2020-11-16T20:18:31.734 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:18:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/ec-error-rollforward/0/new.json 2020-11-16T20:18:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCX3rJfTdIMFBAAwKIfsDYghGvv/hT4QYicyg== --osd-uuid 717f0e44-cc28-4b90-922e-c09b8573b2fb 2020-11-16T20:18:31.776 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:31.770+0000 7fa4c1655d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:18:31.778 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:31.774+0000 7fa4c1655d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:18:31.779 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:31.774+0000 7fa4c1655d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:18:31.780 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:31.774+0000 7fa4c1655d80 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-11-16T20:18:31.780 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:31.774+0000 7fa4c1655d80 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-11-16T20:18:31.780 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:31.774+0000 7fa4c1655d80 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-11-16T20:18:31.821 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:31.818+0000 7fa4c1655d80 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2020-11-16T20:18:36.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2020-11-16T20:18:36.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:18:36.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:18:36.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:18:36.200 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:18:36.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:18:36.703 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:18:36.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:18:36.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:18:36.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:18:36.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:18:36.729 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:36.726+0000 7f60dceced80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:18:36.731 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:36.726+0000 7f60dceced80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:18:36.733 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:36.726+0000 7f60dceced80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:18:37.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:18:37.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:18:37.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:18:37.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:18:37.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:18:37.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:18:37.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:18:37.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:18:37.110 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:18:37.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:18:37.275 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:37.270+0000 7f60dceced80 -1 Falling back to public interface 2020-11-16T20:18:37.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:18:38.136 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:38.130+0000 7f60dceced80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:19:56.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:19:56.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:19:56.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:19:56.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:19:56.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:19:56.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:19:56.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2020-11-16T20:19:56.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/ec-error-rollforward 2020-11-16T20:19:56.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:19:56.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:19:56.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:19:56.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/ec-error-rollforward/1 2020-11-16T20:19:56.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-11-16T20:19:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:19:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:19:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:19:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-11-16T20:19:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-11-16T20:19:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:19:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:19:56.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-11-16T20:19:56.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:19:56.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:19:56.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:19:56.326 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:19:56.326 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.326 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:19:56.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:19:56.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:19:56.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:19:56.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-11-16T20:19:56.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-11-16T20:19:56.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:19:56.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:19:56.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:19:56.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:19:56.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/ec-error-rollforward/1 2020-11-16T20:19:56.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:19:56.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=75d43568-02ff-4df9-858f-23d0e546d39b 2020-11-16T20:19:56.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 75d43568-02ff-4df9-858f-23d0e546d39b' 2020-11-16T20:19:56.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:19:56.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCe3rJfeGxRNBAAIFgdHExOpuFk0Ll6SpAv+Q== 2020-11-16T20:19:56.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCe3rJfeGxRNBAAIFgdHExOpuFk0Ll6SpAv+Q=="}' 2020-11-16T20:19:56.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 75d43568-02ff-4df9-858f-23d0e546d39b -i td/ec-error-rollforward/1/new.json 2020-11-16T20:19:56.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/ec-error-rollforward/1/new.json 2020-11-16T20:19:56.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCe3rJfeGxRNBAAIFgdHExOpuFk0Ll6SpAv+Q== --osd-uuid 75d43568-02ff-4df9-858f-23d0e546d39b 2020-11-16T20:19:56.330 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:39.498+0000 7fdf88f16d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.330 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:39.514+0000 7fdf88f16d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.330 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:39.514+0000 7fdf88f16d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.330 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:39.514+0000 7fdf88f16d80 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-11-16T20:19:56.330 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:39.514+0000 7fdf88f16d80 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-11-16T20:19:56.330 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:39.514+0000 7fdf88f16d80 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-11-16T20:19:56.330 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:39.586+0000 7fdf88f16d80 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2020-11-16T20:19:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2020-11-16T20:19:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:19:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:19:56.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:19:56.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:19:56.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:19:56.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:19:56.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:19:56.333 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:44.730+0000 7f0371ab2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.333 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:44.734+0000 7f0371ab2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.333 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:44.738+0000 7f0371ab2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:19:56.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:19:56.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:19:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:19:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:19:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:19:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:19:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:45.294+0000 7f0371ab2d80 -1 Falling back to public interface 2020-11-16T20:19:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:19:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:45.918+0000 7f0371ab2d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:19:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:19:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:19:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:19:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:19:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:19:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:19:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2020-11-16T20:19:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/ec-error-rollforward 2020-11-16T20:19:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:19:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:19:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:19:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/ec-error-rollforward/2 2020-11-16T20:19:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-11-16T20:19:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:19:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:19:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:19:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-11-16T20:19:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-11-16T20:19:56.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:19:56.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:19:56.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-11-16T20:19:56.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:19:56.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:19:56.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:19:56.338 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:19:56.339 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.339 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:19:56.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:19:56.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:19:56.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:19:56.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-11-16T20:19:56.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-11-16T20:19:56.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:19:56.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:19:56.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:19:56.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:19:56.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/ec-error-rollforward/2 2020-11-16T20:19:56.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:19:56.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c7e40b07-0405-46f6-b21a-95ce32d945e0 2020-11-16T20:19:56.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 c7e40b07-0405-46f6-b21a-95ce32d945e0' 2020-11-16T20:19:56.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:19:56.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCm3rJfkBvxNhAAlHhjOtPZw2pIYqqgSpCRsw== 2020-11-16T20:19:56.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCm3rJfkBvxNhAAlHhjOtPZw2pIYqqgSpCRsw=="}' 2020-11-16T20:19:56.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c7e40b07-0405-46f6-b21a-95ce32d945e0 -i td/ec-error-rollforward/2/new.json 2020-11-16T20:19:56.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/ec-error-rollforward/2/new.json 2020-11-16T20:19:56.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCm3rJfkBvxNhAAlHhjOtPZw2pIYqqgSpCRsw== --osd-uuid c7e40b07-0405-46f6-b21a-95ce32d945e0 2020-11-16T20:19:56.342 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:47.426+0000 7f812b446d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.342 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:47.430+0000 7f812b446d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.342 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:47.434+0000 7f812b446d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.342 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:47.434+0000 7f812b446d80 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-11-16T20:19:56.342 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:47.434+0000 7f812b446d80 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-11-16T20:19:56.343 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:47.434+0000 7f812b446d80 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-11-16T20:19:56.343 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:47.474+0000 7f812b446d80 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2020-11-16T20:19:56.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2020-11-16T20:19:56.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:19:56.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:19:56.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:19:56.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:19:56.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:19:56.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:19:56.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:19:56.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.344 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:52.421+0000 7f325fb64d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.344 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:52.425+0000 7f325fb64d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.344 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:52.425+0000 7f325fb64d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:19:56.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:19:56.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:19:56.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:19:56.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:19:56.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:19:56.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:19:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:52.965+0000 7f325fb64d80 -1 Falling back to public interface 2020-11-16T20:19:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:19:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:53.837+0000 7f325fb64d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:19:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:19:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:19:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:19:56.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:19:56.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:19:56.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:19:56.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2020-11-16T20:19:56.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/ec-error-rollforward 2020-11-16T20:19:56.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:19:56.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T20:19:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:19:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/ec-error-rollforward/3 2020-11-16T20:19:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-11-16T20:19:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:19:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:19:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:19:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-11-16T20:19:56.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-11-16T20:19:56.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:19:56.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:19:56.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-11-16T20:19:56.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:19:56.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:19:56.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:19:56.349 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:19:56.350 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.350 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:19:56.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:19:56.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:19:56.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:19:56.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-11-16T20:19:56.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-11-16T20:19:56.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:19:56.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:19:56.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:19:56.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:19:56.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/ec-error-rollforward/3 2020-11-16T20:19:56.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:19:56.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a2ee7bdf-d74b-400b-9d86-ea085cb77fd0 2020-11-16T20:19:56.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 a2ee7bdf-d74b-400b-9d86-ea085cb77fd0' 2020-11-16T20:19:56.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:19:56.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCu3rJfoiNNIhAAujsLJprL//+JnBKzPYTWpg== 2020-11-16T20:19:56.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCu3rJfoiNNIhAAujsLJprL//+JnBKzPYTWpg=="}' 2020-11-16T20:19:56.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a2ee7bdf-d74b-400b-9d86-ea085cb77fd0 -i td/ec-error-rollforward/3/new.json 2020-11-16T20:19:56.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/ec-error-rollforward/3/new.json 2020-11-16T20:19:56.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCu3rJfoiNNIhAAujsLJprL//+JnBKzPYTWpg== --osd-uuid a2ee7bdf-d74b-400b-9d86-ea085cb77fd0 2020-11-16T20:19:56.353 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:55.189+0000 7f1f1a3b6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.353 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:55.201+0000 7f1f1a3b6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.353 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:55.205+0000 7f1f1a3b6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.353 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:55.205+0000 7f1f1a3b6d80 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-11-16T20:19:56.353 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:55.205+0000 7f1f1a3b6d80 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-11-16T20:19:56.354 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:55.205+0000 7f1f1a3b6d80 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-11-16T20:19:56.354 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:18:55.261+0000 7f1f1a3b6d80 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2020-11-16T20:19:56.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2020-11-16T20:19:56.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:19:56.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T20:19:56.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:19:56.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T20:19:56.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:19:56.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:19:56.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:19:56.355 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:00.373+0000 7f9f9b4dbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.355 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:00.373+0000 7f9f9b4dbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.355 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:00.377+0000 7f9f9b4dbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T20:19:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:19:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T20:19:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:19:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:19:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:19:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:19:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:00.917+0000 7f9f9b4dbd80 -1 Falling back to public interface 2020-11-16T20:19:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:19:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:01.833+0000 7f9f9b4dbd80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T20:19:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:19:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:19:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:19:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:19:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:19:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:19:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2020-11-16T20:19:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2020-11-16T20:19:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'ec' already exists 2020-11-16T20:19:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2020-11-16T20:19:56.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2020-11-16T20:19:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 61658 2020-11-16T20:19:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=63107 2020-11-16T20:19:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2020-11-16T20:19:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2020-11-16T20:19:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 63128' 2020-11-16T20:19:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2020-11-16T20:19:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 63129' 2020-11-16T20:19:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2020-11-16T20:19:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 63130' 2020-11-16T20:19:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2020-11-16T20:19:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2020-11-16T20:19:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2020-11-16T20:19:56.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:19:56.361 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:19:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:19:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:19:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:19:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:19:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 63107 63128 63129 63130 2020-11-16T20:19:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2020-11-16T20:19:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2020-11-16T20:19:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/ec-error-rollforward 2020-11-16T20:19:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:19:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T20:19:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:19:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/ec-error-rollforward/0 2020-11-16T20:19:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-11-16T20:19:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:19:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:19:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:19:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-11-16T20:19:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-11-16T20:19:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:19:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:19:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-11-16T20:19:56.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:19:56.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:19:56.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:19:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:19:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:19:56.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:19:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-11-16T20:19:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-11-16T20:19:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:19:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:19:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:19:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:19:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/ec-error-rollforward/0 2020-11-16T20:19:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T20:19:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/ec-error-rollforward/0/whoami 2020-11-16T20:19:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T20:19:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:19:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:19:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:19:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:08.389+0000 7f95cb3a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:08.393+0000 7f95cb3a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:08.397+0000 7f95cb3a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T20:19:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:19:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:19:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:19:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:19:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:19:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:19:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:08.965+0000 7f95cb3a6d80 -1 Falling back to public interface 2020-11-16T20:19:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:19:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:19:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:19:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2020-11-16T20:19:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/ec-error-rollforward 2020-11-16T20:19:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:19:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T20:19:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:19:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/ec-error-rollforward/1 2020-11-16T20:19:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-11-16T20:19:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:19:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:19:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:19:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-11-16T20:19:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-11-16T20:19:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:19:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:19:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-11-16T20:19:56.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:19:56.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:19:56.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:19:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:19:56.372 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.372 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:19:56.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:19:56.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-11-16T20:19:56.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-11-16T20:19:56.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:19:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:19:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:19:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:19:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/ec-error-rollforward/1 2020-11-16T20:19:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T20:19:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/ec-error-rollforward/1/whoami 2020-11-16T20:19:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T20:19:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:19:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:19:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:19:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:09.185+0000 7fe3ab5a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:09.185+0000 7fe3ab5a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.375 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:09.189+0000 7fe3ab5a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T20:19:56.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:19:56.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:19:56.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:19:56.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:19:56.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:19:56.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:19:56.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.376 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:09.737+0000 7fe3ab5a6d80 -1 Falling back to public interface 2020-11-16T20:19:56.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:19:56.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:19:56.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:19:56.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2020-11-16T20:19:56.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/ec-error-rollforward 2020-11-16T20:19:56.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:19:56.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-11-16T20:19:56.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:19:56.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/ec-error-rollforward/2 2020-11-16T20:19:56.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-11-16T20:19:56.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:19:56.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:19:56.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:19:56.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-11-16T20:19:56.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-11-16T20:19:56.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:19:56.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:19:56.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-11-16T20:19:56.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:19:56.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:19:56.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:19:56.379 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:19:56.379 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.379 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:19:56.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:19:56.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-11-16T20:19:56.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-11-16T20:19:56.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:19:56.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:19:56.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:19:56.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:19:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/ec-error-rollforward/2 2020-11-16T20:19:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-11-16T20:19:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/ec-error-rollforward/2/whoami 2020-11-16T20:19:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-11-16T20:19:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:19:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:19:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:19:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:09.993+0000 7f5941dbbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:09.993+0000 7f5941dbbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:09.997+0000 7f5941dbbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-11-16T20:19:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:19:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:19:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:19:56.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:19:56.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:19:56.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:19:56.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.383 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:10.561+0000 7f5941dbbd80 -1 Falling back to public interface 2020-11-16T20:19:56.383 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:10.641+0000 7f95cb3a6d80 -1 osd.0 25 log_to_monitors {default=true} 2020-11-16T20:19:56.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:19:56.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:19:56.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:19:56.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2020-11-16T20:19:56.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/ec-error-rollforward 2020-11-16T20:19:56.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:19:56.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2020-11-16T20:19:56.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:19:56.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/ec-error-rollforward/3 2020-11-16T20:19:56.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-11-16T20:19:56.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:19:56.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:19:56.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:19:56.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-11-16T20:19:56.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-11-16T20:19:56.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:19:56.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:19:56.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-11-16T20:19:56.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:19:56.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:19:56.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:19:56.386 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:19:56.386 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.387 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:19:56.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' 2020-11-16T20:19:56.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:19:56.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-11-16T20:19:56.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-11-16T20:19:56.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:19:56.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:19:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:19:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:19:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/ec-error-rollforward/3 2020-11-16T20:19:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2020-11-16T20:19:56.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=dc3e5fd6-1df8-44c1-9491-37c3e3b77311 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.58817/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:19:56.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/ec-error-rollforward/3/whoami 2020-11-16T20:19:56.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2020-11-16T20:19:56.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:19:56.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:19:56.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:19:56.389 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:10.801+0000 7f6bc9b2bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.389 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:11.061+0000 7f6bc9b2bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.389 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:11.065+0000 7f6bc9b2bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:19:56.389 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:11.337+0000 7fe3ab5a6d80 -1 osd.1 25 log_to_monitors {default=true} 2020-11-16T20:19:56.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2020-11-16T20:19:56.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:19:56.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T20:19:56.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:19:56.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:19:56.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:19:56.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:19:56.391 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:11.613+0000 7f6bc9b2bd80 -1 Falling back to public interface 2020-11-16T20:19:56.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:19:56.391 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:12.717+0000 7f5941dbbd80 -1 osd.2 24 log_to_monitors {default=true} 2020-11-16T20:19:56.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:19:56.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:19:56.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:19:56.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:19:56.392 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:13.669+0000 7f6bc9b2bd80 -1 osd.3 25 log_to_monitors {default=true} 2020-11-16T20:19:56.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:19:56.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:19:56.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:19:56.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:19:56.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:19:56.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:19:56.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:19:56.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:19:56.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2020-11-16T20:19:56.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:19:56.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:19:56.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:19:56.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:19:56.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:19:56.394 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:19:56.394 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:19:56.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:19:56.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:19:56.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:19:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:19:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:19:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:19:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:19:56.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:19:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:19:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:19:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:19:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:3' 2020-11-16T20:19:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:19:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:19:56.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:19:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964116996 2020-11-16T20:19:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964116996 2020-11-16T20:19:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-115964116996' 2020-11-16T20:19:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:19:56.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:19:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051587 2020-11-16T20:19:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051587 2020-11-16T20:19:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-115964116996 1-124554051587' 2020-11-16T20:19:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:19:56.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:19:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920771 2020-11-16T20:19:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920771 2020-11-16T20:19:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-115964116996 1-124554051587 2-141733920771' 2020-11-16T20:19:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:19:56.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:19:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822659 2020-11-16T20:19:56.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822659 2020-11-16T20:19:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-115964116996 1-124554051587 2-141733920771 3-154618822659' 2020-11-16T20:19:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:19:56.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-115964116996 2020-11-16T20:19:56.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:19:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:19:56.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-115964116996 2020-11-16T20:19:56.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:19:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964116996 2020-11-16T20:19:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 115964116996' 2020-11-16T20:19:56.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:19:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 115964116996 2020-11-16T20:19:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:19:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:19:56.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:19:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964116996 -lt 115964116996 2020-11-16T20:19:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:19:56.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-124554051587 2020-11-16T20:19:56.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:19:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:19:56.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:19:56.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-124554051587 2020-11-16T20:19:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051587 2020-11-16T20:19:56.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 124554051587' 2020-11-16T20:19:56.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:19:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051588 -lt 124554051587 2020-11-16T20:19:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:19:56.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-141733920771 2020-11-16T20:19:56.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:19:56.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:19:56.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-141733920771 2020-11-16T20:19:56.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:19:56.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920771 2020-11-16T20:19:56.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2020-11-16T20:19:56.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:19:56.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920771 -lt 141733920771 2020-11-16T20:19:56.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:19:56.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-154618822659 2020-11-16T20:19:56.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:19:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:19:56.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-154618822659 2020-11-16T20:19:56.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:19:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822659 2020-11-16T20:19:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 154618822659' 2020-11-16T20:19:56.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:19:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822660 -lt 154618822659 2020-11-16T20:19:56.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:19:56.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:19:56.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:19:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:19:56.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:19:56.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:19:56.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:19:56.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:19:56.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:19:56.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:19:56.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:19:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:19:56.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:19:56.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:19:56.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:19:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:19:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:19:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:19:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2020-11-16T20:19:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-11-16T20:19:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:19:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-11-16T20:19:56.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:19:56.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:19:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:19:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:19:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:19:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:19:56.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:19:56.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:19:56.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:19:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:19:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:19:56.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:19:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:19:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:19:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:19:56.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:19:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:19:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:19:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-11-16T20:19:56.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:19:56.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:19:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.58817 2020-11-16T20:19:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:19:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:19:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T20:19:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/ec-error-rollforward 0 2020-11-16T20:19:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-11-16T20:19:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T20:19:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-11-16T20:19:56.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:19:56.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:19:56.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:19:56.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:19:56.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:19:56.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:19:56.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:19:56.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:19:56.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:19:56.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:19:56.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:19:56.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:19:56.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:19:56.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:19:56.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:19:56.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:19:56.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:19:56.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T20:19:56.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-11-16T20:19:56.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:19:56.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.58817 2020-11-16T20:19:56.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.58817 2020-11-16T20:19:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:19:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:19:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T20:19:56.415 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:19:56.415 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/59670,v1:127.0.0.1:6803/59670] [v2:127.0.0.1:6804/59670,v1:127.0.0.1:6805/59670] exists,up 717f0e44-cc28-4b90-922e-c09b8573b2fb 2020-11-16T20:19:56.415 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 75d43568-02ff-4df9-858f-23d0e546d39b 2020-11-16T20:19:56.415 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:19:56.416 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:19:56.416 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:19:56.416 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:19:56.416 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:19:56.416 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/60664,v1:127.0.0.1:6811/60664] [v2:127.0.0.1:6812/60664,v1:127.0.0.1:6813/60664] exists,up 75d43568-02ff-4df9-858f-23d0e546d39b 2020-11-16T20:19:56.416 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 c7e40b07-0405-46f6-b21a-95ce32d945e0 2020-11-16T20:19:56.416 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:19:56.417 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:19:56.417 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:19:56.417 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:19:56.417 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:19:56.417 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/61658,v1:127.0.0.1:6819/61658] [v2:127.0.0.1:6820/61658,v1:127.0.0.1:6821/61658] exists,up c7e40b07-0405-46f6-b21a-95ce32d945e0 2020-11-16T20:19:56.417 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 a2ee7bdf-d74b-400b-9d86-ea085cb77fd0 2020-11-16T20:19:56.417 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T20:19:56.417 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T20:19:56.418 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T20:19:56.418 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:19:56.418 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:19:56.418 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/62653,v1:127.0.0.1:6827/62653] [v2:127.0.0.1:6828/62653,v1:127.0.0.1:6829/62653] exists,up a2ee7bdf-d74b-400b-9d86-ea085cb77fd0 2020-11-16T20:19:56.418 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:19:56.418 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:19:56.418 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/59670,v1:127.0.0.1:6803/59670] [v2:127.0.0.1:6804/59670,v1:127.0.0.1:6805/59670] exists,up 717f0e44-cc28-4b90-922e-c09b8573b2fb 2020-11-16T20:19:56.419 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:19:56.419 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:19:56.419 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/60664,v1:127.0.0.1:6811/60664] [v2:127.0.0.1:6812/60664,v1:127.0.0.1:6813/60664] exists,up 75d43568-02ff-4df9-858f-23d0e546d39b 2020-11-16T20:19:56.419 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:19:56.419 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:19:56.419 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/61658,v1:127.0.0.1:6819/61658] [v2:127.0.0.1:6820/61658,v1:127.0.0.1:6821/61658] exists,up c7e40b07-0405-46f6-b21a-95ce32d945e0 2020-11-16T20:19:56.419 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T20:19:56.419 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:19:56.420 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:19:56.420 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:19:56.420 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 36 up_thru 0 down_at 29 last_clean_interval [20,25) [v2:127.0.0.1:6826/64175,v1:127.0.0.1:6827/64175] [v2:127.0.0.1:6828/64175,v1:127.0.0.1:6829/64175] exists,up a2ee7bdf-d74b-400b-9d86-ea085cb77fd0 2020-11-16T20:19:56.420 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 115964116996 2020-11-16T20:19:56.420 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 124554051587 2020-11-16T20:19:56.420 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 141733920771 2020-11-16T20:19:56.420 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 154618822659 2020-11-16T20:19:56.421 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T20:19:56.421 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T20:19:56.436 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh... 2020-11-16T20:19:56.437 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh 2020-11-16T20:19:56.499 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T20:19:56.512 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T20:19:56.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:19:56.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:19:56.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T20:19:56.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T20:19:56.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T20:19:56.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T20:19:56.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T20:19:56.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T20:19:56.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-backfill-prio 2020-11-16T20:19:56.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2020-11-16T20:19:56.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:22: run: shift 2020-11-16T20:19:56.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-11-16T20:19:56.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-11-16T20:19:56.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2020-11-16T20:19:56.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: uuidgen 2020-11-16T20:19:56.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none ' 2020-11-16T20:19:56.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:19:56.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T20:19:56.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: export objects=50 2020-11-16T20:19:56.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: objects=50 2020-11-16T20:19:56.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: export poolprefix=test 2020-11-16T20:19:56.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: poolprefix=test 2020-11-16T20:19:56.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: export FORCE_PRIO=254 2020-11-16T20:19:56.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: FORCE_PRIO=254 2020-11-16T20:19:56.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: export DEGRADED_PRIO=150 2020-11-16T20:19:56.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: DEGRADED_PRIO=150 2020-11-16T20:19:56.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: export NORMAL_PRIO=110 2020-11-16T20:19:56.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: NORMAL_PRIO=110 2020-11-16T20:19:56.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: set 2020-11-16T20:19:56.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T20:19:56.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: local 'funcs=TEST_backfill_pool_priority 2020-11-16T20:19:56.521 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_priority' 2020-11-16T20:19:56.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-11-16T20:19:56.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-11-16T20:19:56.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-11-16T20:19:56.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-11-16T20:19:56.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-11-16T20:19:56.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:19:56.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-11-16T20:19:56.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:19:56.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:19:56.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:19:56.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:19:56.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:19:56.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:19:56.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:19:56.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:19:56.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:19:56.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:19:56.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:19:56.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:19:56.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:19:56.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:19:56.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:19:56.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:19:56.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:19:56.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:19:56.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-11-16T20:19:56.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:19:56.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:19:56.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.65265 2020-11-16T20:19:56.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:19:56.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:19:56.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-11-16T20:19:56.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:19:56.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:19:56.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.65265 2020-11-16T20:19:56.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:19:56.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T20:19:56.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T20:19:56.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:19:56.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-11-16T20:19:56.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_pool_priority td/osd-backfill-prio 2020-11-16T20:19:56.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:326: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2020-11-16T20:19:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:327: TEST_backfill_pool_priority: local pools=3 2020-11-16T20:19:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:328: TEST_backfill_pool_priority: local OSDS=2 2020-11-16T20:19:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2020-11-16T20:19:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-11-16T20:19:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:19:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:19:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:19:56.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-11-16T20:19:56.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-11-16T20:19:56.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:19:56.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:19:56.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:19:56.856 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:19:56.856 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:56.856 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:19:56.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:19:56.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:19:57.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:19:57.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:19:57.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:19:57.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:19:57.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:19:57.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:19:57.061 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:19:57.061 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:19:57.061 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:19:57.062 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:19:57.062 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:19:57.062 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:19:57.063 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-mon.a.asok 2020-11-16T20:20:01.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:20:01.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.65265/ceph-mon.a.asok config get fsid 2020-11-16T20:20:01.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:20:01.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:20:01.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:20:01.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:20:01.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:20:01.119 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:20:01.119 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:20:01.120 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:20:01.120 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:20:01.120 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:20:01.120 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:20:01.120 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-mon.a.asok 2020-11-16T20:20:01.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:20:01.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.65265/ceph-mon.a.asok config get mon_host 2020-11-16T20:20:01.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2020-11-16T20:20:01.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-prio 2020-11-16T20:20:01.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:20:01.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:20:01.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:20:01.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-prio/x 2020-11-16T20:20:01.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:20:01.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:20:01.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:20:01.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:20:01.123 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:20:01.123 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:20:01.124 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:20:01.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:20:01.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:20:01.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:20:01.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:332: TEST_backfill_pool_priority: export CEPH_ARGS 2020-11-16T20:20:01.125 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: expr 2 - 1 2020-11-16T20:20:01.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: seq 0 1 2020-11-16T20:20:01.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:20:01.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2020-11-16T20:20:01.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2020-11-16T20:20:01.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:20:01.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:20:01.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:20:01.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/0 2020-11-16T20:20:01.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T20:20:01.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:20:01.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:20:01.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:20:01.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-11-16T20:20:01.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-11-16T20:20:01.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:20:01.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:20:01.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-11-16T20:20:01.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:20:01.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:20:01.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:20:01.129 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:20:01.129 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:20:01.130 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:20:01.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:20:01.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:20:01.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:20:01.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:20:01.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:20:01.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-11-16T20:20:01.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-11-16T20:20:01.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:20:01.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:20:01.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:20:01.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:20:01.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:20:01.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/0 2020-11-16T20:20:01.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:20:01.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a92c4b31-a28f-407f-b65a-0ca27eab539f 2020-11-16T20:20:01.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 a92c4b31-a28f-407f-b65a-0ca27eab539f' 2020-11-16T20:20:01.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:20:01.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDt3rJfB+i2NBAADg6B5esOurxdLEcKlurCMg== 2020-11-16T20:20:01.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDt3rJfB+i2NBAADg6B5esOurxdLEcKlurCMg=="}' 2020-11-16T20:20:01.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a92c4b31-a28f-407f-b65a-0ca27eab539f -i td/osd-backfill-prio/0/new.json 2020-11-16T20:20:01.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/0/new.json 2020-11-16T20:20:01.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDt3rJfB+i2NBAADg6B5esOurxdLEcKlurCMg== --osd-uuid a92c4b31-a28f-407f-b65a-0ca27eab539f 2020-11-16T20:20:01.134 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:58.275+0000 7f95280c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:01.134 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:58.279+0000 7f95280c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:01.134 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:58.279+0000 7f95280c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:01.134 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:58.279+0000 7f95280c0d80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-11-16T20:20:01.134 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:58.279+0000 7f95280c0d80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-11-16T20:20:01.135 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:58.279+0000 7f95280c0d80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-11-16T20:20:01.135 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:19:58.323+0000 7f95280c0d80 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-11-16T20:20:01.135 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 a92c4b31-a28f-407f-b65a-0ca27eab539f 2020-11-16T20:20:01.135 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:20:02.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-11-16T20:20:02.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:20:02.845 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:20:02.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:20:02.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:20:03.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:20:03.351 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:20:03.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:20:03.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:20:03.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:20:03.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:20:03.382 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:03.375+0000 7f66702d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:03.384 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:03.379+0000 7f66702d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:03.383+0000 7f66702d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:20:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:20:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:20:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:20:03.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:20:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:20:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:20:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:20:03.789 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:20:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:20:03.940 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:03.935+0000 7f66702d6d80 -1 Falling back to public interface 2020-11-16T20:20:04.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:20:04.797 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:04.783+0000 7f66702d6d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:20:05.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:20:05.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:20:05.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:20:05.191 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:20:05.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:20:05.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:20:05.604 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/66122,v1:127.0.0.1:6803/66122] [v2:127.0.0.1:6804/66122,v1:127.0.0.1:6805/66122] exists,up a92c4b31-a28f-407f-b65a-0ca27eab539f 2020-11-16T20:20:05.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:20:05.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:20:05.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:20:05.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:20:05.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2020-11-16T20:20:05.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2020-11-16T20:20:05.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:20:05.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:20:05.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:20:05.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/1 2020-11-16T20:20:05.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T20:20:05.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:20:05.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:20:05.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:20:05.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-11-16T20:20:05.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-11-16T20:20:05.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:20:05.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:20:05.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-11-16T20:20:05.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:20:05.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:20:05.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:20:05.609 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:20:05.610 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:20:05.610 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:20:05.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:20:05.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:20:05.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:20:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:20:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:20:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-11-16T20:20:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-11-16T20:20:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:20:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:20:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:20:05.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:20:05.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:20:05.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/1 2020-11-16T20:20:05.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:20:05.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=66eda0af-984c-4574-8238-f3d5a4b907ae 2020-11-16T20:20:05.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 66eda0af-984c-4574-8238-f3d5a4b907ae' 2020-11-16T20:20:05.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:20:05.613 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 66eda0af-984c-4574-8238-f3d5a4b907ae 2020-11-16T20:20:05.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD13rJfS21TJRAA3kviw06MHfbiZRXVukUaNQ== 2020-11-16T20:20:05.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD13rJfS21TJRAA3kviw06MHfbiZRXVukUaNQ=="}' 2020-11-16T20:20:05.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 66eda0af-984c-4574-8238-f3d5a4b907ae -i td/osd-backfill-prio/1/new.json 2020-11-16T20:20:06.203 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:20:06.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/1/new.json 2020-11-16T20:20:06.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD13rJfS21TJRAA3kviw06MHfbiZRXVukUaNQ== --osd-uuid 66eda0af-984c-4574-8238-f3d5a4b907ae 2020-11-16T20:20:06.254 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:06.251+0000 7fdb45289d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:06.263 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:06.259+0000 7fdb45289d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:06.264 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:06.259+0000 7fdb45289d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:06.265 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:06.259+0000 7fdb45289d80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-11-16T20:20:06.265 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:06.259+0000 7fdb45289d80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-11-16T20:20:06.265 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:06.259+0000 7fdb45289d80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-11-16T20:20:06.329 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:06.323+0000 7fdb45289d80 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-11-16T20:20:10.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-11-16T20:20:10.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:20:10.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:20:10.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:20:10.965 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:20:11.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:20:11.469 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:20:11.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:20:11.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:20:11.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:20:11.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:20:11.499 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:11.495+0000 7f07e8615d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:11.499+0000 7f07e8615d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:11.499+0000 7f07e8615d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:20:11.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:20:11.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:20:11.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:20:11.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:20:11.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:20:11.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:20:11.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:20:11.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:20:11.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:20:11.859 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:20:12.040 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:12.035+0000 7f07e8615d80 -1 Falling back to public interface 2020-11-16T20:20:12.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:20:12.838 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:20:12.831+0000 7f07e8615d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:20:13.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:20:13.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:20:13.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:20:13.240 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:20:13.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:20:13.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:20:13.644 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/67116,v1:127.0.0.1:6811/67116] [v2:127.0.0.1:6812/67116,v1:127.0.0.1:6813/67116] exists,up 66eda0af-984c-4574-8238-f3d5a4b907ae 2020-11-16T20:20:13.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:20:13.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:20:13.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:20:13.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: seq 1 3 2020-11-16T20:20:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-11-16T20:20:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test1 1 1 2020-11-16T20:20:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2020-11-16T20:20:14.284 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' already exists 2020-11-16T20:20:14.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:20:15.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-11-16T20:20:16.487 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T20:20:16.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-11-16T20:20:16.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test2 1 1 2020-11-16T20:20:16.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2020-11-16T20:20:17.226 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' already exists 2020-11-16T20:20:17.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:20:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-11-16T20:20:18.937 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 2 2020-11-16T20:20:18.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-11-16T20:20:18.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test3 1 1 2020-11-16T20:20:18.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2020-11-16T20:20:19.510 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' already exists 2020-11-16T20:20:19.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:20:20.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2020-11-16T20:20:21.505 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 2 2020-11-16T20:20:21.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: sleep 5 2020-11-16T20:20:26.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:346: TEST_backfill_pool_priority: wait_for_clean 2020-11-16T20:20:26.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:20:26.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:20:26.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:20:26.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:20:26.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:20:26.524 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:20:26.525 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:20:26.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:20:26.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:20:26.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:20:26.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:20:26.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:20:26.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:20:26.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:20:26.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:20:27.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:20:27.139 INFO:tasks.workunit.client.0.smithi189.stderr:1' 2020-11-16T20:20:27.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:20:27.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:20:27.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:20:27.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2020-11-16T20:20:27.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2020-11-16T20:20:27.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2020-11-16T20:20:27.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:20:27.307 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:20:27.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2020-11-16T20:20:27.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2020-11-16T20:20:27.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2020-11-16T20:20:27.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:20:27.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2020-11-16T20:20:27.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:20:27.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:20:27.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2020-11-16T20:20:27.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:20:27.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2020-11-16T20:20:27.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-11-16T20:20:27.491 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836487 2020-11-16T20:20:27.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:20:27.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2020-11-16T20:20:27.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:20:27.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2020-11-16T20:20:27.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:20:27.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:20:27.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2020-11-16T20:20:27.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:20:27.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2020-11-16T20:20:27.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-11-16T20:20:27.898 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672965 2020-11-16T20:20:27.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:20:28.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672965 -lt 42949672965 2020-11-16T20:20:28.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:20:28.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:20:28.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:20:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2020-11-16T20:20:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:20:28.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:20:28.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:20:28.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:20:28.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:20:28.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:20:28.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:20:29.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2020-11-16T20:20:29.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:20:29.185 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:20:29.185 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:20:29.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2020-11-16T20:20:29.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:20:29.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:20:29.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:348: TEST_backfill_pool_priority: ceph pg dump pgs 2020-11-16T20:20:30.042 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:20:30.043 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:20:21.446131+0000 0'0 26:17 [1,0] 1 [1,0] 1 0'0 2020-11-16T20:20:19.303686+0000 0'0 2020-11-16T20:20:19.303686+0000 0 2020-11-16T20:20:30.043 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:20:16.889093+0000 0'0 26:26 [1,0] 1 [1,0] 1 0'0 2020-11-16T20:20:14.071629+0000 0'0 2020-11-16T20:20:14.071629+0000 0 2020-11-16T20:20:30.043 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:20:20.023400+0000 0'0 26:23 [0,1] 0 [0,1] 0 0'0 2020-11-16T20:20:16.920166+0000 0'0 2020-11-16T20:20:16.920166+0000 0 2020-11-16T20:20:30.043 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:20:30.043 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:20:30.045 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:20:30.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:352: TEST_backfill_pool_priority: local PG1 2020-11-16T20:20:30.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:353: TEST_backfill_pool_priority: local POOLNUM1 2020-11-16T20:20:30.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:354: TEST_backfill_pool_priority: local pool1 2020-11-16T20:20:30.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local chk_osd1_1 2020-11-16T20:20:30.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local chk_osd1_2 2020-11-16T20:20:30.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local PG2 2020-11-16T20:20:30.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local POOLNUM2 2020-11-16T20:20:30.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:360: TEST_backfill_pool_priority: local pool2 2020-11-16T20:20:30.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local chk_osd2_1 2020-11-16T20:20:30.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local chk_osd2_2 2020-11-16T20:20:30.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: seq 1 3 2020-11-16T20:20:30.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-11-16T20:20:30.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2020-11-16T20:20:30.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-11-16T20:20:30.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-11-16T20:20:30.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=1 2020-11-16T20:20:30.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-11-16T20:20:30.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=0 2020-11-16T20:20:30.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z '' ']' 2020-11-16T20:20:30.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: PG1=1.0 2020-11-16T20:20:30.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: POOLNUM1=1 2020-11-16T20:20:30.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:373: TEST_backfill_pool_priority: pool1=test1 2020-11-16T20:20:30.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: chk_osd1_1=1 2020-11-16T20:20:30.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: chk_osd1_2=0 2020-11-16T20:20:30.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-11-16T20:20:30.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2020-11-16T20:20:30.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-11-16T20:20:30.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-11-16T20:20:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=0 2020-11-16T20:20:30.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-11-16T20:20:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=1 2020-11-16T20:20:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2020-11-16T20:20:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2020-11-16T20:20:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: PG2=2.0 2020-11-16T20:20:30.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: POOLNUM2=2 2020-11-16T20:20:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:380: TEST_backfill_pool_priority: pool2=test2 2020-11-16T20:20:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: chk_osd2_1=0 2020-11-16T20:20:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: chk_osd2_2=1 2020-11-16T20:20:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: break 2020-11-16T20:20:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2020-11-16T20:20:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:388: TEST_backfill_pool_priority: '[' test2 = '' ']' 2020-11-16T20:20:30.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: seq 1 3 2020-11-16T20:20:30.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-11-16T20:20:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2020-11-16T20:20:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-11-16T20:20:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2020-11-16T20:20:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-11-16T20:20:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2020-11-16T20:20:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:398: TEST_backfill_pool_priority: delete_pool test3 2020-11-16T20:20:30.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2020-11-16T20:20:30.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-11-16T20:20:31.408 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' does not exist 2020-11-16T20:20:31.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:402: TEST_backfill_pool_priority: pool1_extra_prio=1 2020-11-16T20:20:31.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:403: TEST_backfill_pool_priority: pool2_extra_prio=2 2020-11-16T20:20:31.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: expr 150 + 1 + 1 2020-11-16T20:20:31.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_prio=152 2020-11-16T20:20:31.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: expr 150 + 1 + 2 2020-11-16T20:20:31.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_prio=153 2020-11-16T20:20:31.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2020-11-16T20:20:32.275 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 1 2020-11-16T20:20:32.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2020-11-16T20:20:33.053 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 recovery_priority to 1 2020-11-16T20:20:33.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:410: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2020-11-16T20:20:33.948 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 1 2020-11-16T20:20:33.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2020-11-16T20:20:34.701 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 recovery_priority to 2 2020-11-16T20:20:34.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: wait_for_clean 2020-11-16T20:20:34.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:20:34.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:20:34.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:20:34.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:20:34.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:20:34.719 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:20:34.720 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:20:34.720 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:20:34.720 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:20:34.720 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:20:34.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:20:34.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:20:34.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:20:34.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:20:34.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:20:35.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:20:35.319 INFO:tasks.workunit.client.0.smithi189.stderr:1' 2020-11-16T20:20:35.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:20:35.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:20:35.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:20:35.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2020-11-16T20:20:35.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2020-11-16T20:20:35.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2020-11-16T20:20:35.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:20:35.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:20:35.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672969 2020-11-16T20:20:35.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672969 2020-11-16T20:20:35.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2020-11-16T20:20:35.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:20:35.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2020-11-16T20:20:35.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:20:35.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:20:35.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2020-11-16T20:20:35.682 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:20:35.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2020-11-16T20:20:35.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-11-16T20:20:35.684 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836490 2020-11-16T20:20:35.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:20:36.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2020-11-16T20:20:36.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:20:36.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672969 2020-11-16T20:20:36.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:20:36.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:20:36.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2020-11-16T20:20:36.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:20:36.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672969 2020-11-16T20:20:36.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-11-16T20:20:36.109 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672969 2020-11-16T20:20:36.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:20:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672967 -lt 42949672969 2020-11-16T20:20:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:20:37.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:20:37.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:20:37.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672969 -lt 42949672969 2020-11-16T20:20:37.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:20:37.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:20:37.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:20:38.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2020-11-16T20:20:38.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:20:38.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:20:38.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:20:38.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:20:38.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:20:38.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:20:38.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:20:38.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:20:38.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:20:38.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:20:38.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:20:39.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2020-11-16T20:20:39.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:20:39.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:20:39.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-11-16T20:20:39.268 INFO:tasks.workunit.client.0.smithi189.stderr:10+0 records in 2020-11-16T20:20:39.268 INFO:tasks.workunit.client.0.smithi189.stderr:10+0 records out 2020-11-16T20:20:39.268 INFO:tasks.workunit.client.0.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0528959 s, 198 MB/s 2020-11-16T20:20:39.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: p=1 2020-11-16T20:20:39.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-11-16T20:20:39.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-11-16T20:20:39.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:39.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-11-16T20:20:39.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:39.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-11-16T20:20:40.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:40.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-11-16T20:20:40.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:40.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-11-16T20:20:40.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:40.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-11-16T20:20:41.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:41.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-11-16T20:20:41.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:41.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-11-16T20:20:41.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:41.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-11-16T20:20:42.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:42.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-11-16T20:20:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-11-16T20:20:43.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:43.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-11-16T20:20:43.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:43.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-11-16T20:20:44.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:44.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-11-16T20:20:44.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:44.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-11-16T20:20:44.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:44.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-11-16T20:20:45.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:45.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-11-16T20:20:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-11-16T20:20:45.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:45.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-11-16T20:20:46.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:46.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-11-16T20:20:47.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:47.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-11-16T20:20:47.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:47.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-11-16T20:20:47.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:47.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-11-16T20:20:48.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:48.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-11-16T20:20:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-11-16T20:20:50.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:50.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-11-16T20:20:50.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:50.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-11-16T20:20:50.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:50.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-11-16T20:20:51.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:51.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-11-16T20:20:51.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:51.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-11-16T20:20:51.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:51.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-11-16T20:20:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-11-16T20:20:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:52.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-11-16T20:20:53.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:53.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-11-16T20:20:53.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:53.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-11-16T20:20:53.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:53.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-11-16T20:20:54.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:54.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-11-16T20:20:54.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:54.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-11-16T20:20:54.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:54.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-11-16T20:20:55.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:55.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-11-16T20:20:56.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:56.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-11-16T20:20:56.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:56.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-11-16T20:20:56.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:20:56.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-11-16T20:21:01.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:01.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-11-16T20:21:01.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:01.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-11-16T20:21:01.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:01.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-11-16T20:21:01.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:01.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-11-16T20:21:01.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:01.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-11-16T20:21:01.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:01.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-11-16T20:21:01.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:01.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-11-16T20:21:01.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:01.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-11-16T20:21:01.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 1 + 1 2020-11-16T20:21:01.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=2 2020-11-16T20:21:01.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-11-16T20:21:01.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-11-16T20:21:01.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:01.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2020-11-16T20:21:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2020-11-16T20:21:02.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:02.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2020-11-16T20:21:02.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:02.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2020-11-16T20:21:02.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:02.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2020-11-16T20:21:03.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:03.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2020-11-16T20:21:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2020-11-16T20:21:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2020-11-16T20:21:04.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:04.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2020-11-16T20:21:04.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:04.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2020-11-16T20:21:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2020-11-16T20:21:05.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:05.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2020-11-16T20:21:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2020-11-16T20:21:06.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:06.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2020-11-16T20:21:06.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:06.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2020-11-16T20:21:07.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:07.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2020-11-16T20:21:07.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:07.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2020-11-16T20:21:08.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:08.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2020-11-16T20:21:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2020-11-16T20:21:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2020-11-16T20:21:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2020-11-16T20:21:09.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:09.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2020-11-16T20:21:10.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:10.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2020-11-16T20:21:10.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:10.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2020-11-16T20:21:12.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:12.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2020-11-16T20:21:12.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:12.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2020-11-16T20:21:12.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:12.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2020-11-16T20:21:13.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:13.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2020-11-16T20:21:13.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:13.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2020-11-16T20:21:13.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:13.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2020-11-16T20:21:14.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2020-11-16T20:21:15.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:15.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2020-11-16T20:21:15.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:15.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2020-11-16T20:21:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2020-11-16T20:21:16.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:16.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2020-11-16T20:21:16.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:16.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2020-11-16T20:21:16.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:16.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2020-11-16T20:21:17.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:17.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2020-11-16T20:21:18.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:18.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2020-11-16T20:21:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2020-11-16T20:21:18.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:18.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2020-11-16T20:21:19.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:19.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2020-11-16T20:21:19.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:19.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2020-11-16T20:21:19.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:19.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2020-11-16T20:21:20.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:20.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2020-11-16T20:21:20.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:20.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2020-11-16T20:21:21.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:21.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2020-11-16T20:21:21.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:21.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2020-11-16T20:21:22.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:22.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2020-11-16T20:21:22.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-11-16T20:21:22.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2020-11-16T20:21:23.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 2 + 1 2020-11-16T20:21:23.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=3 2020-11-16T20:21:23.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2020-11-16T20:21:23.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test1 2020-11-16T20:21:23.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1-p1 2020-11-16T20:21:23.545 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test1 obj1-p1 2020-11-16T20:21:23.545 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test1 2020-11-16T20:21:23.545 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1-p1 2020-11-16T20:21:23.545 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test1 obj1-p1 2020-11-16T20:21:23.546 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T20:21:23.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T20:21:23.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-11-16T20:21:23.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T20:21:24.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: local otherosd=null 2020-11-16T20:21:24.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:427: TEST_backfill_pool_priority: ceph pg dump pgs 2020-11-16T20:21:24.704 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:21:24.705 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-11-16T20:20:32.407857+0000 37'150 37:191 [1] 1 [1] 1 0'0 2020-11-16T20:20:14.071629+0000 0'0 2020-11-16T20:20:14.071629+0000 0 2020-11-16T20:21:24.705 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 46 0 0 0 0 480247808 0 0 37 37 active+clean 2020-11-16T20:20:34.373502+0000 37'137 37:176 [0] 0 [0] 0 0'0 2020-11-16T20:20:16.920166+0000 0'0 2020-11-16T20:20:16.920166+0000 0 2020-11-16T20:21:24.705 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:21:24.705 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:21:24.707 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:21:24.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: ERRORS=0 2020-11-16T20:21:24.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-11-16T20:21:25.475 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T20:21:25.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-11-16T20:21:26.247 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 2 2020-11-16T20:21:26.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:432: TEST_backfill_pool_priority: sleep 5 2020-11-16T20:21:31.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: get_asok_path osd.1 2020-11-16T20:21:31.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:21:31.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:21:31.269 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:21:31.269 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:21:31.269 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:21:31.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.1.asok 2020-11-16T20:21:31.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: CEPH_ARGS= 2020-11-16T20:21:31.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:21:31.384 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 2020-11-16T20:21:31.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: echo osd.1 2020-11-16T20:21:31.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-11-16T20:21:31.385 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:21:31.385 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:21:31.385 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:21:31.385 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:21:31.385 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:21:31.385 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:21:31.386 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:21:31.386 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:21:31.386 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 152, 2020-11-16T20:21:31.386 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:21:31.386 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:21:31.386 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:21:31.386 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:21:31.387 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:21:31.387 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:21:31.387 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:21:31.387 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:21:31.387 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:21:31.387 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:21:31.387 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "2.0", 2020-11-16T20:21:31.388 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 153, 2020-11-16T20:21:31.388 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:21:31.388 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:21:31.388 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:21:31.388 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:21:31.388 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:21:31.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.0 2020-11-16T20:21:31.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T20:21:31.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T20:21:31.389 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:21:31.390 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:21:31.390 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:21:31.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.0.asok 2020-11-16T20:21:31.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2020-11-16T20:21:31.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.0.asok dump_recovery_reservations 2020-11-16T20:21:31.501 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 2020-11-16T20:21:31.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.0 2020-11-16T20:21:31.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-11-16T20:21:31.502 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:21:31.503 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:21:31.503 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:21:31.503 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:21:31.503 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:21:31.503 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:21:31.503 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:21:31.503 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "2.0", 2020-11-16T20:21:31.504 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 153, 2020-11-16T20:21:31.504 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:21:31.504 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:21:31.504 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:21:31.504 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:21:31.504 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:21:31.504 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:21:31.504 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:21:31.505 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:21:31.505 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:21:31.505 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:21:31.505 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:21:31.505 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 152, 2020-11-16T20:21:31.505 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:21:31.505 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:21:31.506 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:21:31.506 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:21:31.506 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:21:31.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-11-16T20:21:31.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-11-16T20:21:31.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-11-16T20:21:31.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: ITEM=1.0 2020-11-16T20:21:31.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:442: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-11-16T20:21:31.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-11-16T20:21:31.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-11-16T20:21:31.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: PRIO=152 2020-11-16T20:21:31.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-11-16T20:21:31.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-11-16T20:21:31.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-11-16T20:21:31.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-11-16T20:21:31.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: ITEM=1.0 2020-11-16T20:21:31.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:457: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-11-16T20:21:31.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-11-16T20:21:31.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-11-16T20:21:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: PRIO=152 2020-11-16T20:21:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-11-16T20:21:31.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-11-16T20:21:31.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-11-16T20:21:31.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-11-16T20:21:31.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: ITEM=2.0 2020-11-16T20:21:31.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:472: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-11-16T20:21:31.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-11-16T20:21:31.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-11-16T20:21:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: PRIO=153 2020-11-16T20:21:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-11-16T20:21:31.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-11-16T20:21:31.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-11-16T20:21:31.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-11-16T20:21:31.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: ITEM=2.0 2020-11-16T20:21:31.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:487: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-11-16T20:21:31.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-11-16T20:21:31.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-11-16T20:21:31.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: PRIO=153 2020-11-16T20:21:31.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-11-16T20:21:31.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:500: TEST_backfill_pool_priority: wait_for_clean 2020-11-16T20:21:31.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:21:31.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:21:31.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:21:31.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:21:31.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:21:31.562 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:21:31.562 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:21:31.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:21:31.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:21:31.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:21:31.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:21:31.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:21:31.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:21:31.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:21:31.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:21:32.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:21:32.536 INFO:tasks.workunit.client.0.smithi189.stderr:1' 2020-11-16T20:21:32.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:21:32.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:21:32.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:21:32.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836504 2020-11-16T20:21:32.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836504 2020-11-16T20:21:32.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504' 2020-11-16T20:21:32.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:21:32.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:21:32.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672982 2020-11-16T20:21:32.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672982 2020-11-16T20:21:32.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504 1-42949672982' 2020-11-16T20:21:32.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:21:32.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836504 2020-11-16T20:21:32.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:21:32.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:21:32.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836504 2020-11-16T20:21:32.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:21:32.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836504 2020-11-16T20:21:32.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2020-11-16T20:21:32.897 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836504 2020-11-16T20:21:32.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:21:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836502 -lt 21474836504 2020-11-16T20:21:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:21:34.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:21:34.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:21:34.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836504 -lt 21474836504 2020-11-16T20:21:34.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:21:34.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672982 2020-11-16T20:21:34.757 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:21:34.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:21:34.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672982 2020-11-16T20:21:34.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:21:34.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672982 2020-11-16T20:21:34.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2020-11-16T20:21:34.763 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672982 2020-11-16T20:21:34.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:21:35.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672982 -lt 42949672982 2020-11-16T20:21:35.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:21:35.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:35.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:35.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2020-11-16T20:21:35.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:35.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:35.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:35.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:35.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:35.720 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:35.720 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:36.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:36.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:36.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:36.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:36.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:36.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:21:36.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:21:36.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:21:36.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:36.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:36.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:36.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:36.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:36.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:36.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:36.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:37.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:37.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:37.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:37.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:37.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:37.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:38.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17714610 2020-11-16T20:21:38.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17714610 '!=' null 2020-11-16T20:21:38.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:38.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:38.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:38.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:38.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:38.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:38.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:38.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:38.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:38.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:38.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:38.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:38.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:38.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:38.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:39.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:39.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:39.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:39.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:39.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:39.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:39.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:39.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:39.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:39.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20964860 2020-11-16T20:21:39.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20964860 '!=' null 2020-11-16T20:21:39.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:39.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:39.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:39.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:39.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:39.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:39.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:39.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:39.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:39.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:39.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:40.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:40.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:40.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:40.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:40.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:40.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:40.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:41.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20964860 2020-11-16T20:21:41.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20964860 '!=' null 2020-11-16T20:21:41.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:41.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:41.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:41.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:41.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:41.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:41.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:41.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:41.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:41.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:41.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:41.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:41.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:41.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:41.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:42.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:42.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:42.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:42.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:42.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:42.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:42.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:42.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:42.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:43.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23585481 2020-11-16T20:21:43.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23585481 '!=' null 2020-11-16T20:21:43.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:43.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:43.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:43.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:43.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:43.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:43.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:43.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:43.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:43.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:43.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:43.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:43.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:43.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:43.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:43.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:43.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:44.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20091457 2020-11-16T20:21:44.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20091457 '!=' null 2020-11-16T20:21:44.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:44.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:44.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:44.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:44.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:44.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:44.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:44.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:44.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:44.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:44.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:45.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:45.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:45.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:45.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:45.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:45.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:45.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:45.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:45.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:45.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:45.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:45.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:45.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:46.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20091438 2020-11-16T20:21:46.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20091438 '!=' null 2020-11-16T20:21:46.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:46.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:46.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:46.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:46.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:46.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:46.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:46.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:46.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:46.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:46.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:47.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:47.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:47.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:47.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:47.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:47.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:47.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:47.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:47.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:47.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:47.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:47.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:47.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:48.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712357 2020-11-16T20:21:48.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712357 '!=' null 2020-11-16T20:21:48.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:48.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:48.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:48.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:48.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:48.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:48.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:48.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:48.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:48.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:48.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:48.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:48.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:48.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:48.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:49.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:49.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:49.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:49.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:49.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:49.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:49.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:49.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:49.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:49.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712357 2020-11-16T20:21:49.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712357 '!=' null 2020-11-16T20:21:49.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:49.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:49.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:49.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:49.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:49.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:49.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:49.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:49.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:49.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:49.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:50.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:50.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:50.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:50.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:50.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:50.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:50.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:50.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:50.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:50.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:50.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:50.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:50.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:51.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23585872 2020-11-16T20:21:51.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23585872 '!=' null 2020-11-16T20:21:51.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:51.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:51.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:51.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:51.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:51.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:51.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:51.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:51.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:51.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:51.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:51.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:51.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:51.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:51.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:52.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:52.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:53.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586055 2020-11-16T20:21:53.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586055 '!=' null 2020-11-16T20:21:53.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:53.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:53.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:53.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:53.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:53.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:53.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:53.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:57.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:57.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:57.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:57.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:57.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:57.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:57.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:57.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:57.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:57.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:57.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:57.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:57.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:57.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:57.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18344741 2020-11-16T20:21:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18344741 '!=' null 2020-11-16T20:21:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:57.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:57.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:57.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:57.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:57.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:57.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:57.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:57.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:57.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:57.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:57.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:57.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712469 2020-11-16T20:21:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712469 '!=' null 2020-11-16T20:21:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:57.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:57.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:57.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:57.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:57.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:57.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:57.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:57.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:57.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:57.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:57.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:57.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:57.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:57.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:57.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:57.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:57.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:57.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:57.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:57.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:57.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712598 2020-11-16T20:21:57.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712598 '!=' null 2020-11-16T20:21:57.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:57.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:57.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:58.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:58.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:58.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:58.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:58.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:58.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:58.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:58.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:58.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:58.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:58.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:58.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:21:58.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:21:58.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:21:58.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:21:58.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:21:58.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:21:58.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:21:58.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:21:58.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:21:58.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:21:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712598 2020-11-16T20:21:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712598 '!=' null 2020-11-16T20:21:59.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:21:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:21:59.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:21:59.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:21:59.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:21:59.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:21:59.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:21:59.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:21:59.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:21:59.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:21:59.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:21:59.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:21:59.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:21:59.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:21:59.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:00.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:00.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:00.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:00.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:00.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:00.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:00.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:00.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:00.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586313 2020-11-16T20:22:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586313 '!=' null 2020-11-16T20:22:00.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:00.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:00.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:00.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:00.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:00.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:00.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:00.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:00.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:00.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:01.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:22:01.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:01.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:01.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:01.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:01.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:01.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:01.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:01.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:01.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:01.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:01.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:01.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:02.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24459553 2020-11-16T20:22:02.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24459553 '!=' null 2020-11-16T20:22:02.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:02.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:02.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:02.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:02.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:02.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:02.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:02.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:02.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:02.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:02.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:03.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:22:03.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:03.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:03.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:03.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:03.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:03.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:03.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:03.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:03.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:03.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:03.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:03.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:04.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19218614 2020-11-16T20:22:04.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19218614 '!=' null 2020-11-16T20:22:04.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:04.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:04.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:04.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:04.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:04.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:04.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:04.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:04.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:04.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:04.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:04.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:22:04.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:04.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:04.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:05.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:05.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:05.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:05.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:05.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:05.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:05.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:05.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:05.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:06.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24460272 2020-11-16T20:22:06.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24460272 '!=' null 2020-11-16T20:22:06.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:06.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:06.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:06.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:06.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:06.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:06.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:10.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:10.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:10.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:10.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:22:10.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:10.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:10.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:10.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:10.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24460272 2020-11-16T20:22:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24460272 '!=' null 2020-11-16T20:22:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:10.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:10.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:10.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:10.307 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:10.307 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:10.307 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:22:10.308 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:10.308 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:10.308 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:10.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:10.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:10.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:10.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:10.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:10.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24460664 2020-11-16T20:22:10.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24460664 '!=' null 2020-11-16T20:22:10.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:10.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:10.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:10.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:10.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:10.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:10.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:10.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:10.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:10.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:10.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:10.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:22:10.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:10.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:10.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:10.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:10.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:10.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:10.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:10.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:10.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:10.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:10.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:10.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:11.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22713698 2020-11-16T20:22:11.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22713698 '!=' null 2020-11-16T20:22:11.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:11.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:11.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:11.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:11.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:11.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:11.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:11.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:11.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:11.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:11.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:12.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:22:12.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:12.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:12.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:12.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:12.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:12.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:12.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:12.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:12.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:12.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:12.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:12.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:13.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20093028 2020-11-16T20:22:13.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20093028 '!=' null 2020-11-16T20:22:13.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:13.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:13.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:13.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:13.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:13.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:13.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:13.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:13.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:14.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:22:14.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:14.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:14.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:14.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:14.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:14.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:14.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:15.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851415 2020-11-16T20:22:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851415 '!=' null 2020-11-16T20:22:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:15.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:15.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:15.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:15.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:15.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:15.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:15.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:15.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:22:15.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:15.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:15.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:16.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:16.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:17.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17472135 2020-11-16T20:22:17.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17472135 '!=' null 2020-11-16T20:22:17.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:17.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:17.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:17.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:17.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:17.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:17.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:17.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:17.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:17.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:17.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:17.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:22:17.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:17.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:17.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:18.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:18.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:18.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:18.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:18.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:18.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:18.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:18.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:18.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:18.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13104062 2020-11-16T20:22:18.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13104062 '!=' null 2020-11-16T20:22:18.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:18.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:18.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:19.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:19.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:19.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:19.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:19.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:19.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:19.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:19.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:19.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:22:19.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:19.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:19.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:19.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T20:22:19.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:22:19.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:22:19.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:22:19.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:22:19.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:22:19.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:22:19.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:22:19.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:22:20.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724352 2020-11-16T20:22:20.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724352 '!=' null 2020-11-16T20:22:20.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:22:20.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:20.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:20.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:20.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:20.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:20.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:20.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:20.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:20.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:20.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:20.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:22:20.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:20.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:20.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:21.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 2 2020-11-16T20:22:21.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 0 2020-11-16T20:22:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:22:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=1 2020-11-16T20:22:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:22:21.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:22:21.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:22:21.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:22:21.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:22:21.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:22:21.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:22:21.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:22:21.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:22:21.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:22:21.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:22:21.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:22:21.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:22:21.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:22:22.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2020-11-16T20:22:22.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:22:22.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:22:22.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:502: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2020-11-16T20:22:22.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:506: TEST_backfill_pool_priority: echo TEST PASSED 2020-11-16T20:22:22.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: delete_pool test1 2020-11-16T20:22:22.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2020-11-16T20:22:22.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-11-16T20:22:22.233 INFO:tasks.workunit.client.0.smithi189.stdout:TEST PASSED 2020-11-16T20:22:22.749 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' does not exist 2020-11-16T20:22:22.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:510: TEST_backfill_pool_priority: delete_pool test2 2020-11-16T20:22:22.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2020-11-16T20:22:22.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-11-16T20:22:23.310 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' does not exist 2020-11-16T20:22:23.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:511: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2020-11-16T20:22:23.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:22:23.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:22:23.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:22:23.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:22:23.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:22:23.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:22:23.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: return 0 2020-11-16T20:22:23.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-11-16T20:22:23.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-11-16T20:22:23.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:22:23.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-11-16T20:22:23.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:22:23.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:22:23.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:22:23.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:22:23.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:22:23.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:22:23.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:22:23.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:22:23.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:22:23.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:22:23.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:22:23.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:22:23.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:22:23.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:22:23.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:22:23.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:22:23.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:22:23.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:22:23.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-11-16T20:22:23.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:22:23.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:23.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.65265 2020-11-16T20:22:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:22:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:22:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-11-16T20:22:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-11-16T20:22:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-11-16T20:22:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-11-16T20:22:23.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-11-16T20:22:23.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:22:23.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-11-16T20:22:23.679 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:22:23.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:22:23.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:22:23.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:22:23.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:22:23.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:22:23.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:22:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:22:23.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:22:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:22:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:22:23.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:22:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:22:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:22:23.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:22:23.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:22:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:22:23.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:22:23.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-11-16T20:22:23.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:22:23.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:23.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:23.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.65265 2020-11-16T20:22:23.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:22:23.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:22:23.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-11-16T20:22:23.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:22:23.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:23.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:23.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.65265 2020-11-16T20:22:23.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:22:23.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T20:22:23.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:22:23.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-11-16T20:22:23.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_priority td/osd-backfill-prio 2020-11-16T20:22:23.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:46: TEST_backfill_priority: local dir=td/osd-backfill-prio 2020-11-16T20:22:23.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:47: TEST_backfill_priority: local pools=10 2020-11-16T20:22:23.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:48: TEST_backfill_priority: local OSDS=5 2020-11-16T20:22:23.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: expr 150 + 1 2020-11-16T20:22:23.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: local degraded_prio=151 2020-11-16T20:22:23.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:51: TEST_backfill_priority: local max_tries=10 2020-11-16T20:22:23.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:53: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2020-11-16T20:22:23.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-11-16T20:22:23.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:22:23.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:22:23.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:22:23.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-11-16T20:22:23.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-11-16T20:22:24.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:22:24.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:22:24.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:22:24.011 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:22:24.011 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:24.012 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:24.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:24.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:22:24.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:22:24.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:22:24.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:22:24.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:22:24.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:22:24.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:22:24.211 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:22:24.211 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:22:24.211 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:22:24.212 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:22:24.212 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:24.212 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:24.212 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-mon.a.asok 2020-11-16T20:22:24.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:22:24.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.65265/ceph-mon.a.asok config get fsid 2020-11-16T20:22:24.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:22:24.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:22:24.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:22:24.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:22:24.317 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:22:24.317 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:22:24.317 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:22:24.317 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:22:24.318 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:22:24.318 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:24.318 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:24.318 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-mon.a.asok 2020-11-16T20:22:24.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:22:24.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.65265/ceph-mon.a.asok config get mon_host 2020-11-16T20:22:24.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:54: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2020-11-16T20:22:24.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-prio 2020-11-16T20:22:24.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:22:24.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:22:24.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:22:24.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-prio/x 2020-11-16T20:22:24.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:22:24.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:22:24.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:22:24.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:22:24.996 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:22:24.996 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:24.996 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:24.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:24.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:22:24.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:22:25.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:55: TEST_backfill_priority: export CEPH_ARGS 2020-11-16T20:22:25.026 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: expr 5 - 1 2020-11-16T20:22:25.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: seq 0 4 2020-11-16T20:22:25.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:22:25.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2020-11-16T20:22:25.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2020-11-16T20:22:25.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:22:25.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:22:25.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:22:25.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/0 2020-11-16T20:22:25.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T20:22:25.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:22:25.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:22:25.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:22:25.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-11-16T20:22:25.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-11-16T20:22:25.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:22:25.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:22:25.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-11-16T20:22:25.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:22:25.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:22:25.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:22:25.034 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:22:25.034 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:25.034 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:25.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:25.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:25.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:22:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:22:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:22:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-11-16T20:22:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-11-16T20:22:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:22:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:22:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:22:25.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:22:25.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:22:25.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/0 2020-11-16T20:22:25.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:22:25.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b70d4b58-0a9f-458c-b7ab-17ead01bfe75 2020-11-16T20:22:25.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 b70d4b58-0a9f-458c-b7ab-17ead01bfe75' 2020-11-16T20:22:25.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:22:25.037 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 b70d4b58-0a9f-458c-b7ab-17ead01bfe75 2020-11-16T20:22:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCB37JfPUuaAxAAQOJ2YYHIU7d8ButjpDz4Hg== 2020-11-16T20:22:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCB37JfPUuaAxAAQOJ2YYHIU7d8ButjpDz4Hg=="}' 2020-11-16T20:22:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b70d4b58-0a9f-458c-b7ab-17ead01bfe75 -i td/osd-backfill-prio/0/new.json 2020-11-16T20:22:25.444 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:22:25.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/0/new.json 2020-11-16T20:22:25.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCB37JfPUuaAxAAQOJ2YYHIU7d8ButjpDz4Hg== --osd-uuid b70d4b58-0a9f-458c-b7ab-17ead01bfe75 2020-11-16T20:22:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:25.491+0000 7fb46878fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:25.495+0000 7fb46878fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:25.495+0000 7fb46878fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:25.495+0000 7fb46878fd80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-11-16T20:22:25.503 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:25.499+0000 7fb46878fd80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-11-16T20:22:25.503 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:25.499+0000 7fb46878fd80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-11-16T20:22:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:25.543+0000 7fb46878fd80 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-11-16T20:22:29.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-11-16T20:22:29.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:22:29.944 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:22:29.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:22:29.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:22:30.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:22:30.433 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:22:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:22:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:22:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:22:30.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:22:30.466 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:30.459+0000 7f681d7ddd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:30.468 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:30.463+0000 7f681d7ddd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:30.469 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:30.463+0000 7f681d7ddd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:22:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:22:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:22:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:22:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:22:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:22:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:22:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:22:30.823 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:22:30.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:22:31.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:31.011+0000 7f681d7ddd80 -1 Falling back to public interface 2020-11-16T20:22:31.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:22:31.911 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:31.907+0000 7f681d7ddd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:22:32.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:22:32.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:22:32.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:22:32.202 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:22:32.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:22:32.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:22:32.623 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/74132,v1:127.0.0.1:6803/74132] [v2:127.0.0.1:6804/74132,v1:127.0.0.1:6805/74132] exists,up b70d4b58-0a9f-458c-b7ab-17ead01bfe75 2020-11-16T20:22:32.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:22:32.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:22:32.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:22:32.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:22:32.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2020-11-16T20:22:32.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2020-11-16T20:22:32.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:22:32.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:22:32.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:22:32.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/1 2020-11-16T20:22:32.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T20:22:32.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:22:32.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:22:32.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:22:32.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-11-16T20:22:32.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-11-16T20:22:32.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:22:32.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:22:32.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-11-16T20:22:32.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:22:32.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:22:32.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:22:32.628 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:22:32.629 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:32.629 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:32.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:32.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:32.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:22:32.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:22:32.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:22:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-11-16T20:22:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-11-16T20:22:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:22:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:22:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:22:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:22:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:22:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/1 2020-11-16T20:22:32.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:22:32.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c388a43c-d91e-45ed-9e66-175ca012d304 2020-11-16T20:22:32.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 c388a43c-d91e-45ed-9e66-175ca012d304' 2020-11-16T20:22:32.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:22:32.632 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 c388a43c-d91e-45ed-9e66-175ca012d304 2020-11-16T20:22:32.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCI37JfI0OmJhAA9K+xJHNxWO0yqwwOYb+G5Q== 2020-11-16T20:22:32.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCI37JfI0OmJhAA9K+xJHNxWO0yqwwOYb+G5Q=="}' 2020-11-16T20:22:32.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c388a43c-d91e-45ed-9e66-175ca012d304 -i td/osd-backfill-prio/1/new.json 2020-11-16T20:22:33.194 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:22:33.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/1/new.json 2020-11-16T20:22:33.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCI37JfI0OmJhAA9K+xJHNxWO0yqwwOYb+G5Q== --osd-uuid c388a43c-d91e-45ed-9e66-175ca012d304 2020-11-16T20:22:33.241 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:33.235+0000 7fa412cb1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:33.261 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:33.255+0000 7fa412cb1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:33.259+0000 7fa412cb1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:33.259+0000 7fa412cb1d80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-11-16T20:22:33.264 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:33.259+0000 7fa412cb1d80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-11-16T20:22:33.265 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:33.259+0000 7fa412cb1d80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-11-16T20:22:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:33.355+0000 7fa412cb1d80 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-11-16T20:22:38.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-11-16T20:22:38.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:22:38.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:22:38.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:22:38.026 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:22:38.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:22:38.506 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:22:38.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:22:38.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:22:38.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:22:38.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:22:38.538 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:38.531+0000 7faa7d13bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:38.539 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:38.535+0000 7faa7d13bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:38.541 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:38.535+0000 7faa7d13bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:38.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:22:38.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:22:38.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:22:38.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:22:38.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:22:38.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:22:38.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:22:38.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:22:38.908 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:22:38.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:22:39.084 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:39.079+0000 7faa7d13bd80 -1 Falling back to public interface 2020-11-16T20:22:39.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:22:39.913 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:39.903+0000 7faa7d13bd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:22:40.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:22:40.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:22:40.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:22:40.342 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:22:40.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:22:40.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:22:40.741 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/75126,v1:127.0.0.1:6811/75126] [v2:127.0.0.1:6812/75126,v1:127.0.0.1:6813/75126] exists,up c388a43c-d91e-45ed-9e66-175ca012d304 2020-11-16T20:22:40.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:22:40.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:22:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:22:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:22:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2020-11-16T20:22:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2020-11-16T20:22:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:22:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:22:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:22:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/2 2020-11-16T20:22:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T20:22:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:22:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:22:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:22:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2020-11-16T20:22:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2020-11-16T20:22:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:22:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:22:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-11-16T20:22:40.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:22:40.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:22:40.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:22:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:22:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:40.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:22:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:22:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:22:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-11-16T20:22:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-11-16T20:22:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:22:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:22:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:22:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:22:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:22:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/2 2020-11-16T20:22:40.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:22:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7a658b1e-394b-4aae-be8f-8a29bbdd002f 2020-11-16T20:22:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 7a658b1e-394b-4aae-be8f-8a29bbdd002f' 2020-11-16T20:22:40.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:22:40.749 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 7a658b1e-394b-4aae-be8f-8a29bbdd002f 2020-11-16T20:22:40.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCQ37JfHSBpLRAAwVLh6f3GwX/e2CKaoX+05A== 2020-11-16T20:22:40.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCQ37JfHSBpLRAAwVLh6f3GwX/e2CKaoX+05A=="}' 2020-11-16T20:22:40.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7a658b1e-394b-4aae-be8f-8a29bbdd002f -i td/osd-backfill-prio/2/new.json 2020-11-16T20:22:41.323 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:22:41.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/2/new.json 2020-11-16T20:22:41.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCQ37JfHSBpLRAAwVLh6f3GwX/e2CKaoX+05A== --osd-uuid 7a658b1e-394b-4aae-be8f-8a29bbdd002f 2020-11-16T20:22:41.369 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:41.363+0000 7f2c50a14d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:41.374 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:41.367+0000 7f2c50a14d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:41.375 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:41.371+0000 7f2c50a14d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:41.376 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:41.371+0000 7f2c50a14d80 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-11-16T20:22:41.376 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:41.371+0000 7f2c50a14d80 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-11-16T20:22:41.376 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:41.371+0000 7f2c50a14d80 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-11-16T20:22:41.441 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:41.435+0000 7f2c50a14d80 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2020-11-16T20:22:46.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2020-11-16T20:22:46.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:22:46.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:22:46.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:22:46.128 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:22:46.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:22:46.644 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:22:46.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:22:46.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:22:46.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:22:46.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:22:46.678 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:46.671+0000 7f3261a14d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:46.681 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:46.675+0000 7f3261a14d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:46.682 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:46.675+0000 7f3261a14d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:47.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:22:47.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:22:47.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:22:47.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:22:47.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:22:47.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:22:47.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:22:47.053 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:22:47.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:22:47.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:22:47.232 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:47.227+0000 7f3261a14d80 -1 Falling back to public interface 2020-11-16T20:22:47.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:22:48.098 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:48.091+0000 7f3261a14d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:22:48.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:22:48.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:22:48.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:22:48.465 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:22:48.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:22:48.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:22:48.843 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/76120,v1:127.0.0.1:6819/76120] [v2:127.0.0.1:6820/76120,v1:127.0.0.1:6821/76120] exists,up 7a658b1e-394b-4aae-be8f-8a29bbdd002f 2020-11-16T20:22:48.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:22:48.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:22:48.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:22:48.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:22:48.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2020-11-16T20:22:48.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2020-11-16T20:22:48.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:22:48.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T20:22:48.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:22:48.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/3 2020-11-16T20:22:48.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T20:22:48.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:22:48.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:22:48.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:22:48.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2020-11-16T20:22:48.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2020-11-16T20:22:48.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:22:48.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:22:48.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-11-16T20:22:48.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:22:48.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:22:48.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:22:48.847 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:22:48.848 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:48.848 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:48.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:48.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:48.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:22:48.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:22:48.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:22:48.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-11-16T20:22:48.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-11-16T20:22:48.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:22:48.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:22:48.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:22:48.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:22:48.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:22:48.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/3 2020-11-16T20:22:48.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:22:48.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d7d9a084-d0b0-472c-9307-100a95bea890 2020-11-16T20:22:48.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 d7d9a084-d0b0-472c-9307-100a95bea890' 2020-11-16T20:22:48.851 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 d7d9a084-d0b0-472c-9307-100a95bea890 2020-11-16T20:22:48.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:22:48.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCY37JfM2yiMxAAwnj5xa49g2wopyu51kuOMg== 2020-11-16T20:22:48.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCY37JfM2yiMxAAwnj5xa49g2wopyu51kuOMg=="}' 2020-11-16T20:22:48.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d7d9a084-d0b0-472c-9307-100a95bea890 -i td/osd-backfill-prio/3/new.json 2020-11-16T20:22:49.482 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T20:22:49.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/3/new.json 2020-11-16T20:22:49.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCY37JfM2yiMxAAwnj5xa49g2wopyu51kuOMg== --osd-uuid d7d9a084-d0b0-472c-9307-100a95bea890 2020-11-16T20:22:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:49.523+0000 7fea57fa8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:49.539 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:49.531+0000 7fea57fa8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:49.540 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:49.535+0000 7fea57fa8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:49.541 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:49.535+0000 7fea57fa8d80 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-11-16T20:22:49.541 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:49.535+0000 7fea57fa8d80 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-11-16T20:22:49.542 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:49.535+0000 7fea57fa8d80 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-11-16T20:22:49.605 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:49.599+0000 7fea57fa8d80 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2020-11-16T20:22:54.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2020-11-16T20:22:54.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:22:54.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T20:22:54.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:22:54.282 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T20:22:54.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T20:22:54.793 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T20:22:54.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:22:54.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:22:54.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:22:54.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:22:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:54.819+0000 7fd2432aed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:54.819+0000 7fd2432aed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:54.819+0000 7fd2432aed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:55.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T20:22:55.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:22:55.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T20:22:55.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:22:55.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:22:55.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:22:55.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:22:55.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:22:55.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:22:55.176 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:22:55.372 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:55.367+0000 7fd2432aed80 -1 Falling back to public interface 2020-11-16T20:22:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:22:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:56.323+0000 7fd2432aed80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T20:22:56.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:22:56.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:22:56.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:22:56.549 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:22:56.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:22:56.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:22:56.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:22:57.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:22:57.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:22:57.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:22:57.993 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:22:57.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:22:57.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:22:58.393 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/77114,v1:127.0.0.1:6827/77114] [v2:127.0.0.1:6828/77114,v1:127.0.0.1:6829/77114] exists,up d7d9a084-d0b0-472c-9307-100a95bea890 2020-11-16T20:22:58.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:22:58.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:22:58.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:22:58.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:22:58.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2020-11-16T20:22:58.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2020-11-16T20:22:58.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:22:58.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T20:22:58.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:22:58.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/4 2020-11-16T20:22:58.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T20:22:58.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:22:58.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:22:58.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:22:58.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2020-11-16T20:22:58.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2020-11-16T20:22:58.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:22:58.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:22:58.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-11-16T20:22:58.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:22:58.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:22:58.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:22:58.399 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:22:58.399 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:22:58.399 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:22:58.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:58.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' 2020-11-16T20:22:58.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:22:58.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:22:58.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:22:58.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-11-16T20:22:58.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-11-16T20:22:58.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:22:58.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:22:58.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:22:58.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:22:58.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:22:58.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/4 2020-11-16T20:22:58.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:22:58.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b0a07b89-4e5f-46a3-a985-2bce9e59537c 2020-11-16T20:22:58.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 b0a07b89-4e5f-46a3-a985-2bce9e59537c' 2020-11-16T20:22:58.402 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 b0a07b89-4e5f-46a3-a985-2bce9e59537c 2020-11-16T20:22:58.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:22:58.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCi37JfjpKXGBAAmSAN4TN9rth1tq5/ycQDaA== 2020-11-16T20:22:58.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCi37JfjpKXGBAAmSAN4TN9rth1tq5/ycQDaA=="}' 2020-11-16T20:22:58.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b0a07b89-4e5f-46a3-a985-2bce9e59537c -i td/osd-backfill-prio/4/new.json 2020-11-16T20:22:58.952 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T20:22:58.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/4/new.json 2020-11-16T20:22:58.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCi37JfjpKXGBAAmSAN4TN9rth1tq5/ycQDaA== --osd-uuid b0a07b89-4e5f-46a3-a985-2bce9e59537c 2020-11-16T20:22:59.000 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:58.995+0000 7f050f1c4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:59.002 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:58.995+0000 7f050f1c4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:59.004 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:58.999+0000 7f050f1c4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:22:59.005 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:58.999+0000 7f050f1c4d80 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-11-16T20:22:59.005 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:58.999+0000 7f050f1c4d80 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-11-16T20:22:59.006 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:58.999+0000 7f050f1c4d80 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-11-16T20:22:59.062 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:22:59.055+0000 7f050f1c4d80 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2020-11-16T20:23:03.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2020-11-16T20:23:03.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:23:03.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T20:23:03.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:23:03.826 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T20:23:04.316 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T20:23:04.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T20:23:04.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=cb08b89e-be31-4c34-a9b2-2d9130bf8956 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.65265/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:23:04.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:23:04.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:23:04.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:23:04.349 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:23:04.342+0000 7f066894cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:23:04.352 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:23:04.346+0000 7f066894cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:23:04.355 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:23:04.350+0000 7f066894cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:23:04.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T20:23:04.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:23:04.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T20:23:04.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:23:04.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:23:04.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:23:04.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:23:04.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:23:04.705 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:23:04.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:23:04.897 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:23:04.890+0000 7f066894cd80 -1 Falling back to public interface 2020-11-16T20:23:05.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:23:05.766 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:23:05.754+0000 7f066894cd80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T20:23:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:23:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:23:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:23:06.103 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:23:06.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:23:06.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:23:06.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:23:07.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:23:07.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:23:07.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:23:07.503 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:23:07.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:23:07.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:23:07.896 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/78134,v1:127.0.0.1:6835/78134] [v2:127.0.0.1:6836/78134,v1:127.0.0.1:6837/78134] exists,up b0a07b89-4e5f-46a3-a985-2bce9e59537c 2020-11-16T20:23:07.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:23:07.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:23:07.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:23:07.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: seq 1 10 2020-11-16T20:23:07.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:07.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test1 1 1 2020-11-16T20:23:07.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2020-11-16T20:23:08.445 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' already exists 2020-11-16T20:23:08.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:23:09.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test1 size 2 2020-11-16T20:23:10.243 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T20:23:10.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:10.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test2 1 1 2020-11-16T20:23:10.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2020-11-16T20:23:10.794 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' already exists 2020-11-16T20:23:10.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:23:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test2 size 2 2020-11-16T20:23:12.681 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 2 2020-11-16T20:23:12.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:12.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test3 1 1 2020-11-16T20:23:12.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2020-11-16T20:23:13.308 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' already exists 2020-11-16T20:23:13.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:23:14.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-11-16T20:23:15.125 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 2 2020-11-16T20:23:15.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:15.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test4 1 1 2020-11-16T20:23:15.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test4 1 1 2020-11-16T20:23:15.807 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test4' already exists 2020-11-16T20:23:15.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:23:16.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test4 size 2 2020-11-16T20:23:17.744 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 2 2020-11-16T20:23:17.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:17.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test5 1 1 2020-11-16T20:23:17.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test5 1 1 2020-11-16T20:23:18.331 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test5' already exists 2020-11-16T20:23:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:23:19.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test5 size 2 2020-11-16T20:23:20.616 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 5 size to 2 2020-11-16T20:23:20.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:20.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test6 1 1 2020-11-16T20:23:20.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test6 1 1 2020-11-16T20:23:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test6' already exists 2020-11-16T20:23:21.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:23:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test6 size 2 2020-11-16T20:23:23.006 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 6 size to 2 2020-11-16T20:23:23.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:23.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test7 1 1 2020-11-16T20:23:23.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test7 1 1 2020-11-16T20:23:23.617 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test7' already exists 2020-11-16T20:23:23.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:23:24.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-11-16T20:23:25.646 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 7 size to 2 2020-11-16T20:23:25.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:25.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test8 1 1 2020-11-16T20:23:25.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test8 1 1 2020-11-16T20:23:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test8' already exists 2020-11-16T20:23:26.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:23:27.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test8 size 2 2020-11-16T20:23:28.115 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 8 size to 2 2020-11-16T20:23:28.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:28.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test9 1 1 2020-11-16T20:23:28.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test9 1 1 2020-11-16T20:23:29.036 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test9' already exists 2020-11-16T20:23:29.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:23:30.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test9 size 2 2020-11-16T20:23:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 9 size to 2 2020-11-16T20:23:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test10 1 1 2020-11-16T20:23:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test10 1 1 2020-11-16T20:23:31.404 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test10' already exists 2020-11-16T20:23:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:23:32.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test10 size 2 2020-11-16T20:23:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 10 size to 2 2020-11-16T20:23:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:67: TEST_backfill_priority: sleep 5 2020-11-16T20:23:38.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:69: TEST_backfill_priority: wait_for_clean 2020-11-16T20:23:38.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:23:38.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:23:38.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:23:38.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:23:38.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:23:38.380 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:23:38.380 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:23:38.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:23:38.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:23:38.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:23:38.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:23:38.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:23:38.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:23:38.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:23:38.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:23:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:23:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:23:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:23:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:23:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T20:23:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:23:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:23:39.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:23:39.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836496 2020-11-16T20:23:39.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836496 2020-11-16T20:23:39.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496' 2020-11-16T20:23:39.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:23:39.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:23:39.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705678 2020-11-16T20:23:39.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705678 2020-11-16T20:23:39.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-38654705678' 2020-11-16T20:23:39.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:23:39.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:23:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542158 2020-11-16T20:23:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542158 2020-11-16T20:23:39.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-38654705678 2-60129542158' 2020-11-16T20:23:39.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:23:39.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:23:39.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378635 2020-11-16T20:23:39.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378635 2020-11-16T20:23:39.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-38654705678 2-60129542158 3-81604378635' 2020-11-16T20:23:39.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:23:39.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:23:39.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215113 2020-11-16T20:23:39.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215113 2020-11-16T20:23:39.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-38654705678 2-60129542158 3-81604378635 4-103079215113' 2020-11-16T20:23:39.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:23:39.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836496 2020-11-16T20:23:39.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:23:39.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:23:39.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2020-11-16T20:23:39.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:23:39.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836496 2020-11-16T20:23:39.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2020-11-16T20:23:39.903 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836496 2020-11-16T20:23:39.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:23:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836494 -lt 21474836496 2020-11-16T20:23:40.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:23:41.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:23:41.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:23:41.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836496 2020-11-16T20:23:41.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:23:41.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705678 2020-11-16T20:23:41.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:23:41.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:23:41.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705678 2020-11-16T20:23:41.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:23:41.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705678 2020-11-16T20:23:41.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705678' 2020-11-16T20:23:41.717 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705678 2020-11-16T20:23:41.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:23:42.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705678 -lt 38654705678 2020-11-16T20:23:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:23:42.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542158 2020-11-16T20:23:42.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:23:42.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:23:42.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542158 2020-11-16T20:23:42.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:23:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542158 2020-11-16T20:23:42.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542158' 2020-11-16T20:23:42.115 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542158 2020-11-16T20:23:42.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:23:42.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542158 -lt 60129542158 2020-11-16T20:23:42.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:23:42.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378635 2020-11-16T20:23:42.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:23:42.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:23:42.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378635 2020-11-16T20:23:42.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:23:42.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378635 2020-11-16T20:23:42.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378635' 2020-11-16T20:23:42.510 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378635 2020-11-16T20:23:42.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:23:42.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378635 -lt 81604378635 2020-11-16T20:23:42.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:23:42.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:23:42.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215113 2020-11-16T20:23:42.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:23:42.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215113 2020-11-16T20:23:42.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:23:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215113 2020-11-16T20:23:42.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215113' 2020-11-16T20:23:42.888 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215113 2020-11-16T20:23:42.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:23:43.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215114 -lt 103079215113 2020-11-16T20:23:43.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:23:43.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:23:43.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:23:43.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 10 == 0 2020-11-16T20:23:43.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:23:43.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:23:43.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:23:43.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:23:43.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:23:43.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:23:43.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:23:44.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=10 2020-11-16T20:23:44.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:23:44.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:23:44.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:23:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 10 = 10 2020-11-16T20:23:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:23:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:23:44.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:71: TEST_backfill_priority: ceph pg dump pgs 2020-11-16T20:23:44.952 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:23:44.953 INFO:tasks.workunit.client.0.smithi189.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:23:34.816905+0000 0'0 79:22 [2,1] 2 [2,1] 2 0'0 2020-11-16T20:23:31.184626+0000 0'0 2020-11-16T20:23:31.184626+0000 0 2020-11-16T20:23:44.953 INFO:tasks.workunit.client.0.smithi189.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:23:31.573272+0000 0'0 78:19 [1,2] 1 [1,2] 1 0'0 2020-11-16T20:23:28.541692+0000 0'0 2020-11-16T20:23:28.541692+0000 0 2020-11-16T20:23:44.953 INFO:tasks.workunit.client.0.smithi189.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:23:30.021099+0000 0'0 78:28 [3,4] 3 [3,4] 3 0'0 2020-11-16T20:23:26.025655+0000 0'0 2020-11-16T20:23:26.025655+0000 0 2020-11-16T20:23:44.953 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:23:25.953522+0000 0'0 78:34 [1,4] 1 [1,4] 1 0'0 2020-11-16T20:23:23.408790+0000 0'0 2020-11-16T20:23:23.408790+0000 0 2020-11-16T20:23:44.953 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:23:12.913829+0000 0'0 78:51 [3,1] 3 [3,1] 3 0'0 2020-11-16T20:23:10.563803+0000 0'0 2020-11-16T20:23:10.563803+0000 0 2020-11-16T20:23:44.954 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:23:10.956391+0000 0'0 78:55 [1,0] 1 [1,0] 1 0'0 2020-11-16T20:23:08.268659+0000 0'0 2020-11-16T20:23:08.268659+0000 0 2020-11-16T20:23:44.954 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:23:15.669714+0000 0'0 78:48 [1,2] 1 [1,2] 1 0'0 2020-11-16T20:23:13.066896+0000 0'0 2020-11-16T20:23:13.066896+0000 0 2020-11-16T20:23:44.954 INFO:tasks.workunit.client.0.smithi189.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:23:19.822030+0000 0'0 78:40 [4,0] 4 [4,0] 4 0'0 2020-11-16T20:23:15.523867+0000 0'0 2020-11-16T20:23:15.523867+0000 0 2020-11-16T20:23:44.954 INFO:tasks.workunit.client.0.smithi189.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:23:21.610029+0000 0'0 78:41 [3,4] 3 [3,4] 3 0'0 2020-11-16T20:23:18.120805+0000 0'0 2020-11-16T20:23:18.120805+0000 0 2020-11-16T20:23:44.954 INFO:tasks.workunit.client.0.smithi189.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:23:25.475469+0000 0'0 78:26 [4,2] 4 [4,2] 4 0'0 2020-11-16T20:23:21.049329+0000 0'0 2020-11-16T20:23:21.049329+0000 0 2020-11-16T20:23:44.954 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:23:44.954 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:23:44.955 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:23:44.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:75: TEST_backfill_priority: local PG1 2020-11-16T20:23:44.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:76: TEST_backfill_priority: local POOLNUM1 2020-11-16T20:23:44.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:77: TEST_backfill_priority: local pool1 2020-11-16T20:23:44.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:78: TEST_backfill_priority: local chk_osd1_1 2020-11-16T20:23:44.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:79: TEST_backfill_priority: local chk_osd1_2 2020-11-16T20:23:44.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:81: TEST_backfill_priority: local PG2 2020-11-16T20:23:44.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:82: TEST_backfill_priority: local POOLNUM2 2020-11-16T20:23:44.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:83: TEST_backfill_priority: local pool2 2020-11-16T20:23:44.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:84: TEST_backfill_priority: local chk_osd2 2020-11-16T20:23:44.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:86: TEST_backfill_priority: local PG3 2020-11-16T20:23:44.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:87: TEST_backfill_priority: local POOLNUM3 2020-11-16T20:23:44.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:88: TEST_backfill_priority: local pool3 2020-11-16T20:23:44.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: seq 1 10 2020-11-16T20:23:44.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:44.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 1.0 --format=json 2020-11-16T20:23:44.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-11-16T20:23:45.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-11-16T20:23:45.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-11-16T20:23:45.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-11-16T20:23:45.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-11-16T20:23:45.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z '' ']' 2020-11-16T20:23:45.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:97: TEST_backfill_priority: PG1=1.0 2020-11-16T20:23:45.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:98: TEST_backfill_priority: POOLNUM1=1 2020-11-16T20:23:45.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:99: TEST_backfill_priority: pool1=test1 2020-11-16T20:23:45.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:100: TEST_backfill_priority: chk_osd1_1=1 2020-11-16T20:23:45.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:101: TEST_backfill_priority: chk_osd1_2=0 2020-11-16T20:23:45.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:45.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 2.0 --format=json 2020-11-16T20:23:45.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-11-16T20:23:45.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-11-16T20:23:45.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-11-16T20:23:45.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-11-16T20:23:45.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=1 2020-11-16T20:23:45.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-11-16T20:23:45.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2020-11-16T20:23:45.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2020-11-16T20:23:45.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:45.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 3.0 --format=json 2020-11-16T20:23:45.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-11-16T20:23:46.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-11-16T20:23:46.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-11-16T20:23:46.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-11-16T20:23:46.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-11-16T20:23:46.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-11-16T20:23:46.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2020-11-16T20:23:46.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:104: TEST_backfill_priority: PG2=3.0 2020-11-16T20:23:46.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:105: TEST_backfill_priority: POOLNUM2=3 2020-11-16T20:23:46.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:106: TEST_backfill_priority: pool2=test3 2020-11-16T20:23:46.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:107: TEST_backfill_priority: chk_osd2=2 2020-11-16T20:23:46.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:46.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 4.0 --format=json 2020-11-16T20:23:46.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-11-16T20:23:46.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-11-16T20:23:46.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-11-16T20:23:46.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-11-16T20:23:46.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-11-16T20:23:46.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-11-16T20:23:46.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2020-11-16T20:23:46.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2020-11-16T20:23:46.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:46.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 5.0 --format=json 2020-11-16T20:23:46.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-11-16T20:23:46.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-11-16T20:23:46.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-11-16T20:23:46.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-11-16T20:23:46.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-11-16T20:23:46.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-11-16T20:23:46.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2020-11-16T20:23:46.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-11-16T20:23:46.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:46.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 6.0 --format=json 2020-11-16T20:23:46.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-11-16T20:23:47.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-11-16T20:23:47.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-11-16T20:23:47.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-11-16T20:23:47.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-11-16T20:23:47.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-11-16T20:23:47.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2020-11-16T20:23:47.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2020-11-16T20:23:47.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:47.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 7.0 --format=json 2020-11-16T20:23:47.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-11-16T20:23:47.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-11-16T20:23:47.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-11-16T20:23:47.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-11-16T20:23:47.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-11-16T20:23:47.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-11-16T20:23:47.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2020-11-16T20:23:47.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-11-16T20:23:47.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:110: TEST_backfill_priority: PG3=7.0 2020-11-16T20:23:47.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:111: TEST_backfill_priority: POOLNUM3=7 2020-11-16T20:23:47.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:112: TEST_backfill_priority: pool3=test7 2020-11-16T20:23:47.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:113: TEST_backfill_priority: break 2020-11-16T20:23:47.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:116: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2020-11-16T20:23:47.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:118: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2020-11-16T20:23:47.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: seq 1 10 2020-11-16T20:23:47.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:47.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2020-11-16T20:23:47.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:47.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2020-11-16T20:23:47.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test2 2020-11-16T20:23:47.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2020-11-16T20:23:47.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-11-16T20:23:48.114 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' does not exist 2020-11-16T20:23:48.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:48.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2020-11-16T20:23:48.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:48.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2020-11-16T20:23:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test4 2020-11-16T20:23:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test4 2020-11-16T20:23:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-11-16T20:23:48.695 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test4' does not exist 2020-11-16T20:23:48.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:48.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2020-11-16T20:23:48.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test5 2020-11-16T20:23:48.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test5 2020-11-16T20:23:48.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-11-16T20:23:49.222 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test5' does not exist 2020-11-16T20:23:49.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:49.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2020-11-16T20:23:49.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test6 2020-11-16T20:23:49.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test6 2020-11-16T20:23:49.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-11-16T20:23:49.773 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test6' does not exist 2020-11-16T20:23:49.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:49.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2020-11-16T20:23:49.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:49.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2020-11-16T20:23:49.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test8 2020-11-16T20:23:49.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test8 2020-11-16T20:23:49.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-11-16T20:23:50.333 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test8' does not exist 2020-11-16T20:23:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2020-11-16T20:23:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test9 2020-11-16T20:23:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test9 2020-11-16T20:23:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-11-16T20:23:50.865 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test9' does not exist 2020-11-16T20:23:50.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-11-16T20:23:50.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2020-11-16T20:23:50.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test10 2020-11-16T20:23:50.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test10 2020-11-16T20:23:50.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-11-16T20:23:51.406 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test10' does not exist 2020-11-16T20:23:51.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:132: TEST_backfill_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2020-11-16T20:23:52.232 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 1 2020-11-16T20:23:52.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:133: TEST_backfill_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2020-11-16T20:23:53.086 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 7 size to 1 2020-11-16T20:23:53.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:134: TEST_backfill_priority: wait_for_clean 2020-11-16T20:23:53.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:23:53.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:23:53.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:23:53.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:23:53.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:23:53.102 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:23:53.102 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:23:53.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:23:53.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:23:53.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:23:53.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:23:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:23:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:23:53.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:23:53.301 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:23:53.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:23:53.844 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:23:53.844 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:23:53.845 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:23:53.845 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T20:23:53.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:23:53.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:23:53.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:23:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836501 2020-11-16T20:23:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836501 2020-11-16T20:23:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501' 2020-11-16T20:23:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:23:54.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:23:54.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705683 2020-11-16T20:23:54.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705683 2020-11-16T20:23:54.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-38654705683' 2020-11-16T20:23:54.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:23:54.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:23:54.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542162 2020-11-16T20:23:54.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542162 2020-11-16T20:23:54.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-38654705683 2-60129542162' 2020-11-16T20:23:54.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:23:54.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:23:54.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378640 2020-11-16T20:23:54.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378640 2020-11-16T20:23:54.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-38654705683 2-60129542162 3-81604378640' 2020-11-16T20:23:54.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:23:54.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:23:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215118 2020-11-16T20:23:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215118 2020-11-16T20:23:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-38654705683 2-60129542162 3-81604378640 4-103079215118' 2020-11-16T20:23:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:23:54.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836501 2020-11-16T20:23:54.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:23:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:23:54.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:23:54.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836501 2020-11-16T20:23:54.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836501 2020-11-16T20:23:54.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2020-11-16T20:23:54.745 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836501 2020-11-16T20:23:54.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:23:55.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836501 -lt 21474836501 2020-11-16T20:23:55.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:23:55.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705683 2020-11-16T20:23:55.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:23:55.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:23:55.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705683 2020-11-16T20:23:55.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:23:55.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705683 2020-11-16T20:23:55.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705683' 2020-11-16T20:23:55.142 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705683 2020-11-16T20:23:55.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:23:55.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705683 -lt 38654705683 2020-11-16T20:23:55.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:23:55.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542162 2020-11-16T20:23:55.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:23:55.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:23:55.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:23:55.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542162 2020-11-16T20:23:55.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542162 2020-11-16T20:23:55.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542162' 2020-11-16T20:23:55.542 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542162 2020-11-16T20:23:55.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:23:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542163 -lt 60129542162 2020-11-16T20:23:55.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:23:55.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378640 2020-11-16T20:23:55.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:23:55.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:23:55.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378640 2020-11-16T20:23:55.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:23:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378640 2020-11-16T20:23:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378640' 2020-11-16T20:23:55.939 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378640 2020-11-16T20:23:55.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:23:56.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378640 -lt 81604378640 2020-11-16T20:23:56.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:23:56.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215118 2020-11-16T20:23:56.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:23:56.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:23:56.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215118 2020-11-16T20:23:56.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:23:56.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215118 2020-11-16T20:23:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215118' 2020-11-16T20:23:56.331 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215118 2020-11-16T20:23:56.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:23:56.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215116 -lt 103079215118 2020-11-16T20:23:56.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:23:57.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:23:57.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:23:58.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215118 -lt 103079215118 2020-11-16T20:23:58.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:23:58.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:23:58.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:23:58.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2020-11-16T20:23:58.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:23:58.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:23:58.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:23:58.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:23:58.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:23:58.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:23:58.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:23:58.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2020-11-16T20:23:58.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:23:58.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:23:58.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:23:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2020-11-16T20:23:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:23:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:23:59.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:136: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-11-16T20:23:59.539 INFO:tasks.workunit.client.0.smithi189.stderr:10+0 records in 2020-11-16T20:23:59.539 INFO:tasks.workunit.client.0.smithi189.stderr:10+0 records out 2020-11-16T20:23:59.539 INFO:tasks.workunit.client.0.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0521412 s, 201 MB/s 2020-11-16T20:23:59.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:137: TEST_backfill_priority: p=1 2020-11-16T20:23:59.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-11-16T20:23:59.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-11-16T20:23:59.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:23:59.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-11-16T20:24:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-11-16T20:24:00.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:00.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-11-16T20:24:01.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:01.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-11-16T20:24:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-11-16T20:24:03.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:03.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-11-16T20:24:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:03.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-11-16T20:24:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-11-16T20:24:05.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:05.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-11-16T20:24:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-11-16T20:24:06.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:06.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-11-16T20:24:07.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:07.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-11-16T20:24:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-11-16T20:24:08.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:08.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-11-16T20:24:09.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:09.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-11-16T20:24:10.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:10.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-11-16T20:24:10.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:10.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-11-16T20:24:11.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:11.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-11-16T20:24:12.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:12.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-11-16T20:24:13.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:13.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-11-16T20:24:13.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:13.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-11-16T20:24:14.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:14.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-11-16T20:24:15.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:15.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-11-16T20:24:15.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:15.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-11-16T20:24:18.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:18.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-11-16T20:24:18.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:18.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-11-16T20:24:19.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:19.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-11-16T20:24:20.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:20.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-11-16T20:24:20.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:20.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-11-16T20:24:21.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:21.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-11-16T20:24:22.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:22.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-11-16T20:24:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-11-16T20:24:23.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:23.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-11-16T20:24:24.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:24.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-11-16T20:24:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-11-16T20:24:25.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:25.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-11-16T20:24:26.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:26.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-11-16T20:24:27.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:27.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-11-16T20:24:28.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:28.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-11-16T20:24:29.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:29.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-11-16T20:24:30.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:30.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-11-16T20:24:30.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:30.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-11-16T20:24:31.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:31.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-11-16T20:24:32.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:32.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-11-16T20:24:32.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:32.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-11-16T20:24:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:33.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-11-16T20:24:34.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:34.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-11-16T20:24:36.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:36.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-11-16T20:24:37.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:37.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-11-16T20:24:37.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:37.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-11-16T20:24:38.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 1 + 1 2020-11-16T20:24:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=2 2020-11-16T20:24:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-11-16T20:24:38.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-11-16T20:24:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2020-11-16T20:24:38.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:38.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2020-11-16T20:24:39.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:39.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2020-11-16T20:24:39.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:39.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2020-11-16T20:24:39.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:39.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2020-11-16T20:24:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:40.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2020-11-16T20:24:41.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:41.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2020-11-16T20:24:41.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:41.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2020-11-16T20:24:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:41.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2020-11-16T20:24:42.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:42.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2020-11-16T20:24:42.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:42.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2020-11-16T20:24:43.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:43.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2020-11-16T20:24:43.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:44.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2020-11-16T20:24:44.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:44.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2020-11-16T20:24:44.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:44.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2020-11-16T20:24:44.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:44.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2020-11-16T20:24:45.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:45.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2020-11-16T20:24:45.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:45.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2020-11-16T20:24:45.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:45.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2020-11-16T20:24:46.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:46.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2020-11-16T20:24:47.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:47.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2020-11-16T20:24:47.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:47.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2020-11-16T20:24:48.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:48.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2020-11-16T20:24:48.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:48.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2020-11-16T20:24:48.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:48.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2020-11-16T20:24:49.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:49.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2020-11-16T20:24:49.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:49.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2020-11-16T20:24:50.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:50.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2020-11-16T20:24:50.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:50.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2020-11-16T20:24:50.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:50.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2020-11-16T20:24:51.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:51.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2020-11-16T20:24:51.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:51.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2020-11-16T20:24:51.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:51.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2020-11-16T20:24:52.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:52.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2020-11-16T20:24:53.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:53.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2020-11-16T20:24:53.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:53.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2020-11-16T20:24:53.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:53.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2020-11-16T20:24:54.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:54.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2020-11-16T20:24:54.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:54.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2020-11-16T20:24:54.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:54.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2020-11-16T20:24:55.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:55.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2020-11-16T20:24:56.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:56.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2020-11-16T20:24:56.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:56.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2020-11-16T20:24:56.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:56.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2020-11-16T20:24:57.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:57.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2020-11-16T20:24:58.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:24:58.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2020-11-16T20:25:00.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:00.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2020-11-16T20:25:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:00.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2020-11-16T20:25:01.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:01.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2020-11-16T20:25:01.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:01.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2020-11-16T20:25:02.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 2 + 1 2020-11-16T20:25:02.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=3 2020-11-16T20:25:02.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-11-16T20:25:02.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-11-16T20:25:02.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:02.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2020-11-16T20:25:02.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:02.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2020-11-16T20:25:02.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:02.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2020-11-16T20:25:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:03.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2020-11-16T20:25:03.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:03.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2020-11-16T20:25:04.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:04.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2020-11-16T20:25:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:05.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2020-11-16T20:25:05.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:05.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2020-11-16T20:25:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:05.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2020-11-16T20:25:06.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:06.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2020-11-16T20:25:06.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:06.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2020-11-16T20:25:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2020-11-16T20:25:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:07.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2020-11-16T20:25:07.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:07.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2020-11-16T20:25:08.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:08.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2020-11-16T20:25:08.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:08.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2020-11-16T20:25:08.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:08.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2020-11-16T20:25:09.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:09.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2020-11-16T20:25:10.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:10.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2020-11-16T20:25:11.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:11.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2020-11-16T20:25:11.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:11.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2020-11-16T20:25:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2020-11-16T20:25:12.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:12.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2020-11-16T20:25:12.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:12.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2020-11-16T20:25:12.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:12.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2020-11-16T20:25:13.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:13.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2020-11-16T20:25:14.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:14.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2020-11-16T20:25:14.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:14.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2020-11-16T20:25:14.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2020-11-16T20:25:15.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:15.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2020-11-16T20:25:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2020-11-16T20:25:15.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:15.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2020-11-16T20:25:16.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:16.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2020-11-16T20:25:17.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:17.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2020-11-16T20:25:17.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:17.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2020-11-16T20:25:17.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:17.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2020-11-16T20:25:17.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:17.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2020-11-16T20:25:18.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:18.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2020-11-16T20:25:18.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:18.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2020-11-16T20:25:19.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:19.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2020-11-16T20:25:19.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:19.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2020-11-16T20:25:20.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:20.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2020-11-16T20:25:21.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:21.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2020-11-16T20:25:21.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:21.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2020-11-16T20:25:22.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:22.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2020-11-16T20:25:22.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:22.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2020-11-16T20:25:23.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:23.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2020-11-16T20:25:23.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:23.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2020-11-16T20:25:23.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:23.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2020-11-16T20:25:24.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-11-16T20:25:24.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2020-11-16T20:25:24.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 3 + 1 2020-11-16T20:25:24.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=4 2020-11-16T20:25:24.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: get_not_primary test1 obj1-p1 2020-11-16T20:25:24.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test1 2020-11-16T20:25:24.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1-p1 2020-11-16T20:25:24.739 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test1 obj1-p1 2020-11-16T20:25:24.739 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test1 2020-11-16T20:25:24.739 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1-p1 2020-11-16T20:25:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test1 obj1-p1 2020-11-16T20:25:24.740 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T20:25:25.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T20:25:25.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-11-16T20:25:25.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T20:25:25.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: local otherosd=0 2020-11-16T20:25:25.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:149: TEST_backfill_priority: ceph pg dump pgs 2020-11-16T20:25:26.205 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:25:26.205 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 42 0 0 0 0 434110464 0 0 24 24 active+clean 2020-11-16T20:23:53.661841+0000 91'124 91:173 [1] 1 [1] 1 0'0 2020-11-16T20:23:23.408790+0000 0'0 2020-11-16T20:23:23.408790+0000 0 2020-11-16T20:25:26.205 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-11-16T20:23:10.956391+0000 91'150 91:213 [1,0] 1 [1,0] 1 0'0 2020-11-16T20:23:08.268659+0000 0'0 2020-11-16T20:23:08.268659+0000 0 2020-11-16T20:25:26.206 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-11-16T20:23:52.566249+0000 91'150 91:213 [1] 1 [1] 1 0'0 2020-11-16T20:23:13.066896+0000 0'0 2020-11-16T20:23:13.066896+0000 0 2020-11-16T20:25:26.206 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:25:26.206 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:25:26.207 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:25:26.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:150: TEST_backfill_priority: ERRORS=0 2020-11-16T20:25:26.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph osd set nobackfill 2020-11-16T20:25:26.891 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T20:25:26.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:153: TEST_backfill_priority: ceph osd set noout 2020-11-16T20:25:27.693 INFO:tasks.workunit.client.0.smithi189.stderr:noout is set 2020-11-16T20:25:27.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:157: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-11-16T20:25:28.512 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 7 size to 2 2020-11-16T20:25:28.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:158: TEST_backfill_priority: sleep 2 2020-11-16T20:25:30.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: get_asok_path osd.1 2020-11-16T20:25:30.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:25:30.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:25:30.528 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:25:30.528 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:25:30.528 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:25:30.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.1.asok 2020-11-16T20:25:30.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: CEPH_ARGS= 2020-11-16T20:25:30.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:25:30.638 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:25:30.638 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:25:30.638 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:25:30.638 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:25:30.638 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:25:30.639 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:25:30.639 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:30.639 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:25:30.639 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:25:30.639 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:25:30.639 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:30.639 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:25:30.640 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:25:30.640 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:25:30.640 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:25:30.640 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:25:30.640 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:25:30.640 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:25:30.640 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:30.640 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:25:30.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: seq 1 10 2020-11-16T20:25:30.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-11-16T20:25:30.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-11-16T20:25:30.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-11-16T20:25:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 1 = 10 ']' 2020-11-16T20:25:31.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-11-16T20:25:33.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-11-16T20:25:33.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-11-16T20:25:33.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-11-16T20:25:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:166: TEST_backfill_priority: break 2020-11-16T20:25:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:174: TEST_backfill_priority: flush_pg_stats 2020-11-16T20:25:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:25:33.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:25:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:25:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:25:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:25:33.830 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:25:33.831 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T20:25:33.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:25:33.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:33.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:25:34.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836523 2020-11-16T20:25:34.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836523 2020-11-16T20:25:34.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836523' 2020-11-16T20:25:34.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:34.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:25:34.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705705 2020-11-16T20:25:34.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705705 2020-11-16T20:25:34.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836523 1-38654705705' 2020-11-16T20:25:34.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:34.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:25:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542184 2020-11-16T20:25:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542184 2020-11-16T20:25:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836523 1-38654705705 2-60129542184' 2020-11-16T20:25:34.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:34.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:25:34.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378662 2020-11-16T20:25:34.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378662 2020-11-16T20:25:34.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836523 1-38654705705 2-60129542184 3-81604378662' 2020-11-16T20:25:34.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:34.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:25:34.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215140 2020-11-16T20:25:34.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215140 2020-11-16T20:25:34.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836523 1-38654705705 2-60129542184 3-81604378662 4-103079215140' 2020-11-16T20:25:34.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:34.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:34.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836523 2020-11-16T20:25:34.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:25:34.682 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836523 2020-11-16T20:25:34.682 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:34.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836523 2020-11-16T20:25:34.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2020-11-16T20:25:34.683 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836523 2020-11-16T20:25:34.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:25:35.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836523 -lt 21474836523 2020-11-16T20:25:35.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:35.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705705 2020-11-16T20:25:35.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:35.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:25:35.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705705 2020-11-16T20:25:35.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:35.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705705 2020-11-16T20:25:35.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705705' 2020-11-16T20:25:35.144 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705705 2020-11-16T20:25:35.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:25:35.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705705 -lt 38654705705 2020-11-16T20:25:35.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:35.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542184 2020-11-16T20:25:35.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:35.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:25:35.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542184 2020-11-16T20:25:35.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:35.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542184 2020-11-16T20:25:35.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542184' 2020-11-16T20:25:35.546 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542184 2020-11-16T20:25:35.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:25:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542185 -lt 60129542184 2020-11-16T20:25:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:35.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378662 2020-11-16T20:25:35.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:25:35.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:35.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378662 2020-11-16T20:25:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378662 2020-11-16T20:25:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378662' 2020-11-16T20:25:35.942 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378662 2020-11-16T20:25:35.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:25:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378662 -lt 81604378662 2020-11-16T20:25:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:36.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215140 2020-11-16T20:25:36.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:36.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:25:36.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215140 2020-11-16T20:25:36.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:36.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215140 2020-11-16T20:25:36.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215140' 2020-11-16T20:25:36.343 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215140 2020-11-16T20:25:36.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:25:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215140 -lt 103079215140 2020-11-16T20:25:36.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: get_asok_path osd.1 2020-11-16T20:25:36.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:25:36.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:25:36.754 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:25:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:25:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:25:36.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.1.asok 2020-11-16T20:25:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: CEPH_ARGS= 2020-11-16T20:25:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:25:36.855 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:25:36.856 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:25:36.856 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:25:36.856 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:25:36.856 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:25:36.856 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:25:36.857 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:36.857 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:25:36.857 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:25:36.857 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:25:36.857 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:36.857 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:25:36.857 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:25:36.857 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:25:36.858 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:25:36.858 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:25:36.858 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:25:36.858 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:25:36.858 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:36.859 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:25:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:177: TEST_backfill_priority: ceph osd out osd.0 2020-11-16T20:25:37.396 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T20:25:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:178: TEST_backfill_priority: sleep 2 2020-11-16T20:25:39.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:179: TEST_backfill_priority: flush_pg_stats 2020-11-16T20:25:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:25:39.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:25:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:25:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:25:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:25:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:25:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T20:25:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:25:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:39.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:25:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836526 2020-11-16T20:25:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836526 2020-11-16T20:25:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836526' 2020-11-16T20:25:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:40.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:25:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705708 2020-11-16T20:25:40.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705708 2020-11-16T20:25:40.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836526 1-38654705708' 2020-11-16T20:25:40.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:40.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:25:40.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542188 2020-11-16T20:25:40.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542188 2020-11-16T20:25:40.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836526 1-38654705708 2-60129542188' 2020-11-16T20:25:40.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:40.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:25:40.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378665 2020-11-16T20:25:40.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378665 2020-11-16T20:25:40.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836526 1-38654705708 2-60129542188 3-81604378665' 2020-11-16T20:25:40.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:40.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:25:40.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215143 2020-11-16T20:25:40.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215143 2020-11-16T20:25:40.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836526 1-38654705708 2-60129542188 3-81604378665 4-103079215143' 2020-11-16T20:25:40.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:40.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836526 2020-11-16T20:25:40.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:40.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:25:40.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836526 2020-11-16T20:25:40.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:40.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836526 2020-11-16T20:25:40.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2020-11-16T20:25:40.699 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836526 2020-11-16T20:25:40.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:25:41.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836526 -lt 21474836526 2020-11-16T20:25:41.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:41.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705708 2020-11-16T20:25:41.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:41.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:25:41.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705708 2020-11-16T20:25:41.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:41.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705708 2020-11-16T20:25:41.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705708' 2020-11-16T20:25:41.092 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705708 2020-11-16T20:25:41.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:25:41.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705708 -lt 38654705708 2020-11-16T20:25:41.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:41.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542188 2020-11-16T20:25:41.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:41.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:25:41.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:41.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542188 2020-11-16T20:25:41.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542188 2020-11-16T20:25:41.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542188' 2020-11-16T20:25:41.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:25:41.491 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542188 2020-11-16T20:25:41.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542188 -lt 60129542188 2020-11-16T20:25:41.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:41.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378665 2020-11-16T20:25:41.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:41.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:25:41.882 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:41.882 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378665 2020-11-16T20:25:41.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378665 2020-11-16T20:25:41.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378665' 2020-11-16T20:25:41.883 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378665 2020-11-16T20:25:41.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:25:42.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378665 -lt 81604378665 2020-11-16T20:25:42.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:42.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:42.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215143 2020-11-16T20:25:42.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:25:42.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215143 2020-11-16T20:25:42.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:42.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215143 2020-11-16T20:25:42.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215143' 2020-11-16T20:25:42.277 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215143 2020-11-16T20:25:42.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:25:42.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215143 -lt 103079215143 2020-11-16T20:25:42.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: get_asok_path osd.1 2020-11-16T20:25:42.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:25:42.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:25:42.666 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:25:42.666 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:25:42.667 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:25:42.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.1.asok 2020-11-16T20:25:42.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: CEPH_ARGS= 2020-11-16T20:25:42.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:25:42.755 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:25:42.755 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:25:42.756 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:25:42.756 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:25:42.756 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:25:42.756 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:42.756 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:25:42.756 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:25:42.756 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:42.757 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:25:42.757 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:25:42.757 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:25:42.757 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:42.757 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:25:42.757 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:42.757 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:25:42.758 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:25:42.758 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:42.758 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:25:42.758 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:25:42.758 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:25:42.758 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:42.758 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:25:42.758 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:25:42.759 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:25:42.759 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:25:42.759 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:25:42.759 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:25:42.759 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:25:42.759 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:42.759 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:25:42.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:181: TEST_backfill_priority: ceph pg dump pgs 2020-11-16T20:25:43.135 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:25:43.135 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-11-16T20:25:33.330869+0000 91'150 103:219 [1,4] 1 [1] 1 0'0 2020-11-16T20:23:23.408790+0000 0'0 2020-11-16T20:23:23.408790+0000 0 2020-11-16T20:25:43.135 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-11-16T20:25:38.825993+0000 91'150 104:231 [1,2] 1 [1,0] 1 0'0 2020-11-16T20:23:08.268659+0000 0'0 2020-11-16T20:23:08.268659+0000 0 2020-11-16T20:25:43.136 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-11-16T20:23:52.566249+0000 91'150 103:223 [1] 1 [1] 1 0'0 2020-11-16T20:23:13.066896+0000 0'0 2020-11-16T20:23:13.066896+0000 0 2020-11-16T20:25:43.136 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:25:43.136 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:25:43.137 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:25:43.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-11-16T20:25:43.855 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 2 2020-11-16T20:25:43.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:184: TEST_backfill_priority: sleep 2 2020-11-16T20:25:45.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:185: TEST_backfill_priority: flush_pg_stats 2020-11-16T20:25:45.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:25:45.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:25:46.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:25:46.283 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:25:46.284 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:25:46.284 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:25:46.284 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T20:25:46.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:25:46.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:46.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:25:46.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836529 2020-11-16T20:25:46.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836529 2020-11-16T20:25:46.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836529' 2020-11-16T20:25:46.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:46.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:25:46.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705712 2020-11-16T20:25:46.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705712 2020-11-16T20:25:46.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836529 1-38654705712' 2020-11-16T20:25:46.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:46.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:25:46.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542191 2020-11-16T20:25:46.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542191 2020-11-16T20:25:46.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836529 1-38654705712 2-60129542191' 2020-11-16T20:25:46.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:46.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:25:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378668 2020-11-16T20:25:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378668 2020-11-16T20:25:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836529 1-38654705712 2-60129542191 3-81604378668' 2020-11-16T20:25:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:46.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:25:47.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215146 2020-11-16T20:25:47.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215146 2020-11-16T20:25:47.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836529 1-38654705712 2-60129542191 3-81604378668 4-103079215146' 2020-11-16T20:25:47.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:47.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836529 2020-11-16T20:25:47.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:47.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:25:47.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836529 2020-11-16T20:25:47.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:47.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836529 2020-11-16T20:25:47.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2020-11-16T20:25:47.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:25:47.108 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836529 2020-11-16T20:25:47.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836529 -lt 21474836529 2020-11-16T20:25:47.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:47.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:47.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705712 2020-11-16T20:25:47.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:25:47.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:47.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705712 2020-11-16T20:25:47.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705712 2020-11-16T20:25:47.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705712' 2020-11-16T20:25:47.496 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705712 2020-11-16T20:25:47.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:25:47.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705712 -lt 38654705712 2020-11-16T20:25:47.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:47.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542191 2020-11-16T20:25:47.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:47.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:25:47.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:47.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542191 2020-11-16T20:25:47.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542191 2020-11-16T20:25:47.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542191' 2020-11-16T20:25:47.903 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542191 2020-11-16T20:25:47.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:25:48.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542189 -lt 60129542191 2020-11-16T20:25:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:25:49.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:25:49.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:25:49.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542191 -lt 60129542191 2020-11-16T20:25:49.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:49.708 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378668 2020-11-16T20:25:49.708 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:49.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:25:49.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378668 2020-11-16T20:25:49.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:49.711 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378668 2020-11-16T20:25:49.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378668 2020-11-16T20:25:49.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378668' 2020-11-16T20:25:49.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:25:50.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378669 -lt 81604378668 2020-11-16T20:25:50.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:50.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:50.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215146 2020-11-16T20:25:50.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:25:50.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215146 2020-11-16T20:25:50.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:50.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215146 2020-11-16T20:25:50.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215146' 2020-11-16T20:25:50.121 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215146 2020-11-16T20:25:50.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:25:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215147 -lt 103079215146 2020-11-16T20:25:50.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: get_asok_path osd.1 2020-11-16T20:25:50.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:25:50.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:25:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:25:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:25:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:25:50.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.1.asok 2020-11-16T20:25:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: CEPH_ARGS= 2020-11-16T20:25:50.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:25:50.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:187: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:25:50.625 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:25:50.625 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:25:50.625 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:25:50.625 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:25:50.625 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:25:50.626 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:50.626 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:25:50.626 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:25:50.626 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:50.626 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:25:50.626 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:25:50.626 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:25:50.627 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:50.627 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:25:50.627 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:25:50.627 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:50.627 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:25:50.627 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:25:50.627 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:50.627 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:25:50.628 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:25:50.628 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:25:50.628 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:50.628 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:25:50.628 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:50.628 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:25:50.628 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:25:50.629 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:50.629 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:25:50.629 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:25:50.629 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:25:50.629 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:50.629 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:25:50.629 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:25:50.629 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:25:50.630 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:25:50.630 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:25:50.630 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:25:50.630 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:25:50.630 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:50.630 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:25:50.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:188: TEST_backfill_priority: ceph pg dump pgs 2020-11-16T20:25:50.987 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:25:50.988 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-11-16T20:25:33.330869+0000 91'150 107:222 [1,4] 1 [1] 1 0'0 2020-11-16T20:23:23.408790+0000 0'0 2020-11-16T20:23:23.408790+0000 0 2020-11-16T20:25:50.988 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-11-16T20:25:38.825993+0000 91'150 107:234 [1,2] 1 [1,0] 1 0'0 2020-11-16T20:23:08.268659+0000 0'0 2020-11-16T20:23:08.268659+0000 0 2020-11-16T20:25:50.988 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfill_wait 2020-11-16T20:25:46.471119+0000 91'150 108:233 [1,2] 1 [1] 1 0'0 2020-11-16T20:23:13.066896+0000 0'0 2020-11-16T20:23:13.066896+0000 0 2020-11-16T20:25:50.988 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:25:50.988 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:25:50.990 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:25:51.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:25:51.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2020-11-16T20:25:51.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: PRIO=110 2020-11-16T20:25:51.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:191: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2020-11-16T20:25:51.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:25:51.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-11-16T20:25:51.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-11-16T20:25:51.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: ITEM=7.0 2020-11-16T20:25:51.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:199: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-11-16T20:25:51.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:25:51.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-11-16T20:25:51.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: PRIO=254 2020-11-16T20:25:51.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:205: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-11-16T20:25:51.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: seq 1 10 2020-11-16T20:25:51.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-11-16T20:25:51.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-11-16T20:25:51.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-11-16T20:25:51.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:216: TEST_backfill_priority: break 2020-11-16T20:25:51.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:224: TEST_backfill_priority: sleep 2 2020-11-16T20:25:53.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: get_asok_path osd.1 2020-11-16T20:25:53.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:25:53.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:25:53.411 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:25:53.412 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:25:53.412 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:25:53.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.1.asok 2020-11-16T20:25:53.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: CEPH_ARGS= 2020-11-16T20:25:53.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:25:53.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:226: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:25:53.533 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:25:53.534 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:25:53.534 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:25:53.534 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:25:53.534 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:25:53.534 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:53.534 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:25:53.535 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:25:53.535 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:53.535 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:25:53.535 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:25:53.535 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:25:53.535 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:53.535 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:25:53.536 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:25:53.536 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:53.536 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 254, 2020-11-16T20:25:53.536 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:25:53.536 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:53.536 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:25:53.536 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:25:53.537 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:25:53.537 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:53.537 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:25:53.537 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:53.537 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:25:53.537 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:25:53.537 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:25:53.537 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:25:53.538 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:25:53.538 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:25:53.538 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:53.538 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:25:53.538 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:25:53.538 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:25:53.538 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:25:53.539 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:25:53.539 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:25:53.539 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:25:53.539 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:25:53.539 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:25:53.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:25:53.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-11-16T20:25:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: PRIO=254 2020-11-16T20:25:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:228: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-11-16T20:25:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:233: TEST_backfill_priority: flush_pg_stats 2020-11-16T20:25:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:25:53.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:25:53.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:25:53.924 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:25:53.925 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:25:53.925 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:25:53.925 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T20:25:53.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:25:53.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:53.926 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:25:54.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836533 2020-11-16T20:25:54.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836533 2020-11-16T20:25:54.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533' 2020-11-16T20:25:54.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:54.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:25:54.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705715 2020-11-16T20:25:54.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705715 2020-11-16T20:25:54.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-38654705715' 2020-11-16T20:25:54.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:54.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:25:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542194 2020-11-16T20:25:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542194 2020-11-16T20:25:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-38654705715 2-60129542194' 2020-11-16T20:25:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:54.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:25:54.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378672 2020-11-16T20:25:54.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378672 2020-11-16T20:25:54.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-38654705715 2-60129542194 3-81604378672' 2020-11-16T20:25:54.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:25:54.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:25:54.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215150 2020-11-16T20:25:54.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215150 2020-11-16T20:25:54.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-38654705715 2-60129542194 3-81604378672 4-103079215150' 2020-11-16T20:25:54.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:54.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836533 2020-11-16T20:25:54.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:54.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:25:54.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836533 2020-11-16T20:25:54.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:54.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836533 2020-11-16T20:25:54.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2020-11-16T20:25:54.761 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836533 2020-11-16T20:25:54.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:25:55.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836533 -lt 21474836533 2020-11-16T20:25:55.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:55.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705715 2020-11-16T20:25:55.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:55.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:25:55.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705715 2020-11-16T20:25:55.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:55.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705715 2020-11-16T20:25:55.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705715' 2020-11-16T20:25:55.174 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705715 2020-11-16T20:25:55.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:25:55.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705715 -lt 38654705715 2020-11-16T20:25:55.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:55.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542194 2020-11-16T20:25:55.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:55.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:25:55.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542194 2020-11-16T20:25:55.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:55.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542194 2020-11-16T20:25:55.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542194' 2020-11-16T20:25:55.651 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542194 2020-11-16T20:25:55.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:25:56.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542195 -lt 60129542194 2020-11-16T20:25:56.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:56.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378672 2020-11-16T20:25:56.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:56.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:25:56.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:56.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378672 2020-11-16T20:25:56.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378672 2020-11-16T20:25:56.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378672' 2020-11-16T20:25:56.055 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378672 2020-11-16T20:25:56.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:25:56.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378672 -lt 81604378672 2020-11-16T20:25:56.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:25:56.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215150 2020-11-16T20:25:56.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:25:56.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:25:56.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215150 2020-11-16T20:25:56.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:25:56.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215150 2020-11-16T20:25:56.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215150' 2020-11-16T20:25:56.448 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215150 2020-11-16T20:25:56.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:25:56.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215148 -lt 103079215150 2020-11-16T20:25:56.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:25:57.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:25:57.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:25:58.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215150 -lt 103079215150 2020-11-16T20:25:58.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:236: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2020-11-16T20:25:58.578 INFO:tasks.workunit.client.0.smithi189.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2020-11-16T20:25:58.578 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T20:25:58.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:237: TEST_backfill_priority: sleep 2 2020-11-16T20:26:00.590 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: get_asok_path osd.1 2020-11-16T20:26:00.590 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:26:00.590 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:26:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:26:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:26:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:26:00.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.1.asok 2020-11-16T20:26:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: CEPH_ARGS= 2020-11-16T20:26:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:00.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:239: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:26:00.705 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:00.705 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:00.706 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:00.706 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:00.706 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:00.706 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:00.706 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:00.706 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:00.706 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:00.707 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:00.707 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:00.707 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:00.707 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:00.707 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:00.707 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:00.707 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:00.708 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:00.708 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:00.708 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:00.708 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:00.708 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:00.708 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:00.708 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:00.708 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:00.709 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:00.709 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:00.709 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:00.709 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:00.709 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:00.709 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:00.709 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:00.710 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:00.710 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:00.710 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:00.710 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:00.710 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:00.710 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:00.710 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:00.711 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:00.711 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:00.711 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:00.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:26:00.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2020-11-16T20:26:00.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: PRIO=151 2020-11-16T20:26:00.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:241: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-11-16T20:26:00.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:26:00.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-11-16T20:26:00.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: eval 'ITEM="3.0"' 2020-11-16T20:26:00.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: ITEM=3.0 2020-11-16T20:26:00.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:248: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2020-11-16T20:26:00.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:26:00.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-11-16T20:26:00.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: PRIO=254 2020-11-16T20:26:00.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:254: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-11-16T20:26:00.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:261: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2020-11-16T20:26:01.073 INFO:tasks.workunit.client.0.smithi189.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2020-11-16T20:26:01.074 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T20:26:01.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:262: TEST_backfill_priority: sleep 5 2020-11-16T20:26:06.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: get_asok_path osd.1 2020-11-16T20:26:06.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:26:06.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:26:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:26:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:26:06.086 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:26:06.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.1.asok 2020-11-16T20:26:06.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: CEPH_ARGS= 2020-11-16T20:26:06.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:06.195 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:06.195 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:06.195 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:06.196 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:06.196 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:06.196 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:06.196 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:06.196 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:06.196 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:06.196 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:06.197 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:06.197 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:06.197 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:06.197 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:06.197 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:06.197 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:06.197 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:06.197 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:06.198 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:06.198 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:06.198 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:06.198 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:06.198 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:06.198 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:06.198 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:06.199 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:06.199 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:06.199 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:06.199 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:06.199 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:06.199 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:06.199 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:06.200 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:06.200 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:06.200 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:06.200 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:06.200 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:06.200 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:06.200 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:06.200 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:06.201 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:06.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:266: TEST_backfill_priority: flush_pg_stats 2020-11-16T20:26:06.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:26:06.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:26:06.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:26:06.588 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:26:06.588 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:26:06.588 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:26:06.588 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T20:26:06.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:26:06.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:26:06.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:26:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836537 2020-11-16T20:26:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836537 2020-11-16T20:26:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836537' 2020-11-16T20:26:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:26:06.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:26:06.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705720 2020-11-16T20:26:06.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705720 2020-11-16T20:26:06.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836537 1-38654705720' 2020-11-16T20:26:06.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:26:06.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:26:07.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542199 2020-11-16T20:26:07.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542199 2020-11-16T20:26:07.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836537 1-38654705720 2-60129542199' 2020-11-16T20:26:07.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:26:07.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:26:07.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378676 2020-11-16T20:26:07.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378676 2020-11-16T20:26:07.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836537 1-38654705720 2-60129542199 3-81604378676' 2020-11-16T20:26:07.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:26:07.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:26:07.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215155 2020-11-16T20:26:07.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215155 2020-11-16T20:26:07.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836537 1-38654705720 2-60129542199 3-81604378676 4-103079215155' 2020-11-16T20:26:07.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:26:07.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836537 2020-11-16T20:26:07.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:26:07.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:26:07.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836537 2020-11-16T20:26:07.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:26:07.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836537 2020-11-16T20:26:07.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2020-11-16T20:26:07.422 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836537 2020-11-16T20:26:07.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:26:07.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836535 -lt 21474836537 2020-11-16T20:26:07.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:26:08.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:26:08.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:26:09.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836538 -lt 21474836537 2020-11-16T20:26:09.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:26:09.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705720 2020-11-16T20:26:09.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:26:09.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:26:09.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705720 2020-11-16T20:26:09.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:26:09.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705720 2020-11-16T20:26:09.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705720' 2020-11-16T20:26:09.225 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705720 2020-11-16T20:26:09.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:26:09.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705720 -lt 38654705720 2020-11-16T20:26:09.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:26:09.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542199 2020-11-16T20:26:09.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:26:09.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:26:09.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542199 2020-11-16T20:26:09.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:26:09.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542199 2020-11-16T20:26:09.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542199' 2020-11-16T20:26:09.622 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542199 2020-11-16T20:26:09.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:26:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542199 -lt 60129542199 2020-11-16T20:26:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:26:10.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378676 2020-11-16T20:26:10.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:26:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:26:10.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378676 2020-11-16T20:26:10.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:26:10.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378676 2020-11-16T20:26:10.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378676' 2020-11-16T20:26:10.024 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378676 2020-11-16T20:26:10.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:26:10.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378677 -lt 81604378676 2020-11-16T20:26:10.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:26:10.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215155 2020-11-16T20:26:10.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:26:10.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:26:10.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215155 2020-11-16T20:26:10.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:26:10.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215155 2020-11-16T20:26:10.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215155' 2020-11-16T20:26:10.421 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215155 2020-11-16T20:26:10.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:26:10.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215155 -lt 103079215155 2020-11-16T20:26:10.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:267: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-11-16T20:26:11.178 INFO:tasks.workunit.client.0.smithi189.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2020-11-16T20:26:11.179 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T20:26:11.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:268: TEST_backfill_priority: sleep 2 2020-11-16T20:26:13.196 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: get_asok_path osd.1 2020-11-16T20:26:13.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:26:13.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:26:13.198 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:26:13.198 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:26:13.198 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:26:13.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.1.asok 2020-11-16T20:26:13.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: CEPH_ARGS= 2020-11-16T20:26:13.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:13.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:271: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:26:17.469 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:17.475 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:17.475 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:17.475 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:17.475 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:17.475 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836541 2020-11-16T20:26:17.475 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705723 2020-11-16T20:26:17.475 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542203 2020-11-16T20:26:17.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:26:17.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-11-16T20:26:17.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: PRIO=151 2020-11-16T20:26:17.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:273: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-11-16T20:26:17.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:26:17.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-11-16T20:26:17.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-11-16T20:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: ITEM=7.0 2020-11-16T20:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:280: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-11-16T20:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-11-16T20:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-11-16T20:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: PRIO=254 2020-11-16T20:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:286: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-11-16T20:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:293: TEST_backfill_priority: ceph osd unset noout 2020-11-16T20:26:17.478 INFO:tasks.workunit.client.0.smithi189.stderr:noout is unset 2020-11-16T20:26:17.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:294: TEST_backfill_priority: ceph osd unset nobackfill 2020-11-16T20:26:17.478 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T20:26:17.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: get_asok_path osd.1 2020-11-16T20:26:17.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:26:17.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:26:17.478 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:26:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:26:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:26:17.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.1.asok 2020-11-16T20:26:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations' 2020-11-16T20:26:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations' 2020-11-16T20:26:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:26:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:26:17.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:26:17.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:26:17.480 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:26:17.480 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:26:17.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:26:17.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:26:17.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:26:17.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:26:17.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:26:17.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:26:17.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:26:17.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:26:17.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:26:17.481 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:26:17.481 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:26:17.482 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:26:17.482 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T20:26:17.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:26:17.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:26:17.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:26:17.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836541 2020-11-16T20:26:17.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836541 2020-11-16T20:26:17.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836541' 2020-11-16T20:26:17.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:26:17.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:26:17.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705723 2020-11-16T20:26:17.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705723 2020-11-16T20:26:17.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836541 1-38654705723' 2020-11-16T20:26:17.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:26:17.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:26:17.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542203 2020-11-16T20:26:17.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542203 2020-11-16T20:26:17.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836541 1-38654705723 2-60129542203' 2020-11-16T20:26:17.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:26:17.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:26:17.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378680 2020-11-16T20:26:17.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378680 2020-11-16T20:26:17.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836541 1-38654705723 2-60129542203 3-81604378680' 2020-11-16T20:26:17.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:26:17.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:26:17.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215158 2020-11-16T20:26:17.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215158 2020-11-16T20:26:17.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836541 1-38654705723 2-60129542203 3-81604378680 4-103079215158' 2020-11-16T20:26:17.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:26:17.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836541 2020-11-16T20:26:17.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:26:17.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:26:17.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836541 2020-11-16T20:26:17.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:26:17.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836541 2020-11-16T20:26:17.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2020-11-16T20:26:17.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:26:17.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836541 -lt 21474836541 2020-11-16T20:26:17.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:26:17.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705723 2020-11-16T20:26:17.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:26:17.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:26:17.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705723 2020-11-16T20:26:17.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:26:17.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705723 2020-11-16T20:26:17.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705723' 2020-11-16T20:26:17.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:26:17.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705724 -lt 38654705723 2020-11-16T20:26:17.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:26:17.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542203 2020-11-16T20:26:17.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:26:17.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:26:17.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542203 2020-11-16T20:26:17.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:26:17.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542203 2020-11-16T20:26:17.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542203' 2020-11-16T20:26:17.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:26:17.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542203 -lt 60129542203 2020-11-16T20:26:17.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:26:17.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378680 2020-11-16T20:26:17.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:26:17.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:26:17.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378680 2020-11-16T20:26:17.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:26:17.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378680 2020-11-16T20:26:17.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378680' 2020-11-16T20:26:17.771 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378680 2020-11-16T20:26:17.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:26:18.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378680 -lt 81604378680 2020-11-16T20:26:18.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:26:18.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215158 2020-11-16T20:26:18.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:26:18.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:26:18.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215158 2020-11-16T20:26:18.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:26:18.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215158 2020-11-16T20:26:18.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215158' 2020-11-16T20:26:18.218 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215158 2020-11-16T20:26:18.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:26:19.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215159 -lt 103079215158 2020-11-16T20:26:19.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:26:19.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:19.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:19.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2020-11-16T20:26:19.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:19.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:19.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:19.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:19.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:19.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:19.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:20.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:20.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:20.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:20.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:26:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:26:20.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:26:20.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:20.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:20.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:20.632 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:20.633 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:20.633 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:20.633 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:20.633 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:20.633 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:20.633 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:20.634 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:20.634 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:20.634 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:20.634 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:20.634 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:20.634 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:20.634 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:20.635 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:20.635 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:20.635 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:20.635 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:20.635 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:20.635 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:20.635 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:20.636 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:20.636 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:20.636 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:20.636 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:20.636 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:20.636 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:20.636 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:20.636 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:20.637 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:20.637 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:20.637 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:20.637 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:20.637 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:20.637 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:20.637 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:20.638 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:20.638 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:20.638 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:20.638 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:20.638 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:20.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:20.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:20.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:20.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:20.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:20.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:20.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:20.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:21.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:21.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:21.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:21.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:21.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:21.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:21.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:21.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:21.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:21.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:21.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:21.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:21.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:22.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3131021 2020-11-16T20:26:22.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3131021 '!=' null 2020-11-16T20:26:22.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:22.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:22.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:22.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:22.461 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:22.461 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:22.461 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:22.461 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:22.461 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:22.462 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:22.462 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:22.462 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:22.462 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:22.462 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:22.462 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:22.463 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:22.463 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:22.463 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:22.463 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:22.463 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:22.463 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:22.463 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:22.463 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:22.464 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:22.464 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:22.464 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:22.464 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:22.464 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:22.464 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:22.464 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:22.465 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:22.465 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:22.465 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:22.465 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:22.465 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:22.465 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:22.465 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:22.465 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:22.466 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:22.466 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:22.466 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:22.466 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:22.467 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:22.467 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:22.467 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:22.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:22.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:22.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:22.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:22.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:22.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:22.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:22.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:22.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:22.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:22.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:22.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:22.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:23.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:23.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:23.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:23.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:23.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:23.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:23.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:23.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:23.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:23.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13999938 2020-11-16T20:26:23.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13999938 '!=' null 2020-11-16T20:26:23.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:23.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:23.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:23.906 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:23.995 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:23.995 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:23.995 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:23.996 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:23.996 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:23.996 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:23.996 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:23.996 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:23.996 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:23.996 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:23.997 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:23.997 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:23.997 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:23.997 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:23.997 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:23.997 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:23.997 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:23.998 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:23.998 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:23.998 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:23.998 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:23.998 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:23.998 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:23.999 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:23.999 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:23.999 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:23.999 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:23.999 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:23.999 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:23.999 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:24.000 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:24.000 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:24.000 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:24.000 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:24.000 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:24.000 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:24.000 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:24.000 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:24.001 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:24.001 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:24.001 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:24.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:24.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:24.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:24.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:24.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:24.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:24.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:24.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:24.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:24.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:24.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:25.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:25.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:25.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:25.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:25.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=11530959 2020-11-16T20:26:25.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 11530959 '!=' null 2020-11-16T20:26:25.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:25.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:25.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:25.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:25.668 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:25.668 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:25.668 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:25.669 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:25.669 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:25.669 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:25.669 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:25.669 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:25.669 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:25.669 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:25.670 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:25.670 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:25.670 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:25.670 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:25.670 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:25.670 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:25.670 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:25.671 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:25.671 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:25.671 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:25.671 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:25.671 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:25.672 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:25.672 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:25.672 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:25.672 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:25.672 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:25.672 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:25.672 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:25.672 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:25.673 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:25.673 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:25.673 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:25.673 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:25.673 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:25.673 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:25.673 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:25.673 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:25.674 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:25.674 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:25.674 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:25.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:25.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:25.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:25.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:25.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:25.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:25.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:25.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:25.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:26.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:26.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:26.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:26.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:26.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:26.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:26.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:26.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:26.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:26.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:26.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:26.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:27.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16676489 2020-11-16T20:26:27.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16676489 '!=' null 2020-11-16T20:26:27.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:27.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:27.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:27.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:27.306 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:27.307 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:27.307 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:27.307 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:27.307 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:27.307 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:27.307 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:27.307 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:27.308 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:27.308 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:27.308 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:27.308 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:27.308 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:27.308 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:27.308 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:27.309 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:27.309 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:27.309 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:27.309 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:27.309 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:27.309 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:27.309 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:27.309 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:27.310 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:27.310 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:27.310 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:27.310 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:27.310 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:27.310 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:27.310 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:27.311 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:27.311 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:27.311 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:27.311 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:27.311 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:27.311 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:27.311 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:27.311 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:27.312 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:27.312 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:27.312 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:27.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:27.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:27.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:27.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:27.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:27.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:27.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:27.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:28.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:28.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:28.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:28.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:28.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:28.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:28.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:28.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:28.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:28.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:28.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:28.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:28.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:29.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850260 2020-11-16T20:26:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850260 '!=' null 2020-11-16T20:26:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:29.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:29.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:29.272 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:29.273 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:29.273 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:29.273 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:29.273 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:29.273 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:29.273 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:29.274 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:29.274 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:29.274 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:29.274 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:29.274 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:29.274 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:29.274 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:29.275 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:29.275 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:29.275 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:29.275 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:29.275 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:29.275 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:29.275 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:29.276 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:29.276 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:29.276 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:29.276 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:29.276 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:29.276 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:29.276 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:29.276 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:29.277 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:29.277 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:29.277 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:29.277 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:29.278 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:29.278 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:29.278 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:29.278 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:29.278 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:29.278 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:29.279 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:29.279 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:29.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:29.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:29.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:29.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:29.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:29.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:29.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:29.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:29.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:29.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:29.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:29.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:29.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:30.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:30.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:30.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:30.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:30.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:30.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:30.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:30.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:30.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:30.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850260 2020-11-16T20:26:30.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850260 '!=' null 2020-11-16T20:26:30.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:30.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:30.778 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:30.778 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:30.874 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:30.874 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:30.875 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:30.875 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:30.875 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:30.875 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:30.875 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:30.875 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:30.875 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:30.876 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:30.876 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:30.876 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:30.876 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:30.876 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:30.876 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:30.876 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:30.877 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:30.877 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:30.877 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:30.877 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:30.877 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:30.877 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:30.877 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:30.878 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:30.878 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:30.878 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:30.878 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:30.878 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:30.878 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:30.878 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:30.878 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:30.879 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:30.879 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:30.879 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:30.879 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:30.879 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:30.879 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:30.879 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:30.880 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:30.880 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:30.880 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:30.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:30.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:30.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:30.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:30.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:30.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:30.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:30.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:30.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:31.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:31.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:31.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:31.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:31.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:31.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:31.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:31.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:31.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:31.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:31.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:31.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:31.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:32.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850628 2020-11-16T20:26:32.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850628 '!=' null 2020-11-16T20:26:32.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:32.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:32.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:32.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:32.419 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:32.419 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:32.419 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:32.419 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:32.419 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:32.420 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:32.420 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:32.420 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:32.420 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:32.420 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:32.420 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:32.420 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:32.420 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:32.421 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:32.421 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:32.421 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:32.421 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:32.421 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:32.421 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:32.421 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:32.422 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:32.422 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:32.422 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:32.422 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:32.422 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:32.422 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:32.422 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:32.423 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:32.423 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:32.423 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:32.423 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:32.423 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:32.423 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:32.423 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:32.424 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:32.424 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:32.424 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:32.424 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:32.424 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:32.424 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:32.424 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:32.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:32.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:32.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:32.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:32.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:32.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:32.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:32.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:32.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:32.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:32.926 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:32.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:32.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:33.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:33.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:33.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:33.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:33.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:33.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:33.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:33.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:33.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:33.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21839653 2020-11-16T20:26:33.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21839653 '!=' null 2020-11-16T20:26:33.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:33.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:33.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:33.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:34.058 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:34.058 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:34.058 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:34.058 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:34.059 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:34.059 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:34.059 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:34.059 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:34.059 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:34.059 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:34.059 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:34.060 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:34.060 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:34.060 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:34.060 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:34.060 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:34.060 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:34.060 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:34.061 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:34.061 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:34.061 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:34.061 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:34.061 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:34.061 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:34.061 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:34.061 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:34.062 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:34.062 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:34.062 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:34.062 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:34.062 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:34.062 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:34.062 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:34.063 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:34.063 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:34.063 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:34.063 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:34.063 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:34.063 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:34.063 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:34.063 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:34.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:34.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:34.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:34.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:34.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:34.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:34.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:34.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:34.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:34.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:34.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:34.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:34.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:35.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:35.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:35.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:35.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:35.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13977200 2020-11-16T20:26:35.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13977200 '!=' null 2020-11-16T20:26:35.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:35.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:35.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:35.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:35.768 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:35.769 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:35.769 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:35.769 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:35.769 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:35.769 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:35.769 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:35.769 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:35.770 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:35.770 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:35.770 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:35.770 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:35.770 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:35.770 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:35.770 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:35.771 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:35.771 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:35.771 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:35.771 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:35.771 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:35.771 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:35.771 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:35.772 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:35.772 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:35.772 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:35.772 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:35.772 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:35.772 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:35.772 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:35.772 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:35.773 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:35.773 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:35.773 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:35.773 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:35.773 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:35.773 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:35.773 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:35.774 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:35.774 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:35.774 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:35.774 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:35.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:35.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:35.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:35.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:35.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:35.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:35.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:35.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:35.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:36.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:36.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:36.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:36.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:36.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:36.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:36.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:36.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21839309 2020-11-16T20:26:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21839309 '!=' null 2020-11-16T20:26:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:37.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:37.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:37.497 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:37.497 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:37.497 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:37.497 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:37.498 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:37.498 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:37.498 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:37.498 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:37.498 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:37.498 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:37.498 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:37.499 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:37.499 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:37.499 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:37.499 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:37.499 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:37.499 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:37.499 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:37.500 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:37.500 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:37.500 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:37.500 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:37.500 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:37.500 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:37.500 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:37.500 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:37.501 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:37.501 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:37.501 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:37.501 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:37.501 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:37.501 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:37.501 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:37.502 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:37.502 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:37.502 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:37.502 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:37.502 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:37.502 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:37.502 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:37.502 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:37.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:37.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:37.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:37.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:37.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:37.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:37.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:37.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:37.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:38.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:38.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:38.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:38.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:38.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:38.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:38.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:38.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:38.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:38.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:38.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:38.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:38.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:39.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850907 2020-11-16T20:26:39.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850907 '!=' null 2020-11-16T20:26:39.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:39.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:39.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:39.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:39.106 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:39.106 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:39.107 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:39.107 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:39.107 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:39.107 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:39.107 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:39.107 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:39.108 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:39.108 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:39.108 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:39.108 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:39.108 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:39.108 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:39.108 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:39.108 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:39.109 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:39.109 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:39.109 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:39.109 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:39.109 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:39.109 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:39.109 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:39.110 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:39.110 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:39.110 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:39.110 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:39.110 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:39.111 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:39.111 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:39.111 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:39.111 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:39.112 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:39.112 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:39.112 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:39.112 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:39.113 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:39.113 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:39.113 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:39.113 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:39.113 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:39.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:39.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:39.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:39.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:39.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:39.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:39.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:39.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:39.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:39.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:39.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:39.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:39.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:40.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:40.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:40.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:40.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:40.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:40.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:40.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:40.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:40.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:40.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850907 2020-11-16T20:26:40.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850907 '!=' null 2020-11-16T20:26:40.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:40.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:40.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:40.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:40.731 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:40.732 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:40.732 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:40.732 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:40.732 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:40.733 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:40.733 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:40.733 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:40.733 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:40.733 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:40.734 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:40.734 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:40.734 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:40.734 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:40.734 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:40.734 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:40.735 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:40.735 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:40.735 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:40.735 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:40.735 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:40.735 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:40.736 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:40.736 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:40.736 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:40.736 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:40.736 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:40.736 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:40.737 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:40.737 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:40.737 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:40.737 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:40.737 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:40.737 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:40.738 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:40.738 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:40.738 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:40.738 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:40.738 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:40.738 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:40.739 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:40.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:40.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:40.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:40.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:40.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:40.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:40.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:41.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:41.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:41.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:41.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:41.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:41.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:41.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:41.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:41.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:41.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:41.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:41.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:41.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:42.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850820 2020-11-16T20:26:42.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850820 '!=' null 2020-11-16T20:26:42.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:42.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:42.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:42.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:42.310 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:42.311 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:42.311 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:42.311 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:42.311 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:42.311 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:42.311 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:42.311 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:42.312 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:42.312 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:42.312 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:42.312 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:42.312 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:42.312 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:42.312 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:42.313 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:42.313 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:42.313 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:42.313 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:42.313 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:42.313 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:42.313 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:42.314 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:42.314 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:42.314 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:42.314 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:42.314 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:42.314 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:42.314 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:42.315 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:42.315 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:42.315 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:42.315 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:42.315 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:42.315 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:42.315 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:42.315 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:42.316 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:42.316 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:42.316 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:42.316 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:42.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:42.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:42.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:42.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:42.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:42.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:42.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:42.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:42.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:42.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:42.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:42.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:43.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:43.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:44.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21839075 2020-11-16T20:26:44.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21839075 '!=' null 2020-11-16T20:26:44.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:44.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:44.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:44.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:44.105 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:44.105 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:44.105 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:44.106 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:44.106 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:44.106 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:44.106 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:44.106 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:44.106 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:44.106 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:44.107 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:44.107 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:44.107 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:44.107 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:44.107 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:44.107 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:44.107 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 151, 2020-11-16T20:26:44.108 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:44.108 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:44.108 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:44.108 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:44.108 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:44.108 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:44.108 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:44.109 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:44.109 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:44.109 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:44.109 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:44.109 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T20:26:44.109 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 254, 2020-11-16T20:26:44.109 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:44.109 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:44.110 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:44.110 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:44.110 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:44.110 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:44.110 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:44.110 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:44.110 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:44.111 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:44.111 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:44.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:44.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:44.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:44.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:44.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:44.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:44.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:44.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:44.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:44.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:44.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:44.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:44.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:45.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:45.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:45.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:45.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:45.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:45.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:45.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:45.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:45.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:46.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850375 2020-11-16T20:26:46.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850375 '!=' null 2020-11-16T20:26:46.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:46.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:46.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:46.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:46.155 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:46.155 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:46.155 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:46.155 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:46.156 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:46.156 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:46.156 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:46.156 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:46.156 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:46.156 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:46.156 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:46.157 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:46.157 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:46.157 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:46.157 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:46.157 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:46.157 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:46.157 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:46.157 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:46.158 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:46.158 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:46.158 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:46.158 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:46.158 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:46.158 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:46.158 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:46.159 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:46.159 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:46.159 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:46.159 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:46.159 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:46.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:46.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:46.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:46.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:46.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:46.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:46.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:46.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:46.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:46.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:26:46.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:46.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:46.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T20:26:47.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:26:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:47.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:47.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:47.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:47.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19655079 2020-11-16T20:26:47.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19655079 '!=' null 2020-11-16T20:26:47.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:47.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:47.789 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:47.789 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:47.890 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:47.890 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:47.891 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:47.891 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:47.891 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:47.891 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:47.891 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:47.891 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:47.891 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:47.892 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:47.892 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:47.892 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:47.892 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:47.892 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:47.892 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:47.892 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:47.893 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:47.893 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:47.893 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:47.893 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:47.893 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:47.893 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:47.893 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:47.893 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:47.894 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:47.894 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:47.894 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:47.894 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:47.894 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:47.894 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:47.894 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:47.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:48.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:48.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:48.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:48.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:48.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:48.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:48.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:48.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:48.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:26:48.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:48.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:48.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:49.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:26:49.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 0 2020-11-16T20:26:49.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:26:49.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=1 2020-11-16T20:26:49.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:49.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:49.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:49.267 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:49.267 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:49.267 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:49.268 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:49.268 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:49.268 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:49.268 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:49.268 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:49.268 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:49.268 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:49.269 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:49.269 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:49.269 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:49.269 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:49.269 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:49.269 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:49.269 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:49.269 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:49.270 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:49.270 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:49.270 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:49.270 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:49.270 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:49.270 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:49.270 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:49.271 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:49.271 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:49.271 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:49.271 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:49.271 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:49.271 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:49.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:49.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:49.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:49.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:49.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:49.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:49.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:49.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:49.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:49.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:26:49.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:49.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:49.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:50.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:26:50.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:26:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:26:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:26:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:26:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:26:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:26:50.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:26:50.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:26:50.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19655531 2020-11-16T20:26:50.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19655531 '!=' null 2020-11-16T20:26:50.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:26:50.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:50.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:26:50.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:26:50.993 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:26:50.994 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:26:50.994 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:50.994 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:50.994 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:26:50.995 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:50.995 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:26:50.995 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:26:50.995 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:50.995 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:26:50.995 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:26:50.995 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:50.996 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:50.996 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:50.996 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:50.996 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:26:50.996 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:26:50.996 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:26:50.996 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:26:50.996 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:26:50.997 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:26:50.997 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:50.997 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:26:50.997 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:26:50.997 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:26:50.997 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:26:50.997 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:26:50.998 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:26:50.998 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:26:50.998 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:26:50.998 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:26:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:26:51.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:26:51.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:26:51.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:26:51.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:26:51.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:26:51.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:26:51.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:26:51.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:26:51.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:26:51.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:26:51.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:26:51.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:26:52.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=9172768 2020-11-16T20:27:12.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 9172768 '!=' null 2020-11-16T20:27:12.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20966574 2020-11-16T20:27:12.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20966574 '!=' null 2020-11-16T20:27:12.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17169615 2020-11-16T20:27:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17169615 '!=' null 2020-11-16T20:27:12.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23436561 2020-11-16T20:27:12.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23436561 '!=' null 2020-11-16T20:27:12.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725081 2020-11-16T20:27:12.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725081 '!=' null 2020-11-16T20:27:12.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725081 2020-11-16T20:27:12.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725081 '!=' null 2020-11-16T20:27:12.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725338 2020-11-16T20:27:12.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725338 '!=' null 2020-11-16T20:27:12.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21840693 2020-11-16T20:27:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21840693 '!=' null 2020-11-16T20:27:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978176 2020-11-16T20:27:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978176 '!=' null 2020-11-16T20:27:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21840591 2020-11-16T20:27:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21840591 '!=' null 2020-11-16T20:27:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13977693 2020-11-16T20:27:12.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13977693 '!=' null 2020-11-16T20:27:12.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13977693 2020-11-16T20:27:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13977693 '!=' null 2020-11-16T20:27:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:12.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:12.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:12.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:12.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:12.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13977521 2020-11-16T20:27:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13977521 '!=' null 2020-11-16T20:27:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:12.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:12.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:12.651 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.651 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.651 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.651 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.652 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.652 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.652 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.652 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.652 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.652 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.652 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.653 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.653 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.653 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.653 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.653 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.653 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.653 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.653 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.654 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.654 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.654 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.654 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.654 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.654 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.654 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.655 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.655 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.655 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.655 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.655 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.655 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.655 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.656 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.656 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.656 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.656 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.656 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.656 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.657 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.657 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.657 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.657 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.657 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.658 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.658 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.658 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.658 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.658 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.658 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.659 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.659 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.659 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.659 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.659 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.659 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.659 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.660 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.660 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.660 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.660 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.660 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.660 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.660 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.661 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.661 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.661 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.661 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.661 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.661 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.661 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.661 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.662 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.662 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.662 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.662 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.662 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.662 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.662 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.663 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.663 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.663 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.663 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.663 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.663 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.663 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.664 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.664 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.664 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.664 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.664 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.664 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.664 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.664 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.665 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.665 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.665 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.665 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.665 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.665 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.665 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.665 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.666 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.666 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.666 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.666 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.666 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.666 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.666 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.667 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.667 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.667 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.667 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.667 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.667 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.667 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.667 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.668 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.668 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.668 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.668 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.668 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.668 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.668 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.669 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.669 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.669 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.669 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.669 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.669 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.669 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.669 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.670 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.670 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.670 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.670 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.670 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.670 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.670 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.671 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.671 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.671 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.671 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.671 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.671 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.671 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.671 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.672 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.672 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.672 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.672 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.672 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.672 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.672 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.673 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.673 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.673 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.673 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.673 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.673 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.673 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.674 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.674 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.674 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.674 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.674 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.674 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.674 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.674 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.675 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.675 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.675 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.675 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.675 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.675 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.675 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.676 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.676 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.676 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.676 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.676 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.676 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.676 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.676 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.677 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.677 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.677 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.677 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.677 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.677 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.677 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.678 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.678 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.678 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.678 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.678 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.678 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.678 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.678 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.679 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.679 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.679 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.679 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.679 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.679 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.679 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.680 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.680 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.680 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.680 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.680 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.680 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.680 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.680 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.681 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.681 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.681 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.681 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.681 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.681 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.681 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.682 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.682 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.682 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.682 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.682 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.682 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.682 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.683 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.683 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.683 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.683 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.683 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.683 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.683 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.683 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.684 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.684 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.684 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.684 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.684 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.684 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.684 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.685 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.685 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.685 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.685 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.685 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.685 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.685 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.685 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.686 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.686 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.686 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.686 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.686 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.686 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.686 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.687 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.687 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.687 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.687 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.687 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.687 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.687 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.687 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.688 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.688 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.688 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.688 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.688 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.688 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.688 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.688 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.689 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.689 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.689 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.689 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.689 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.689 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.689 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.690 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.690 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.690 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.690 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.690 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.690 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.690 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.691 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.691 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.691 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.691 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.691 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.691 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.691 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.692 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.692 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.692 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.692 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.692 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.692 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.692 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.693 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.693 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.693 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.693 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.693 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.693 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.693 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.693 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.694 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.694 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.694 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.694 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.694 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.694 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.694 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.694 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.695 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.695 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.695 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.695 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.695 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.695 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.695 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.696 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.696 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.696 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.696 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.696 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.696 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.696 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.697 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.697 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.697 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.697 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.697 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.697 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.697 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.697 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.698 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.698 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.698 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.698 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.698 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.698 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.698 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.699 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.699 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.699 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.699 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.699 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.699 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.699 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.699 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.700 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.700 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.700 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.700 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.700 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.700 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.700 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.700 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.701 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.701 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.701 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.701 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.701 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.701 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.701 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.702 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.702 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.702 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:12.703 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:12.703 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.703 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.703 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:12.703 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.703 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:12.703 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:12.703 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.704 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:12.704 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:12.704 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.704 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.704 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.704 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.704 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:12.705 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:12.705 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:12.705 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:12.705 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:12.705 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:12.705 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.705 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:12.706 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:12.706 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:12.706 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:12.706 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:12.706 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:12.706 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:12.706 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:12.706 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:12.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:12.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:12.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:12.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:12.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:12.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:12.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:12.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:13.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:13.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:13.196 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:13.196 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:13.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:13.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:13.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:13.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:13.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:14.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20966327 2020-11-16T20:27:14.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20966327 '!=' null 2020-11-16T20:27:14.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:14.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:14.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:14.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:14.255 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:14.256 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:14.256 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:14.256 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:14.256 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T20:27:14.256 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:14.257 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 110, 2020-11-16T20:27:14.257 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T20:27:14.257 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:14.257 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:14.257 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:14.257 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:14.257 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:14.258 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:14.258 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:14.258 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:27:14.258 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:14.258 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:14.258 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T20:27:14.258 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 151, 2020-11-16T20:27:14.258 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:14.259 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:14.259 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:14.259 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:14.259 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:14.259 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:14.259 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:14.259 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:14.260 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:14.260 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:14.260 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:14.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:14.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:14.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:14.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:14.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:14.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:14.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:14.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:14.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:14.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:14.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:14.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:14.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:15.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:15.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:15.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:15.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:15.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:15.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:15.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:15.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:15.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:15.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851105 2020-11-16T20:27:15.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851105 '!=' null 2020-11-16T20:27:15.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:15.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:15.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:15.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:15.829 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:15.830 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:15.830 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:15.830 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:15.830 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:15.830 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:15.830 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:15.831 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:15.831 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:15.831 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:15.831 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:15.831 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:15.831 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:15.831 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:15.832 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:15.832 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:15.832 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:15.832 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:15.832 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:15.832 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:15.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:15.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:15.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:15.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:15.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:15.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:15.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:15.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:15.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:16.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:16.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:16.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:16.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:16.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:16.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:16.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:16.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:16.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:16.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:16.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:16.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:16.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:17.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19656267 2020-11-16T20:27:17.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19656267 '!=' null 2020-11-16T20:27:17.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:17.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:17.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:17.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:17.482 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:17.482 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:17.482 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:17.482 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:17.482 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:17.485 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:17.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:17.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:17.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:17.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:17.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:17.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:17.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:17.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:17.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:17.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:17.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:18.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:18.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:18.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:18.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:18.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:18.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:18.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:18.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:18.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:19.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19656788 2020-11-16T20:27:19.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19656788 '!=' null 2020-11-16T20:27:19.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:19.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:19.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:19.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:19.560 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:19.560 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:19.560 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:19.561 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:19.561 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:19.561 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:19.561 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:19.561 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:19.561 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:19.561 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:19.561 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:19.562 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:19.562 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:19.562 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:19.562 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:19.562 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:19.562 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:19.562 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:19.563 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:19.563 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:19.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:19.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:19.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:19.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:19.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:19.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:19.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:19.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:19.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:20.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:20.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:20.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:20.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:20.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:20.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:20.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:20.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:21.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=9173218 2020-11-16T20:27:21.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 9173218 '!=' null 2020-11-16T20:27:21.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:21.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:21.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:21.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:21.263 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:21.264 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:21.264 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:21.264 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:21.264 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:21.264 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:21.264 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:21.265 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:21.265 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:21.265 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:21.265 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:21.265 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:21.265 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:21.265 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:21.266 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:21.266 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:21.266 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:21.266 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:21.266 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:21.266 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:21.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:21.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:21.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:21.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:21.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:21.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:21.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:21.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:21.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:21.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:27:21.737 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:21.737 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:21.737 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:22.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T20:27:22.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T20:27:22.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:22.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:22.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:22.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:22.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:22.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:22.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:22.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19657336 2020-11-16T20:27:22.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19657336 '!=' null 2020-11-16T20:27:22.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:22.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:22.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:22.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:23.044 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:23.044 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:23.045 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:23.045 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:23.045 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:23.045 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:23.045 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:23.045 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:23.045 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:23.046 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:23.046 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:23.046 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:23.046 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:23.046 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:23.046 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:23.046 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:23.047 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:23.047 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:23.047 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:23.047 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:23.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:23.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:23.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:23.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:23.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:23.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:23.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:23.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:23.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:23.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:23.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:23.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:24.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:24.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 1 2020-11-16T20:27:24.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:27:24.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=2 2020-11-16T20:27:24.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:24.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:24.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:24.100 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:24.101 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:24.101 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:24.101 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:24.101 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:24.101 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:24.101 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:24.102 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:24.102 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:24.102 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:24.102 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:24.102 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:24.102 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:24.102 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:24.103 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:24.103 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:24.103 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:24.103 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:24.103 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:24.103 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:24.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:24.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:24.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:24.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:24.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:24.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:24.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:24.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:24.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:24.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:24.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:24.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:25.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:25.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:25.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:25.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:25.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:26.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17654057 2020-11-16T20:27:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17654057 '!=' null 2020-11-16T20:27:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:26.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:26.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:26.153 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:26.154 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:26.154 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:26.154 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:26.154 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:26.154 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:26.154 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:26.155 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:26.155 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:26.155 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:26.155 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:26.155 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:26.155 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:26.155 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:26.156 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:26.156 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:26.156 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:26.156 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:26.156 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:26.156 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:26.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:26.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:26.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:26.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:26.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:26.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:26.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:26.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:26.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:26.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:26.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:26.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:27.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:27.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:27.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:27.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:27.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:27.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:27.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:27.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:27.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:27.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17782864 2020-11-16T20:27:27.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17782864 '!=' null 2020-11-16T20:27:27.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:27.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:27.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:27.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:27.756 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:27.756 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:27.757 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:27.757 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:27.757 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:27.757 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:27.757 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:27.757 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:27.757 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:27.758 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:27.758 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:27.758 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:27.758 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:27.758 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:27.758 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:27.758 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:27.759 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:27.759 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:27.759 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:27.759 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:27.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:27.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:27.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:27.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:27.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:27.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:27.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:27.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:27.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:28.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:28.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:28.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:28.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:28.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:28.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:28.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:28.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:28.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:28.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:28.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:28.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:28.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:29.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851996 2020-11-16T20:27:29.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851996 '!=' null 2020-11-16T20:27:29.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:29.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:29.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:29.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:29.382 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:29.382 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:29.382 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:29.382 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:29.382 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:29.383 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:29.383 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:29.383 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:29.383 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:29.383 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:29.383 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:29.383 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:29.384 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:29.384 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:29.384 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:29.384 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:29.384 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:29.384 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:29.384 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:29.385 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:29.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:29.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:29.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:29.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:29.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:29.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:29.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:29.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:29.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:29.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:29.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:29.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:29.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:30.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:30.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:30.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:30.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:30.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:30.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:30.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:30.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:30.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:30.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851996 2020-11-16T20:27:30.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851996 '!=' null 2020-11-16T20:27:30.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:30.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:30.906 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:30.906 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:30.996 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:30.997 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:30.997 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:30.997 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:30.997 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:30.997 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:30.997 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:30.998 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:30.998 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:30.998 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:30.998 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:30.998 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:30.998 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:30.998 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:30.999 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:30.999 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:30.999 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:30.999 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:30.999 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:30.999 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:31.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:31.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:31.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:31.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:31.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:31.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:31.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:31.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:31.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:31.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:31.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:31.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:31.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:31.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:31.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:31.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:31.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:31.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:31.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:31.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:31.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:31.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851789 2020-11-16T20:27:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851789 '!=' null 2020-11-16T20:27:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:32.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:32.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:32.740 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:32.740 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:32.740 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:32.740 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:32.740 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:32.741 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:32.741 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:32.741 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:32.741 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:32.741 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:32.741 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:32.741 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:32.742 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:32.742 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:32.742 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:32.742 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:32.742 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:32.742 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:32.742 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:32.743 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:32.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:32.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:32.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:32.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:32.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:32.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:32.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:32.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:32.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:33.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:33.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:33.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:33.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:33.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:33.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:33.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:33.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:33.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:33.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:33.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:33.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:33.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:34.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20966854 2020-11-16T20:27:34.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20966854 '!=' null 2020-11-16T20:27:34.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:34.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:34.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:34.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:34.661 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:34.661 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:34.661 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:34.661 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:34.662 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:34.662 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:34.662 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:34.662 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:34.662 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:34.662 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:34.663 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:34.663 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:34.663 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:34.663 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:34.663 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:34.663 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:34.663 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:34.663 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:34.664 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:34.664 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:34.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:34.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:34.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:34.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:34.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:34.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:34.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:34.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:34.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:35.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:35.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:35.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:35.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:35.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:35.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:35.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:35.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:36.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13977670 2020-11-16T20:27:36.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13977670 '!=' null 2020-11-16T20:27:36.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:36.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:36.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:36.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:36.219 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:36.220 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:36.220 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:36.220 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:36.220 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:36.220 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:36.220 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:36.221 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:36.221 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:36.221 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:36.221 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:36.221 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:36.221 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:36.222 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:36.222 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:36.222 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:36.222 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:36.222 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:36.223 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:36.223 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:36.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:36.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:36.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:36.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:36.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:36.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:36.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:36.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:36.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:36.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:36.698 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:36.698 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:36.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:37.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:37.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:37.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:37.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:37.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:37.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:37.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:37.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:37.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:38.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22713669 2020-11-16T20:27:38.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22713669 '!=' null 2020-11-16T20:27:38.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:38.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:38.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:38.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:38.115 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:38.115 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:38.115 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:38.115 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:38.115 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:38.115 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:38.116 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:38.116 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:38.116 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:38.116 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:38.116 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:38.116 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:38.116 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:38.117 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:38.117 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:38.117 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:38.117 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:38.117 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:38.117 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:38.117 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:38.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:38.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:38.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:38.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:38.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:38.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:38.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:38.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:38.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:38.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:38.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:38.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:38.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:39.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:39.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:39.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:39.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:39.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:39.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:39.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:39.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:39.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:39.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851082 2020-11-16T20:27:39.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851082 '!=' null 2020-11-16T20:27:39.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:39.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:39.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:39.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:39.782 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:39.783 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:39.783 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:39.783 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:39.783 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:39.783 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:39.784 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:39.784 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:39.784 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:39.784 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:39.784 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:39.784 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:39.784 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:39.785 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:39.785 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:39.785 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:39.785 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:39.785 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:39.785 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:39.785 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:39.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:39.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:39.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:39.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:39.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:39.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:39.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:39.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:39.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:40.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:40.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:40.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:40.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:40.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:40.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:40.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:40.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:40.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:40.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:40.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:40.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:40.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:41.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850835 2020-11-16T20:27:41.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850835 '!=' null 2020-11-16T20:27:41.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:41.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:41.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:41.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:41.359 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:41.359 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:41.359 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:41.359 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:41.359 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:41.360 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:41.360 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:41.360 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:41.360 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:41.360 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:41.360 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:41.360 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:41.361 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:41.361 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:41.361 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:41.361 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:41.361 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:41.361 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:41.362 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:41.362 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:41.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:41.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:41.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:41.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:41.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:41.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:41.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:41.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:41.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:41.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:42.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:42.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:42.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:42.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:42.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:42.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:42.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:42.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850835 2020-11-16T20:27:42.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850835 '!=' null 2020-11-16T20:27:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:42.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:42.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:42.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:42.947 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:42.948 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:42.948 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:42.948 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:42.948 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:42.948 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:42.949 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:42.949 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:42.949 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:42.949 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:42.949 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:42.949 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:42.949 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:42.950 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:42.950 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:42.950 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:42.950 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:42.950 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:42.950 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:42.950 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:42.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:43.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:43.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:43.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:43.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:43.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:43.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:43.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:43.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:43.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:43.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:43.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:43.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:44.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:44.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:45.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22713053 2020-11-16T20:27:45.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22713053 '!=' null 2020-11-16T20:27:45.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:45.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:45.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:45.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:45.276 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:45.276 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:45.276 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:45.276 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:45.277 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:45.277 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:45.277 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:45.277 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:45.277 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:45.277 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:45.277 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:45.278 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:45.278 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:45.278 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:45.278 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:45.278 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:45.278 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:45.278 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:45.279 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:45.279 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:45.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:45.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:45.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:45.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:45.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:45.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:45.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:45.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:45.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:45.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:45.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:45.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:45.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:46.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:46.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:46.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:46.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:46.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:46.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:46.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:46.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:46.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:47.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20092959 2020-11-16T20:27:47.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20092959 '!=' null 2020-11-16T20:27:47.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:47.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:47.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:47.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:47.211 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:47.211 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:47.211 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:47.211 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:47.212 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:47.212 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T20:27:47.212 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:47.212 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T20:27:47.212 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 110, 2020-11-16T20:27:47.212 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T20:27:47.212 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:47.213 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:47.213 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:47.213 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:47.213 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:47.213 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:47.213 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:47.213 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:47.214 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:47.214 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:47.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:47.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:47.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:47.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:47.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:47.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:47.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:47.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:47.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:47.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:47.707 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:47.707 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:48.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:48.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:48.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20092959 2020-11-16T20:27:48.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20092959 '!=' null 2020-11-16T20:27:48.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:48.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:48.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:48.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:48.904 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:48.904 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:48.904 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:48.904 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:48.905 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:48.905 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:48.905 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:48.905 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:48.905 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:48.905 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:48.905 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:48.906 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:48.906 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:48.906 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:48.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:49.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:49.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:49.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:49.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:49.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:49.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:49.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:49.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:49.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:49.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:49.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:49.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:49.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:49.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:49.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:49.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:49.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:49.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:49.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:49.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:49.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:50.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17036148 2020-11-16T20:27:50.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17036148 '!=' null 2020-11-16T20:27:50.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:50.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:50.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:50.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:50.458 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:50.459 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:50.459 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:50.459 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:50.459 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:50.459 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:50.459 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:50.460 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:50.460 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:50.460 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:50.460 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:50.460 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:50.460 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:50.460 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:50.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:50.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:50.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:50.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:50.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:50.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:50.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:50.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:50.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:50.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:50.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:50.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:50.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:51.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:51.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:51.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:51.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:51.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:51.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:51.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:51.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:51.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:51.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5241995 2020-11-16T20:27:51.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5241995 '!=' null 2020-11-16T20:27:51.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:51.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:51.988 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:51.988 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:52.089 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:52.089 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:52.089 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:52.089 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:52.090 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:52.090 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:52.090 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:52.090 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:52.090 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:52.090 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:52.090 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:52.091 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:52.091 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:52.091 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:52.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:52.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:52.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:52.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:52.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:52.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:52.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:52.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:52.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:52.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T20:27:52.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:52.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:52.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:53.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T20:27:53.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T20:27:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:27:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:27:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:27:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:27:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:27:53.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:27:53.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:27:53.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=9173246 2020-11-16T20:27:53.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 9173246 '!=' null 2020-11-16T20:27:53.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:27:53.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:53.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T20:27:53.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T20:27:53.633 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:53.634 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T20:27:53.634 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:53.634 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:53.634 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:53.635 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:53.635 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:53.635 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T20:27:53.635 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T20:27:53.635 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T20:27:53.635 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T20:27:53.636 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T20:27:53.636 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:53.636 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:53.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:27:53.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:27:53.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:27:53.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:27:53.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:27:53.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:27:53.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:27:53.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:27:53.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:27:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2020-11-16T20:27:54.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:27:54.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:27:54.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:27:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2020-11-16T20:27:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:27:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:27:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:298: TEST_backfill_priority: ceph pg dump pgs 2020-11-16T20:27:54.925 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T20:27:54.926 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-11-16T20:26:46.177378+0000 91'150 117:347 [1,4] 1 [1,4] 1 0'0 2020-11-16T20:23:23.408790+0000 0'0 2020-11-16T20:23:23.408790+0000 0 2020-11-16T20:27:54.926 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-11-16T20:27:48.629627+0000 91'150 118:356 [1,2] 1 [1,2] 1 0'0 2020-11-16T20:23:08.268659+0000 0'0 2020-11-16T20:23:08.268659+0000 0 2020-11-16T20:27:54.926 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-11-16T20:27:17.287367+0000 91'150 117:360 [1,2] 1 [1,2] 1 0'0 2020-11-16T20:23:13.066896+0000 0'0 2020-11-16T20:23:13.066896+0000 0 2020-11-16T20:27:54.926 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T20:27:54.927 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T20:27:54.928 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:27:54.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: get_asok_path osd.1 2020-11-16T20:27:54.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T20:27:54.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T20:27:54.939 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:27:54.939 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:27:54.939 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:27:54.940 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65265/ceph-osd.1.asok 2020-11-16T20:27:54.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: CEPH_ARGS= 2020-11-16T20:27:54.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.65265/ceph-osd.1.asok dump_pgstate_history 2020-11-16T20:27:55.031 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:27:55.031 INFO:tasks.workunit.client.0.smithi189.stdout: "pgs": [ 2020-11-16T20:27:55.031 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.032 INFO:tasks.workunit.client.0.smithi189.stdout: "pg": "7.0", 2020-11-16T20:27:55.032 INFO:tasks.workunit.client.0.smithi189.stdout: "currently": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.032 INFO:tasks.workunit.client.0.smithi189.stdout: "history": [ 2020-11-16T20:27:55.032 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.032 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "59", 2020-11-16T20:27:55.032 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.033 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.033 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.033 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:23.874658+0000", 2020-11-16T20:27:55.033 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:23.874687+0000" 2020-11-16T20:27:55.033 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.033 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.033 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.033 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:23.874710+0000", 2020-11-16T20:27:55.034 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:23.966954+0000" 2020-11-16T20:27:55.034 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.034 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.034 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.034 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:23.966956+0000", 2020-11-16T20:27:55.034 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:23.967263+0000" 2020-11-16T20:27:55.034 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.035 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.035 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T20:27:55.035 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:23.967267+0000", 2020-11-16T20:27:55.035 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:23.967343+0000" 2020-11-16T20:27:55.035 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.035 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.035 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T20:27:55.036 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:23.967344+0000", 2020-11-16T20:27:55.036 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:24.204786+0000" 2020-11-16T20:27:55.036 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.036 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.036 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.036 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:23.874701+0000", 2020-11-16T20:27:55.036 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:24.204815+0000" 2020-11-16T20:27:55.036 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.037 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.037 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T20:27:55.037 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:24.205388+0000", 2020-11-16T20:27:55.037 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:24.696746+0000" 2020-11-16T20:27:55.037 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.037 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.037 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T20:27:55.038 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:24.696747+0000", 2020-11-16T20:27:55.038 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:24.696829+0000" 2020-11-16T20:27:55.038 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.038 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.038 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.038 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:24.696830+0000", 2020-11-16T20:27:55.038 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.298028+0000" 2020-11-16T20:27:55.039 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.039 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.039 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T20:27:55.039 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:24.204816+0000", 2020-11-16T20:27:55.039 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.298082+0000" 2020-11-16T20:27:55.039 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.039 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.040 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.040 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:23.874687+0000", 2020-11-16T20:27:55.040 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.298123+0000" 2020-11-16T20:27:55.040 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.040 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.040 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.040 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:23.874650+0000", 2020-11-16T20:27:55.040 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.298133+0000" 2020-11-16T20:27:55.041 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.041 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.041 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.041 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.041 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "59", 2020-11-16T20:27:55.041 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.041 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.042 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.042 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.298134+0000", 2020-11-16T20:27:55.042 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.298658+0000" 2020-11-16T20:27:55.042 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.042 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.042 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.042 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.042 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "89", 2020-11-16T20:27:55.043 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.043 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.043 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.043 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.298668+0000", 2020-11-16T20:27:55.043 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.298701+0000" 2020-11-16T20:27:55.043 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.043 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.044 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.044 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.298720+0000", 2020-11-16T20:27:55.044 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.443105+0000" 2020-11-16T20:27:55.044 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.044 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.044 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.044 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.443106+0000", 2020-11-16T20:27:55.044 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.443219+0000" 2020-11-16T20:27:55.045 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.045 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.045 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T20:27:55.045 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.443220+0000", 2020-11-16T20:27:55.045 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.443249+0000" 2020-11-16T20:27:55.045 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.045 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.046 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T20:27:55.046 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.443250+0000", 2020-11-16T20:27:55.046 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.759174+0000" 2020-11-16T20:27:55.046 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.046 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.046 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.046 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.298711+0000", 2020-11-16T20:27:55.047 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.759186+0000" 2020-11-16T20:27:55.047 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.047 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.047 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T20:27:55.047 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.759432+0000", 2020-11-16T20:27:55.047 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.953336+0000" 2020-11-16T20:27:55.047 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.048 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.048 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T20:27:55.048 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.953337+0000", 2020-11-16T20:27:55.048 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:25.953468+0000" 2020-11-16T20:27:55.048 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.048 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.048 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.048 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.953469+0000", 2020-11-16T20:27:55.049 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.175622+0000" 2020-11-16T20:27:55.049 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.049 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.049 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T20:27:55.049 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.759187+0000", 2020-11-16T20:27:55.049 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.175655+0000" 2020-11-16T20:27:55.049 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.049 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.050 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.050 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.298702+0000", 2020-11-16T20:27:55.050 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.175681+0000" 2020-11-16T20:27:55.050 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.050 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.050 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.050 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:25.298659+0000", 2020-11-16T20:27:55.051 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.175688+0000" 2020-11-16T20:27:55.051 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.051 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.051 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.051 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.051 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "89", 2020-11-16T20:27:55.051 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.052 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.052 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.052 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.175689+0000", 2020-11-16T20:27:55.052 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.176035+0000" 2020-11-16T20:27:55.052 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.052 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.052 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.052 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.053 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "96", 2020-11-16T20:27:55.053 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.053 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.053 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.053 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.176041+0000", 2020-11-16T20:27:55.053 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.176061+0000" 2020-11-16T20:27:55.053 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.054 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.054 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.054 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.176075+0000", 2020-11-16T20:27:55.054 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.176647+0000" 2020-11-16T20:27:55.054 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.054 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.054 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.054 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.176648+0000", 2020-11-16T20:27:55.055 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.176745+0000" 2020-11-16T20:27:55.055 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.055 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.055 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T20:27:55.055 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.176745+0000", 2020-11-16T20:27:55.055 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.176759+0000" 2020-11-16T20:27:55.055 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.056 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.056 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T20:27:55.056 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.176760+0000", 2020-11-16T20:27:55.056 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.590702+0000" 2020-11-16T20:27:55.056 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.056 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.056 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.056 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.176069+0000", 2020-11-16T20:27:55.057 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.590744+0000" 2020-11-16T20:27:55.057 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.057 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.057 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T20:27:55.057 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.591138+0000", 2020-11-16T20:27:55.057 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.661604+0000" 2020-11-16T20:27:55.057 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.058 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.058 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T20:27:55.058 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.661606+0000", 2020-11-16T20:27:55.058 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:53.661817+0000" 2020-11-16T20:27:55.058 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.058 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.058 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.058 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.661818+0000", 2020-11-16T20:27:55.059 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:28.587039+0000" 2020-11-16T20:27:55.059 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.059 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.059 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T20:27:55.059 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.590746+0000", 2020-11-16T20:27:55.059 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:28.587097+0000" 2020-11-16T20:27:55.059 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.060 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.060 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.060 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.176062+0000", 2020-11-16T20:27:55.060 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:28.587194+0000" 2020-11-16T20:27:55.060 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.060 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.060 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.060 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:53.176035+0000", 2020-11-16T20:27:55.061 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:28.587209+0000" 2020-11-16T20:27:55.061 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.061 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.061 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.061 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.061 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "97", 2020-11-16T20:27:55.061 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.061 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.062 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.062 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:28.587210+0000", 2020-11-16T20:27:55.062 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:28.588229+0000" 2020-11-16T20:27:55.062 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.062 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.062 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.062 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.063 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "99", 2020-11-16T20:27:55.063 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.063 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.063 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.063 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:28.588245+0000", 2020-11-16T20:27:55.063 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:28.588290+0000" 2020-11-16T20:27:55.063 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.063 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.064 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.064 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:28.588319+0000", 2020-11-16T20:27:55.064 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:28.702558+0000" 2020-11-16T20:27:55.064 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.064 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.064 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.064 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:28.702558+0000", 2020-11-16T20:27:55.065 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:28.702626+0000" 2020-11-16T20:27:55.065 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.065 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.065 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.065 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:28.588305+0000", 2020-11-16T20:27:55.065 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:28.702635+0000" 2020-11-16T20:27:55.065 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.065 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.066 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/WaitActingChange", 2020-11-16T20:27:55.066 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:28.702636+0000", 2020-11-16T20:27:55.066 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:29.080867+0000" 2020-11-16T20:27:55.066 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.066 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.066 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.066 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:28.588291+0000", 2020-11-16T20:27:55.067 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:29.080885+0000" 2020-11-16T20:27:55.067 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.067 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.067 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.067 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:28.588230+0000", 2020-11-16T20:27:55.067 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:29.080889+0000" 2020-11-16T20:27:55.067 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.067 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.068 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.068 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.068 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "99", 2020-11-16T20:27:55.068 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.068 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.068 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.068 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:29.080890+0000", 2020-11-16T20:27:55.069 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:29.081127+0000" 2020-11-16T20:27:55.069 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.069 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.069 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.069 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.069 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "113", 2020-11-16T20:27:55.069 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.069 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.070 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.070 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:29.081131+0000", 2020-11-16T20:27:55.070 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:29.081145+0000" 2020-11-16T20:27:55.070 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.070 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.070 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.070 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:29.081153+0000", 2020-11-16T20:27:55.071 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:29.082042+0000" 2020-11-16T20:27:55.071 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.071 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.071 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.071 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:29.082043+0000", 2020-11-16T20:27:55.071 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:29.082113+0000" 2020-11-16T20:27:55.071 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.072 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.072 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T20:27:55.072 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:29.082114+0000", 2020-11-16T20:27:55.072 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:29.082129+0000" 2020-11-16T20:27:55.072 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.072 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.072 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T20:27:55.073 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:29.082129+0000", 2020-11-16T20:27:55.073 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:29.432512+0000" 2020-11-16T20:27:55.073 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.073 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.073 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.073 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:29.081149+0000", 2020-11-16T20:27:55.073 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:29.432523+0000" 2020-11-16T20:27:55.073 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.074 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.074 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T20:27:55.074 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:29.432814+0000", 2020-11-16T20:27:55.074 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:30.062399+0000" 2020-11-16T20:27:55.074 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.074 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.074 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-11-16T20:27:55.075 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:30.062400+0000", 2020-11-16T20:27:55.075 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:30.062647+0000" 2020-11-16T20:27:55.075 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.075 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.075 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-11-16T20:27:55.075 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:30.062648+0000", 2020-11-16T20:27:55.075 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:30.063651+0000" 2020-11-16T20:27:55.075 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.076 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.076 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Backfilling", 2020-11-16T20:27:55.076 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:30.063652+0000", 2020-11-16T20:27:55.076 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:58.487764+0000" 2020-11-16T20:27:55.076 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.076 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.076 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-11-16T20:27:55.077 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:58.487766+0000", 2020-11-16T20:27:55.077 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:58.488334+0000" 2020-11-16T20:27:55.077 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.077 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.077 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-11-16T20:27:55.077 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:58.488335+0000", 2020-11-16T20:27:55.077 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:11.081288+0000" 2020-11-16T20:27:55.077 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.078 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.078 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-11-16T20:27:55.078 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:26:11.081288+0000", 2020-11-16T20:27:55.078 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:11.081895+0000" 2020-11-16T20:27:55.078 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.078 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.078 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Backfilling", 2020-11-16T20:27:55.079 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:26:11.081895+0000", 2020-11-16T20:27:55.079 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:44.747810+0000" 2020-11-16T20:27:55.079 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.079 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.079 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T20:27:55.079 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:26:44.747811+0000", 2020-11-16T20:27:55.079 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:44.748173+0000" 2020-11-16T20:27:55.079 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.080 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.080 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.080 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:26:44.748175+0000", 2020-11-16T20:27:55.080 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:45.099060+0000" 2020-11-16T20:27:55.080 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.080 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.080 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T20:27:55.081 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:29.432524+0000", 2020-11-16T20:27:55.081 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:45.099103+0000" 2020-11-16T20:27:55.081 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.081 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.081 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.081 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:29.081145+0000", 2020-11-16T20:27:55.081 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:45.099137+0000" 2020-11-16T20:27:55.081 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.082 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.082 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.082 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:29.081127+0000", 2020-11-16T20:27:55.082 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:45.099146+0000" 2020-11-16T20:27:55.082 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.082 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.082 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.083 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.083 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "113", 2020-11-16T20:27:55.083 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.083 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.083 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.083 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:26:45.099147+0000", 2020-11-16T20:27:55.083 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:45.099561+0000" 2020-11-16T20:27:55.083 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.084 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.084 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.084 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.084 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.084 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.084 INFO:tasks.workunit.client.0.smithi189.stdout: "pg": "3.0", 2020-11-16T20:27:55.084 INFO:tasks.workunit.client.0.smithi189.stdout: "currently": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.085 INFO:tasks.workunit.client.0.smithi189.stdout: "history": [ 2020-11-16T20:27:55.085 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.085 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "37", 2020-11-16T20:27:55.085 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.085 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.085 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.085 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:13.828922+0000", 2020-11-16T20:27:55.085 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:13.828959+0000" 2020-11-16T20:27:55.086 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.086 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.086 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.086 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:13.828984+0000", 2020-11-16T20:27:55.086 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:13.858123+0000" 2020-11-16T20:27:55.086 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.086 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.087 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.087 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:13.858124+0000", 2020-11-16T20:27:55.087 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:13.858307+0000" 2020-11-16T20:27:55.087 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.087 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.087 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T20:27:55.087 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:13.858308+0000", 2020-11-16T20:27:55.087 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:13.858345+0000" 2020-11-16T20:27:55.088 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.088 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.088 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T20:27:55.088 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:13.858346+0000", 2020-11-16T20:27:55.088 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:14.117700+0000" 2020-11-16T20:27:55.088 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.088 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.088 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.089 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:13.828971+0000", 2020-11-16T20:27:55.089 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:14.117715+0000" 2020-11-16T20:27:55.089 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.089 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.089 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T20:27:55.089 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:14.117954+0000", 2020-11-16T20:27:55.089 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:14.411451+0000" 2020-11-16T20:27:55.090 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.090 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.090 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T20:27:55.090 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:14.411452+0000", 2020-11-16T20:27:55.090 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:14.411530+0000" 2020-11-16T20:27:55.090 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.090 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.090 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.091 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:14.411531+0000", 2020-11-16T20:27:55.091 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:14.988534+0000" 2020-11-16T20:27:55.091 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.091 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.091 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T20:27:55.091 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:14.117716+0000", 2020-11-16T20:27:55.091 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:14.988552+0000" 2020-11-16T20:27:55.092 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.092 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.092 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.092 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:13.828960+0000", 2020-11-16T20:27:55.092 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:14.988564+0000" 2020-11-16T20:27:55.092 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.092 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.092 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.093 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:13.828910+0000", 2020-11-16T20:27:55.093 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:14.988568+0000" 2020-11-16T20:27:55.093 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.093 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.093 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.093 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.093 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "37", 2020-11-16T20:27:55.093 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.094 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.094 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.094 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:14.988568+0000", 2020-11-16T20:27:55.094 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:14.988763+0000" 2020-11-16T20:27:55.094 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.094 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.094 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.095 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.095 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "87", 2020-11-16T20:27:55.095 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.095 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.095 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.095 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:14.988767+0000", 2020-11-16T20:27:55.095 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:14.988778+0000" 2020-11-16T20:27:55.095 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.096 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.096 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.096 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:14.988785+0000", 2020-11-16T20:27:55.096 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:15.073435+0000" 2020-11-16T20:27:55.096 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.096 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.096 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.097 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:15.073436+0000", 2020-11-16T20:27:55.097 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:15.073562+0000" 2020-11-16T20:27:55.097 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.097 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.097 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T20:27:55.097 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:15.073562+0000", 2020-11-16T20:27:55.097 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:15.073580+0000" 2020-11-16T20:27:55.097 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.098 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.098 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T20:27:55.098 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:15.073581+0000", 2020-11-16T20:27:55.098 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:15.248306+0000" 2020-11-16T20:27:55.098 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.098 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.098 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.099 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:14.988781+0000", 2020-11-16T20:27:55.099 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:15.248319+0000" 2020-11-16T20:27:55.099 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.099 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.099 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T20:27:55.099 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:15.248481+0000", 2020-11-16T20:27:55.099 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:15.669463+0000" 2020-11-16T20:27:55.099 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.100 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.100 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T20:27:55.100 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:15.669464+0000", 2020-11-16T20:27:55.100 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:15.669645+0000" 2020-11-16T20:27:55.100 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.100 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.100 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.101 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:15.669646+0000", 2020-11-16T20:27:55.101 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.035648+0000" 2020-11-16T20:27:55.101 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.101 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.101 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T20:27:55.101 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:15.248319+0000", 2020-11-16T20:27:55.101 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.035666+0000" 2020-11-16T20:27:55.101 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.102 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.102 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.102 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:14.988778+0000", 2020-11-16T20:27:55.102 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.035679+0000" 2020-11-16T20:27:55.102 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.102 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.102 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:14.988763+0000", 2020-11-16T20:27:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.035683+0000" 2020-11-16T20:27:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "87", 2020-11-16T20:27:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.035684+0000", 2020-11-16T20:27:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.035866+0000" 2020-11-16T20:27:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "105", 2020-11-16T20:27:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.035870+0000", 2020-11-16T20:27:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.035881+0000" 2020-11-16T20:27:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.106 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.106 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.106 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.035888+0000", 2020-11-16T20:27:55.106 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.128774+0000" 2020-11-16T20:27:55.106 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.106 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.106 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.107 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.128775+0000", 2020-11-16T20:27:55.107 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.128882+0000" 2020-11-16T20:27:55.107 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.107 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.107 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T20:27:55.107 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.128883+0000", 2020-11-16T20:27:55.107 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.128906+0000" 2020-11-16T20:27:55.107 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.108 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.108 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T20:27:55.108 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.128906+0000", 2020-11-16T20:27:55.108 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.374356+0000" 2020-11-16T20:27:55.108 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.108 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.108 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.109 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.035885+0000", 2020-11-16T20:27:55.109 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.374390+0000" 2020-11-16T20:27:55.109 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.109 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.109 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T20:27:55.109 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.374688+0000", 2020-11-16T20:27:55.109 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.566068+0000" 2020-11-16T20:27:55.109 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.110 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.110 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T20:27:55.110 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.566069+0000", 2020-11-16T20:27:55.110 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:52.566224+0000" 2020-11-16T20:27:55.110 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.110 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.110 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.111 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.566225+0000", 2020-11-16T20:27:55.111 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.340048+0000" 2020-11-16T20:27:55.111 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.111 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.111 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T20:27:55.111 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.374391+0000", 2020-11-16T20:27:55.111 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.340138+0000" 2020-11-16T20:27:55.111 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.112 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.112 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.112 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.035882+0000", 2020-11-16T20:27:55.112 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.340266+0000" 2020-11-16T20:27:55.112 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.112 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.112 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.113 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:52.035866+0000", 2020-11-16T20:27:55.113 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.340280+0000" 2020-11-16T20:27:55.113 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.113 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.113 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.113 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.113 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "106", 2020-11-16T20:27:55.113 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.114 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.114 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.114 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.340281+0000", 2020-11-16T20:27:55.114 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.341193+0000" 2020-11-16T20:27:55.114 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.114 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.114 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.115 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.115 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "107", 2020-11-16T20:27:55.115 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.115 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.115 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.115 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.341210+0000", 2020-11-16T20:27:55.115 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.341247+0000" 2020-11-16T20:27:55.115 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.116 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.116 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.116 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.341269+0000", 2020-11-16T20:27:55.116 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.342520+0000" 2020-11-16T20:27:55.116 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.116 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.116 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.117 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.342521+0000", 2020-11-16T20:27:55.117 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.342670+0000" 2020-11-16T20:27:55.117 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.117 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.117 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.117 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.341259+0000", 2020-11-16T20:27:55.117 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.342699+0000" 2020-11-16T20:27:55.117 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.118 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.118 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/WaitActingChange", 2020-11-16T20:27:55.118 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.342700+0000", 2020-11-16T20:27:55.118 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.594690+0000" 2020-11-16T20:27:55.118 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.118 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.118 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.119 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.341248+0000", 2020-11-16T20:27:55.119 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.594705+0000" 2020-11-16T20:27:55.119 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.119 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.119 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.119 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.341194+0000", 2020-11-16T20:27:55.119 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.594710+0000" 2020-11-16T20:27:55.119 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.120 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.120 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.120 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.120 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "107", 2020-11-16T20:27:55.120 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.120 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.120 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.120 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.594721+0000", 2020-11-16T20:27:55.121 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.594912+0000" 2020-11-16T20:27:55.121 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.121 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.121 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.121 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.121 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "115", 2020-11-16T20:27:55.121 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.122 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.122 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.122 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.594916+0000", 2020-11-16T20:27:55.122 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.594929+0000" 2020-11-16T20:27:55.122 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.122 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.122 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.122 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.594937+0000", 2020-11-16T20:27:55.123 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.733465+0000" 2020-11-16T20:27:55.123 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.123 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.123 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.123 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.733466+0000", 2020-11-16T20:27:55.123 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.733543+0000" 2020-11-16T20:27:55.123 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.124 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.124 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T20:27:55.124 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.733544+0000", 2020-11-16T20:27:55.124 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.733562+0000" 2020-11-16T20:27:55.124 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.124 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.124 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T20:27:55.124 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.733563+0000", 2020-11-16T20:27:55.125 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.919915+0000" 2020-11-16T20:27:55.125 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.125 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.125 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.125 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.594933+0000", 2020-11-16T20:27:55.125 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:45.919943+0000" 2020-11-16T20:27:55.125 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.126 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.126 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T20:27:55.126 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.920203+0000", 2020-11-16T20:27:55.126 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:46.471051+0000" 2020-11-16T20:27:55.126 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.126 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.126 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-11-16T20:27:55.126 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:46.471052+0000", 2020-11-16T20:27:55.127 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:58.488039+0000" 2020-11-16T20:27:55.127 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.127 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.127 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-11-16T20:27:55.127 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:58.488040+0000", 2020-11-16T20:27:55.127 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:58.488897+0000" 2020-11-16T20:27:55.127 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.128 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.128 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Backfilling", 2020-11-16T20:27:55.128 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:58.488899+0000", 2020-11-16T20:27:55.128 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:11.081121+0000" 2020-11-16T20:27:55.128 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.128 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.128 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-11-16T20:27:55.128 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:26:11.081122+0000", 2020-11-16T20:27:55.129 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:11.081413+0000" 2020-11-16T20:27:55.129 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.129 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.129 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-11-16T20:27:55.129 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:26:11.081414+0000", 2020-11-16T20:27:55.129 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:44.748700+0000" 2020-11-16T20:27:55.129 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.129 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.130 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-11-16T20:27:55.130 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:26:44.748701+0000", 2020-11-16T20:27:55.130 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:26:44.749619+0000" 2020-11-16T20:27:55.130 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.130 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.130 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Backfilling", 2020-11-16T20:27:55.130 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:26:44.749620+0000", 2020-11-16T20:27:55.131 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:15.591216+0000" 2020-11-16T20:27:55.131 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.131 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.131 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T20:27:55.131 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:27:15.591218+0000", 2020-11-16T20:27:55.131 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:15.591596+0000" 2020-11-16T20:27:55.131 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.131 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.132 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.132 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:27:15.591597+0000", 2020-11-16T20:27:55.132 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:16.224806+0000" 2020-11-16T20:27:55.132 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.132 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.132 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T20:27:55.132 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.919945+0000", 2020-11-16T20:27:55.133 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:16.224841+0000" 2020-11-16T20:27:55.133 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.133 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.133 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.133 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.594929+0000", 2020-11-16T20:27:55.133 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:16.224870+0000" 2020-11-16T20:27:55.133 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.133 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.134 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.134 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:45.594912+0000", 2020-11-16T20:27:55.134 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:16.224880+0000" 2020-11-16T20:27:55.134 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.134 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.134 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.134 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.134 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "115", 2020-11-16T20:27:55.135 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.135 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.135 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.135 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:27:16.224881+0000", 2020-11-16T20:27:55.135 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:16.225399+0000" 2020-11-16T20:27:55.135 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.135 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.136 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.136 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.136 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.136 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.136 INFO:tasks.workunit.client.0.smithi189.stdout: "pg": "1.0", 2020-11-16T20:27:55.136 INFO:tasks.workunit.client.0.smithi189.stdout: "currently": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.136 INFO:tasks.workunit.client.0.smithi189.stdout: "history": [ 2020-11-16T20:27:55.136 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.137 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "26", 2020-11-16T20:27:55.137 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.137 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.137 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Initial", 2020-11-16T20:27:55.137 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:08.710622+0000", 2020-11-16T20:27:55.137 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:08.710906+0000" 2020-11-16T20:27:55.137 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.137 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.138 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.138 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.138 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "26", 2020-11-16T20:27:55.138 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.138 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.138 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.138 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:08.710907+0000", 2020-11-16T20:27:55.139 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:08.710991+0000" 2020-11-16T20:27:55.139 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.139 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.139 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.139 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.139 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "28", 2020-11-16T20:27:55.139 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.139 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.140 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.140 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:08.711001+0000", 2020-11-16T20:27:55.140 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:08.711027+0000" 2020-11-16T20:27:55.140 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.140 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.140 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.140 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:08.711053+0000", 2020-11-16T20:27:55.141 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:08.712158+0000" 2020-11-16T20:27:55.141 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.141 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.141 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.141 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:08.712159+0000", 2020-11-16T20:27:55.141 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:08.712258+0000" 2020-11-16T20:27:55.141 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.141 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.142 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T20:27:55.142 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:08.712259+0000", 2020-11-16T20:27:55.142 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:08.712271+0000" 2020-11-16T20:27:55.142 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.142 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.142 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T20:27:55.142 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:08.712271+0000", 2020-11-16T20:27:55.143 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:09.020698+0000" 2020-11-16T20:27:55.143 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.143 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.143 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.143 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:08.711043+0000", 2020-11-16T20:27:55.143 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:09.020727+0000" 2020-11-16T20:27:55.143 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.143 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.144 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T20:27:55.144 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:09.021342+0000", 2020-11-16T20:27:55.144 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.041469+0000" 2020-11-16T20:27:55.144 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.144 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.144 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T20:27:55.144 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.041470+0000", 2020-11-16T20:27:55.144 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.041586+0000" 2020-11-16T20:27:55.145 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.145 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.145 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.145 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.041587+0000", 2020-11-16T20:27:55.145 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.464756+0000" 2020-11-16T20:27:55.145 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.145 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.146 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T20:27:55.146 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:09.020729+0000", 2020-11-16T20:27:55.146 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.464794+0000" 2020-11-16T20:27:55.146 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.146 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.146 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.146 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:08.711028+0000", 2020-11-16T20:27:55.147 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.464824+0000" 2020-11-16T20:27:55.147 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.147 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.147 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.147 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:08.710992+0000", 2020-11-16T20:27:55.147 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.464833+0000" 2020-11-16T20:27:55.147 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.147 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.148 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.148 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.148 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "29", 2020-11-16T20:27:55.148 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.148 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.148 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.148 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.464834+0000", 2020-11-16T20:27:55.148 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.465507+0000" 2020-11-16T20:27:55.149 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.149 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.149 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.149 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.149 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "101", 2020-11-16T20:27:55.149 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.149 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.150 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.150 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.465520+0000", 2020-11-16T20:27:55.150 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.465556+0000" 2020-11-16T20:27:55.150 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.150 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.150 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.150 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.465576+0000", 2020-11-16T20:27:55.150 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.561379+0000" 2020-11-16T20:27:55.151 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.151 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.151 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.151 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.561380+0000", 2020-11-16T20:27:55.151 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.561499+0000" 2020-11-16T20:27:55.151 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.151 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.152 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T20:27:55.152 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.561500+0000", 2020-11-16T20:27:55.152 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.561524+0000" 2020-11-16T20:27:55.152 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.152 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.152 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T20:27:55.152 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.561524+0000", 2020-11-16T20:27:55.152 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.794112+0000" 2020-11-16T20:27:55.153 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.153 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.153 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.153 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.465565+0000", 2020-11-16T20:27:55.153 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.794142+0000" 2020-11-16T20:27:55.153 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.153 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.153 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T20:27:55.154 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.794653+0000", 2020-11-16T20:27:55.154 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.956139+0000" 2020-11-16T20:27:55.154 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.154 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.154 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T20:27:55.154 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.956140+0000", 2020-11-16T20:27:55.154 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:23:10.956323+0000" 2020-11-16T20:27:55.155 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.155 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.155 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.155 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.956324+0000", 2020-11-16T20:27:55.155 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.357017+0000" 2020-11-16T20:27:55.155 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.155 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.155 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T20:27:55.156 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.794143+0000", 2020-11-16T20:27:55.156 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.357061+0000" 2020-11-16T20:27:55.156 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.156 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.156 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.156 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.465556+0000", 2020-11-16T20:27:55.156 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.357163+0000" 2020-11-16T20:27:55.157 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.157 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.157 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.157 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:23:10.465508+0000", 2020-11-16T20:27:55.157 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.357172+0000" 2020-11-16T20:27:55.157 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.157 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.157 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.158 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.158 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "101", 2020-11-16T20:27:55.158 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.158 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.158 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.158 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.357173+0000", 2020-11-16T20:27:55.158 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.357757+0000" 2020-11-16T20:27:55.158 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.159 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.159 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.159 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.159 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "103", 2020-11-16T20:27:55.159 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.159 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.159 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.160 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.357765+0000", 2020-11-16T20:27:55.160 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.357795+0000" 2020-11-16T20:27:55.160 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.160 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.160 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.160 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.357813+0000", 2020-11-16T20:27:55.160 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.542615+0000" 2020-11-16T20:27:55.160 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.161 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.161 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.161 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.542616+0000", 2020-11-16T20:27:55.161 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.542688+0000" 2020-11-16T20:27:55.161 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.161 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.161 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.161 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.357804+0000", 2020-11-16T20:27:55.162 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.542697+0000" 2020-11-16T20:27:55.162 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.162 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.162 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/WaitActingChange", 2020-11-16T20:27:55.162 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.542698+0000", 2020-11-16T20:27:55.162 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.876368+0000" 2020-11-16T20:27:55.162 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.163 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.163 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.163 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.357796+0000", 2020-11-16T20:27:55.163 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.876384+0000" 2020-11-16T20:27:55.163 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.163 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.164 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.164 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.357757+0000", 2020-11-16T20:27:55.164 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.876389+0000" 2020-11-16T20:27:55.164 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.164 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.164 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.164 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.164 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "103", 2020-11-16T20:27:55.165 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.165 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.165 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.165 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.876389+0000", 2020-11-16T20:27:55.165 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.876615+0000" 2020-11-16T20:27:55.165 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.165 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.166 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.166 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.166 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "117", 2020-11-16T20:27:55.166 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.166 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.166 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T20:27:55.166 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.876620+0000", 2020-11-16T20:27:55.166 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.876633+0000" 2020-11-16T20:27:55.167 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.167 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.167 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T20:27:55.167 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.876642+0000", 2020-11-16T20:27:55.167 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.992019+0000" 2020-11-16T20:27:55.167 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.168 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.168 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T20:27:55.168 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.992019+0000", 2020-11-16T20:27:55.168 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.992105+0000" 2020-11-16T20:27:55.168 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.168 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.168 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T20:27:55.168 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.992105+0000", 2020-11-16T20:27:55.169 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:37.992126+0000" 2020-11-16T20:27:55.169 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.169 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.169 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T20:27:55.169 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.992127+0000", 2020-11-16T20:27:55.169 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:38.270716+0000" 2020-11-16T20:27:55.169 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.170 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.170 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T20:27:55.170 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.876638+0000", 2020-11-16T20:27:55.170 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:38.270740+0000" 2020-11-16T20:27:55.170 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.170 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.170 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T20:27:55.170 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:38.271241+0000", 2020-11-16T20:27:55.171 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:25:38.825896+0000" 2020-11-16T20:27:55.171 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.171 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.171 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-11-16T20:27:55.171 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:38.825898+0000", 2020-11-16T20:27:55.171 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:15.592106+0000" 2020-11-16T20:27:55.171 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.171 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.172 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-11-16T20:27:55.172 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:27:15.592107+0000", 2020-11-16T20:27:55.172 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:15.593006+0000" 2020-11-16T20:27:55.172 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.172 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.172 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Backfilling", 2020-11-16T20:27:55.172 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:27:15.593007+0000", 2020-11-16T20:27:55.173 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:47.808525+0000" 2020-11-16T20:27:55.173 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.173 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.173 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T20:27:55.173 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:27:47.808526+0000", 2020-11-16T20:27:55.173 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:47.808815+0000" 2020-11-16T20:27:55.173 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.174 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.174 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T20:27:55.174 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:27:47.808816+0000", 2020-11-16T20:27:55.174 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:48.032946+0000" 2020-11-16T20:27:55.174 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.174 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.174 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T20:27:55.174 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:38.270743+0000", 2020-11-16T20:27:55.175 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:48.033005+0000" 2020-11-16T20:27:55.175 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.175 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.175 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T20:27:55.175 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.876634+0000", 2020-11-16T20:27:55.175 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:48.033053+0000" 2020-11-16T20:27:55.175 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.176 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.176 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T20:27:55.176 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:25:37.876615+0000", 2020-11-16T20:27:55.176 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:48.033069+0000" 2020-11-16T20:27:55.176 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.176 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.176 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:27:55.176 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.177 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "117", 2020-11-16T20:27:55.177 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T20:27:55.177 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:27:55.177 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T20:27:55.177 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T20:27:48.033070+0000", 2020-11-16T20:27:55.177 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T20:27:48.033895+0000" 2020-11-16T20:27:55.177 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.177 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.178 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.178 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.178 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:27:55.178 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:27:55.178 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:27:55.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:302: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2020-11-16T20:27:55.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:306: TEST_backfill_priority: echo TEST PASSED 2020-11-16T20:27:55.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:309: TEST_backfill_priority: delete_pool test1 2020-11-16T20:27:55.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2020-11-16T20:27:55.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-11-16T20:27:55.180 INFO:tasks.workunit.client.0.smithi189.stdout:TEST PASSED 2020-11-16T20:27:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' does not exist 2020-11-16T20:27:55.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:310: TEST_backfill_priority: delete_pool test3 2020-11-16T20:27:55.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2020-11-16T20:27:55.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-11-16T20:27:56.143 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' does not exist 2020-11-16T20:27:56.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:311: TEST_backfill_priority: delete_pool test7 2020-11-16T20:27:56.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test7 2020-11-16T20:27:56.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-11-16T20:27:56.877 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test7' does not exist 2020-11-16T20:27:56.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:312: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2020-11-16T20:27:56.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:27:56.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:27:56.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:27:56.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:27:56.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:27:57.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:27:57.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:313: TEST_backfill_priority: return 0 2020-11-16T20:27:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-11-16T20:27:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-11-16T20:27:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:27:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-11-16T20:27:57.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:27:57.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:27:57.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:27:57.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:27:57.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:27:57.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:27:57.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:27:57.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:27:57.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:27:57.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:27:57.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:27:57.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:27:57.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:27:57.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:27:57.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:27:57.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:27:57.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:27:57.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:27:57.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-11-16T20:27:57.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:27:57.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:27:57.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:27:57.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.65265 2020-11-16T20:27:57.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:27:57.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:27:57.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T20:27:57.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-backfill-prio 0 2020-11-16T20:27:57.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-11-16T20:27:57.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T20:27:57.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-11-16T20:27:57.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:27:57.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:27:57.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:27:57.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:27:57.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:27:57.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:27:57.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:27:57.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:27:57.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:27:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:27:57.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:27:57.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:27:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:27:57.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:27:57.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:27:57.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:27:57.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:27:57.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T20:27:57.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-11-16T20:27:57.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:27:57.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:27:57.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65265 2020-11-16T20:27:57.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.65265 2020-11-16T20:27:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:27:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:27:57.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T20:27:57.284 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T20:27:57.285 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T20:27:57.340 INFO:tasks.workunit:Running workunit osd/osd-backfill-recovery-log.sh... 2020-11-16T20:27:57.341 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh 2020-11-16T20:27:57.402 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T20:27:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T20:27:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:27:57.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:27:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T20:27:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T20:27:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T20:27:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T20:27:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T20:27:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T20:27:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-backfill-recovery-log 2020-11-16T20:27:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2020-11-16T20:27:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:22: run: shift 2020-11-16T20:27:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2020-11-16T20:27:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2020-11-16T20:27:57.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2020-11-16T20:27:57.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: uuidgen 2020-11-16T20:27:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none ' 2020-11-16T20:27:57.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:27:57.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: set 2020-11-16T20:27:57.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T20:27:57.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: local 'funcs=TEST_backfill_log_1 2020-11-16T20:27:57.421 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_log_2 2020-11-16T20:27:57.421 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_recovery_1 2020-11-16T20:27:57.421 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_recovery_2' 2020-11-16T20:27:57.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-11-16T20:27:57.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-11-16T20:27:57.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-11-16T20:27:57.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-11-16T20:27:57.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:27:57.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:27:57.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-11-16T20:27:57.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:27:57.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:27:57.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:27:57.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:27:57.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:27:57.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:27:57.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:27:57.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:27:57.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:27:57.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:27:57.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:27:57.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:27:57.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:27:57.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:27:57.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:27:57.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:27:57.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:27:57.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:27:57.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-11-16T20:27:57.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:27:57.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:27:57.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:27:57.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90694 2020-11-16T20:27:57.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:27:57.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:27:57.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-11-16T20:27:57.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:27:57.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:27:57.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:27:57.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.90694 2020-11-16T20:27:57.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:27:57.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T20:27:57.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T20:27:57.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:27:57.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-11-16T20:27:57.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2020-11-16T20:27:57.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:105: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2020-11-16T20:27:57.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:107: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2020-11-16T20:27:57.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-11-16T20:27:57.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-11-16T20:27:57.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-11-16T20:27:57.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=8 2020-11-16T20:27:57.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-11-16T20:27:57.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-11-16T20:27:57.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-11-16T20:27:57.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-11-16T20:27:57.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-11-16T20:27:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:27:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:27:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:27:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-11-16T20:27:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-11-16T20:27:57.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:27:57.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:27:57.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:27:57.742 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:27:57.742 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:27:57.743 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:27:57.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:27:57.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:27:57.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:27:57.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:27:57.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:27:57.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:27:57.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:27:57.965 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:27:57.966 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:27:57.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:27:57.966 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:27:57.966 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:27:57.966 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:27:57.966 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:27:57.967 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90694/ceph-mon.a.asok 2020-11-16T20:27:57.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:27:57.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.90694/ceph-mon.a.asok config get fsid 2020-11-16T20:27:58.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:27:58.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:27:58.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:27:58.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:27:58.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:27:58.070 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:27:58.071 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:27:58.071 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:27:58.071 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:27:58.071 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:27:58.071 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:27:58.072 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90694/ceph-mon.a.asok 2020-11-16T20:27:58.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:27:58.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.90694/ceph-mon.a.asok config get mon_host 2020-11-16T20:27:58.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-11-16T20:27:58.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-recovery-log 2020-11-16T20:27:58.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:27:58.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:27:58.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:27:58.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-11-16T20:27:58.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:27:59.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:27:59.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:27:59.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:27:59.111 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:27:59.111 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:27:59.111 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:27:59.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:27:59.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:27:59.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:27:59.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-11-16T20:27:59.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-11-16T20:27:59.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-11-16T20:27:59.144 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-11-16T20:27:59.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-11-16T20:27:59.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:27:59.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-11-16T20:27:59.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:27:59.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:27:59.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:27:59.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:27:59.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-11-16T20:27:59.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:27:59.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:27:59.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:27:59.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:27:59.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-11-16T20:27:59.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-11-16T20:27:59.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:27:59.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-11-16T20:27:59.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:27:59.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:27:59.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:27:59.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:27:59.152 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:27:59.152 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:27:59.152 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:27:59.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:27:59.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:27:59.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:27:59.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:27:59.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:27:59.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:27:59.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:27:59.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:27:59.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:27:59.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:27:59.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:27:59.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:27:59.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-11-16T20:27:59.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:27:59.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c3e5fbec-6020-4f90-bacd-b4922b61d40e 2020-11-16T20:27:59.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 c3e5fbec-6020-4f90-bacd-b4922b61d40e' 2020-11-16T20:27:59.155 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 c3e5fbec-6020-4f90-bacd-b4922b61d40e 2020-11-16T20:27:59.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:27:59.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDP4LJfJcdfChAA1wlJIgXDYg3xhJdeOCVXow== 2020-11-16T20:27:59.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDP4LJfJcdfChAA1wlJIgXDYg3xhJdeOCVXow=="}' 2020-11-16T20:27:59.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c3e5fbec-6020-4f90-bacd-b4922b61d40e -i td/osd-backfill-recovery-log/0/new.json 2020-11-16T20:27:59.525 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:27:59.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-11-16T20:27:59.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDP4LJfJcdfChAA1wlJIgXDYg3xhJdeOCVXow== --osd-uuid c3e5fbec-6020-4f90-bacd-b4922b61d40e 2020-11-16T20:27:59.563 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:27:59.559+0000 7fd4e1f2fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:27:59.566 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:27:59.563+0000 7fd4e1f2fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:27:59.568 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:27:59.563+0000 7fd4e1f2fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:27:59.568 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:27:59.567+0000 7fd4e1f2fd80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:27:59.569 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:27:59.567+0000 7fd4e1f2fd80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:27:59.569 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:27:59.567+0000 7fd4e1f2fd80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:27:59.612 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:27:59.607+0000 7fd4e1f2fd80 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-11-16T20:28:04.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-11-16T20:28:04.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:28:04.227 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:28:04.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:28:04.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:28:04.731 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:28:04.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:28:04.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:28:04.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:28:04.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:28:04.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:28:04.761 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:04.759+0000 7f12c241cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:04.783+0000 7f12c241cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:04.783+0000 7f12c241cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:05.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:28:05.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:28:05.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:28:05.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:28:05.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:28:05.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:05.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:28:05.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:28:05.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:05.089 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:28:05.326 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:05.323+0000 7f12c241cd80 -1 Falling back to public interface 2020-11-16T20:28:05.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:28:06.193 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:06.191+0000 7f12c241cd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:28:06.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:28:06.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:06.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:28:06.457 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:28:06.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:06.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:28:06.902 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/91550,v1:127.0.0.1:6803/91550] [v2:127.0.0.1:6804/91550,v1:127.0.0.1:6805/91550] exists,up c3e5fbec-6020-4f90-bacd-b4922b61d40e 2020-11-16T20:28:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:28:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:28:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:28:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:28:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-11-16T20:28:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:28:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:28:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:28:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:28:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-11-16T20:28:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:28:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:28:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:28:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:28:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-11-16T20:28:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-11-16T20:28:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:28:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-11-16T20:28:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:28:06.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:28:06.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:28:06.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:28:06.907 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:28:06.907 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:28:06.907 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:28:06.908 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:28:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:28:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:28:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:28:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:28:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:28:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:28:06.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:28:06.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:28:06.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:28:06.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:28:06.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:28:06.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-11-16T20:28:06.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:28:06.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fc2a30dd-0968-4984-bd54-2c26145352d4 2020-11-16T20:28:06.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 fc2a30dd-0968-4984-bd54-2c26145352d4' 2020-11-16T20:28:06.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:28:06.911 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 fc2a30dd-0968-4984-bd54-2c26145352d4 2020-11-16T20:28:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDW4LJfP6I5NxAAukMTeBHS6HeaNnjqsaek1g== 2020-11-16T20:28:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDW4LJfP6I5NxAAukMTeBHS6HeaNnjqsaek1g=="}' 2020-11-16T20:28:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fc2a30dd-0968-4984-bd54-2c26145352d4 -i td/osd-backfill-recovery-log/1/new.json 2020-11-16T20:28:07.517 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:28:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-11-16T20:28:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDW4LJfP6I5NxAAukMTeBHS6HeaNnjqsaek1g== --osd-uuid fc2a30dd-0968-4984-bd54-2c26145352d4 2020-11-16T20:28:07.564 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:07.559+0000 7f5775b48d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:07.598 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:07.595+0000 7f5775b48d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:07.600 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:07.595+0000 7f5775b48d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:07.600 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:07.595+0000 7f5775b48d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:28:07.600 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:07.595+0000 7f5775b48d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:28:07.601 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:07.595+0000 7f5775b48d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:28:07.703 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:07.699+0000 7f5775b48d80 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-11-16T20:28:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-11-16T20:28:12.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:28:12.091 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:28:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:28:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:28:12.599 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:28:12.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:28:12.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:28:12.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:28:12.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:28:12.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:28:12.630 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:12.627+0000 7fb8a93b9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:12.627+0000 7fb8a93b9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:12.631+0000 7fb8a93b9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:12.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:28:12.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:28:12.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:28:12.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:28:12.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:28:12.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:12.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:28:12.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:12.991 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:28:12.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:28:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:13.171+0000 7fb8a93b9d80 -1 Falling back to public interface 2020-11-16T20:28:13.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:28:14.012 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:14.007+0000 7fb8a93b9d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:28:14.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:28:14.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:14.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:28:14.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:14.367 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:28:14.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:28:14.756 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/92544,v1:127.0.0.1:6811/92544] [v2:127.0.0.1:6812/92544,v1:127.0.0.1:6813/92544] exists,up fc2a30dd-0968-4984-bd54-2c26145352d4 2020-11-16T20:28:14.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:28:14.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:28:14.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:28:14.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:28:14.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-11-16T20:28:14.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:28:14.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:28:14.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:28:14.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:28:14.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-11-16T20:28:14.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:28:14.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:28:14.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:28:14.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:28:14.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-11-16T20:28:14.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-11-16T20:28:14.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:28:14.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-11-16T20:28:14.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:28:14.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:28:14.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:28:14.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:28:14.761 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:28:14.761 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:28:14.761 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:28:14.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:28:14.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:28:14.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:28:14.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:28:14.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:28:14.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:28:14.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:28:14.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:28:14.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:28:14.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:28:14.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:28:14.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:28:14.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-11-16T20:28:14.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:28:14.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2b93638e-e6c7-426a-b735-200489295e86 2020-11-16T20:28:14.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 2b93638e-e6c7-426a-b735-200489295e86' 2020-11-16T20:28:14.764 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 2b93638e-e6c7-426a-b735-200489295e86 2020-11-16T20:28:14.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:28:14.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDe4LJfwdF6LhAAeKvO838csQF4tYCFt/dfew== 2020-11-16T20:28:14.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDe4LJfwdF6LhAAeKvO838csQF4tYCFt/dfew=="}' 2020-11-16T20:28:14.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2b93638e-e6c7-426a-b735-200489295e86 -i td/osd-backfill-recovery-log/2/new.json 2020-11-16T20:28:15.282 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:28:15.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-11-16T20:28:15.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDe4LJfwdF6LhAAeKvO838csQF4tYCFt/dfew== --osd-uuid 2b93638e-e6c7-426a-b735-200489295e86 2020-11-16T20:28:15.328 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:15.327+0000 7f2e338a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:15.331 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:15.327+0000 7f2e338a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:15.333 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:15.331+0000 7f2e338a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:15.333 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:15.331+0000 7f2e338a3d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:28:15.334 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:15.331+0000 7f2e338a3d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:28:15.334 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:15.331+0000 7f2e338a3d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:28:15.376 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:15.375+0000 7f2e338a3d80 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-11-16T20:28:20.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-11-16T20:28:20.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:28:20.063 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:28:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:28:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:28:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:28:20.561 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:28:20.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:28:20.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:28:20.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:28:20.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:28:20.592 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:20.587+0000 7ffb94368d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:20.594 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:20.591+0000 7ffb94368d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:20.596 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:20.591+0000 7ffb94368d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:20.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:28:20.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:28:20.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:28:20.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:28:20.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:28:20.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:20.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:28:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:20.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:28:20.972 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:28:21.142 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:21.139+0000 7ffb94368d80 -1 Falling back to public interface 2020-11-16T20:28:21.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:28:21.989 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:21.987+0000 7ffb94368d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:28:22.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:28:22.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:22.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:28:22.334 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:28:22.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:22.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:28:22.776 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/93538,v1:127.0.0.1:6819/93538] [v2:127.0.0.1:6820/93538,v1:127.0.0.1:6821/93538] exists,up 2b93638e-e6c7-426a-b735-200489295e86 2020-11-16T20:28:22.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:28:22.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:28:22.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:28:22.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:28:22.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-11-16T20:28:22.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:28:22.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:28:22.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T20:28:22.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:28:22.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-11-16T20:28:22.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:28:22.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:28:22.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:28:22.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:28:22.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-11-16T20:28:22.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-11-16T20:28:22.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:28:22.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-11-16T20:28:22.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:28:22.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:28:22.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:28:22.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:28:22.785 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:28:22.786 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:28:22.786 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:28:22.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:28:22.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:28:22.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:28:22.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:28:22.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:28:22.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:28:22.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:28:22.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:28:22.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:28:22.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:28:22.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:28:22.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:28:22.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-11-16T20:28:22.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:28:22.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1de761f0-62ba-427a-b64c-e44a7bc4fbd2 2020-11-16T20:28:22.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 1de761f0-62ba-427a-b64c-e44a7bc4fbd2' 2020-11-16T20:28:22.791 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 1de761f0-62ba-427a-b64c-e44a7bc4fbd2 2020-11-16T20:28:22.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:28:22.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDm4LJf9IbxLxAA9Jo8Hj6PpsLQECxRblabww== 2020-11-16T20:28:22.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDm4LJf9IbxLxAA9Jo8Hj6PpsLQECxRblabww=="}' 2020-11-16T20:28:22.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1de761f0-62ba-427a-b64c-e44a7bc4fbd2 -i td/osd-backfill-recovery-log/3/new.json 2020-11-16T20:28:23.433 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T20:28:23.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-11-16T20:28:23.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDm4LJf9IbxLxAA9Jo8Hj6PpsLQECxRblabww== --osd-uuid 1de761f0-62ba-427a-b64c-e44a7bc4fbd2 2020-11-16T20:28:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:23.483+0000 7ff01b748d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:23.483+0000 7ff01b748d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:23.489 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:23.487+0000 7ff01b748d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:23.490 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:23.487+0000 7ff01b748d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:28:23.490 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:23.487+0000 7ff01b748d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:28:23.490 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:23.487+0000 7ff01b748d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:28:23.531 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:23.527+0000 7ff01b748d80 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-11-16T20:28:28.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-11-16T20:28:28.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:28:28.043 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T20:28:28.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T20:28:28.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:28:28.538 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T20:28:28.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T20:28:28.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:28:28.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:28:28.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:28:28.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:28:28.569 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:28.567+0000 7fb4e2f84d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:28.571 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:28.567+0000 7fb4e2f84d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:28.572 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:28.571+0000 7fb4e2f84d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:29.106 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:29.103+0000 7fb4e2f84d80 -1 Falling back to public interface 2020-11-16T20:28:29.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T20:28:29.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:28:29.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T20:28:29.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:28:29.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:28:29.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:29.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:28:29.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:29.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:28:29.129 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:28:29.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:28:30.009 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:29.999+0000 7fb4e2f84d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T20:28:30.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:28:30.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:30.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:28:30.503 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:28:30.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:30.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:28:30.888 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/94532,v1:127.0.0.1:6827/94532] [v2:127.0.0.1:6828/94532,v1:127.0.0.1:6829/94532] exists,up 1de761f0-62ba-427a-b64c-e44a7bc4fbd2 2020-11-16T20:28:30.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:28:30.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:28:30.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:28:30.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:28:30.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-11-16T20:28:30.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:28:30.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:28:30.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T20:28:30.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:28:30.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-11-16T20:28:30.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:28:30.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:28:30.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:28:30.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:28:30.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-11-16T20:28:30.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-11-16T20:28:30.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:28:30.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-11-16T20:28:30.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:28:30.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:28:30.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:28:30.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:28:30.894 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:28:30.894 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:28:30.894 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:28:30.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:28:30.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:28:30.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:28:30.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:28:30.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:28:30.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:28:30.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:28:30.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:28:30.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:28:30.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:28:30.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:28:30.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:28:30.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-11-16T20:28:30.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:28:30.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f2854c36-c982-4975-83c6-4c7c246e5b91 2020-11-16T20:28:30.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 f2854c36-c982-4975-83c6-4c7c246e5b91' 2020-11-16T20:28:30.900 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 f2854c36-c982-4975-83c6-4c7c246e5b91 2020-11-16T20:28:30.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:28:30.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDu4LJfRUqqNhAAThb2j3VpvKuMZRNs4tV5pQ== 2020-11-16T20:28:30.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDu4LJfRUqqNhAAThb2j3VpvKuMZRNs4tV5pQ=="}' 2020-11-16T20:28:30.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f2854c36-c982-4975-83c6-4c7c246e5b91 -i td/osd-backfill-recovery-log/4/new.json 2020-11-16T20:28:31.417 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T20:28:31.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-11-16T20:28:31.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDu4LJfRUqqNhAAThb2j3VpvKuMZRNs4tV5pQ== --osd-uuid f2854c36-c982-4975-83c6-4c7c246e5b91 2020-11-16T20:28:31.463 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:31.459+0000 7f4ecd346d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:31.466 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:31.463+0000 7f4ecd346d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:31.467 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:31.463+0000 7f4ecd346d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:31.467 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:31.463+0000 7f4ecd346d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:28:31.468 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:31.463+0000 7f4ecd346d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:28:31.468 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:31.463+0000 7f4ecd346d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:28:31.591 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:31.587+0000 7f4ecd346d80 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-11-16T20:28:36.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-11-16T20:28:36.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:28:36.247 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T20:28:36.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T20:28:36.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:28:36.755 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T20:28:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T20:28:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:28:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:28:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:28:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:28:36.788 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:36.783+0000 7eff5ae21d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:36.790 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:36.787+0000 7eff5ae21d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:36.791 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:36.787+0000 7eff5ae21d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:37.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T20:28:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:28:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T20:28:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:28:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:28:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:28:41.215 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:28:41.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:41.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:28:41.220 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:37.347+0000 7eff5ae21d80 -1 Falling back to public interface 2020-11-16T20:28:41.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:28:41.221 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:38.199+0000 7eff5ae21d80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T20:28:41.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:28:41.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:41.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:28:41.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:41.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:28:41.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:28:41.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:28:41.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:41.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:28:41.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:41.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:28:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:28:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:28:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:28:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:28:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-11-16T20:28:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:28:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:28:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T20:28:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:28:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-11-16T20:28:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:28:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:28:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:28:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:28:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-11-16T20:28:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-11-16T20:28:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:28:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-11-16T20:28:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:28:41.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:28:41.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:28:41.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:28:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:28:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:28:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:28:41.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:28:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:28:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:28:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:28:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:28:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:28:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:28:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:28:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:28:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:28:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:28:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:28:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-11-16T20:28:41.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:28:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1a2c82ca-8971-489d-89ac-94bf43506813 2020-11-16T20:28:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 1a2c82ca-8971-489d-89ac-94bf43506813' 2020-11-16T20:28:41.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:28:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD44LJfv2lTExAAuckM+qaexGVtOC4YzSxaLQ== 2020-11-16T20:28:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD44LJfv2lTExAAuckM+qaexGVtOC4YzSxaLQ=="}' 2020-11-16T20:28:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1a2c82ca-8971-489d-89ac-94bf43506813 -i td/osd-backfill-recovery-log/5/new.json 2020-11-16T20:28:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-11-16T20:28:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD44LJfv2lTExAAuckM+qaexGVtOC4YzSxaLQ== --osd-uuid 1a2c82ca-8971-489d-89ac-94bf43506813 2020-11-16T20:28:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:40.907+0000 7f48e3777d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:40.911+0000 7f48e3777d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:40.911+0000 7f48e3777d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:40.911+0000 7f48e3777d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:28:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:40.911+0000 7f48e3777d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:28:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:40.911+0000 7f48e3777d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:28:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:40.991+0000 7f48e3777d80 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-11-16T20:28:41.231 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:28:41.231 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:28:41.231 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/95526,v1:127.0.0.1:6835/95526] [v2:127.0.0.1:6836/95526,v1:127.0.0.1:6837/95526] exists,up f2854c36-c982-4975-83c6-4c7c246e5b91 2020-11-16T20:28:41.231 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 1a2c82ca-8971-489d-89ac-94bf43506813 2020-11-16T20:28:41.231 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T20:28:45.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-11-16T20:28:45.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:28:45.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T20:28:45.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:28:45.655 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T20:28:46.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T20:28:46.165 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T20:28:46.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:28:46.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:28:46.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:28:46.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:28:46.194 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:46.190+0000 7f0bd966ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:46.196 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:46.194+0000 7f0bd966ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:46.197 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:46.194+0000 7f0bd966ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:28:46.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T20:28:46.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:28:46.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T20:28:46.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:28:46.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:28:46.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:46.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:28:46.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:46.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T20:28:46.550 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:28:46.742 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:46.738+0000 7f0bd966ad80 -1 Falling back to public interface 2020-11-16T20:28:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:28:47.596 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:28:47.590+0000 7f0bd966ad80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T20:28:47.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:28:47.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:28:47.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:28:47.933 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:28:47.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:28:47.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T20:28:48.326 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/96546,v1:127.0.0.1:6843/96546] [v2:127.0.0.1:6844/96546,v1:127.0.0.1:6845/96546] exists,up 1a2c82ca-8971-489d-89ac-94bf43506813 2020-11-16T20:28:48.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:28:48.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:28:48.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:28:48.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-11-16T20:28:48.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T20:28:48.921 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T20:28:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:28:49.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-11-16T20:28:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-11-16T20:28:50.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:50.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-11-16T20:28:50.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:50.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-11-16T20:28:50.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:50.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-11-16T20:28:51.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:51.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-11-16T20:28:51.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:51.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-11-16T20:28:51.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:51.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-11-16T20:28:51.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:51.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-11-16T20:28:51.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:51.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-11-16T20:28:51.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:51.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-11-16T20:28:51.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:51.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-11-16T20:28:52.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:52.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-11-16T20:28:52.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:52.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-11-16T20:28:52.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:52.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-11-16T20:28:52.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:52.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-11-16T20:28:52.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:52.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-11-16T20:28:52.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:52.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-11-16T20:28:53.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-11-16T20:28:53.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:53.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-11-16T20:28:53.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:53.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-11-16T20:28:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-11-16T20:28:53.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:53.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-11-16T20:28:53.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:53.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-11-16T20:28:53.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:53.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-11-16T20:28:54.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:54.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-11-16T20:28:54.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:54.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-11-16T20:28:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-11-16T20:28:54.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:54.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-11-16T20:28:54.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:54.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-11-16T20:28:54.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:54.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-11-16T20:28:55.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:55.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-11-16T20:28:55.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:55.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-11-16T20:28:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:55.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-11-16T20:28:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-11-16T20:28:55.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:55.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-11-16T20:28:55.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:55.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-11-16T20:28:56.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:56.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-11-16T20:28:56.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:56.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-11-16T20:28:56.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:56.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-11-16T20:28:56.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:56.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-11-16T20:28:56.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:56.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-11-16T20:28:56.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:56.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-11-16T20:28:56.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:56.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-11-16T20:28:57.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:57.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-11-16T20:28:57.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:57.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-11-16T20:28:57.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:57.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-11-16T20:28:57.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:57.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-11-16T20:28:57.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:57.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-11-16T20:28:57.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:57.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-11-16T20:28:57.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:57.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-11-16T20:28:58.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:58.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-11-16T20:28:58.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:28:58.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-11-16T20:28:58.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-11-16T20:29:02.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-11-16T20:29:02.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-11-16T20:29:02.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-11-16T20:29:02.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-11-16T20:29:02.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-11-16T20:29:02.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-11-16T20:29:02.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-11-16T20:29:02.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-11-16T20:29:02.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-11-16T20:29:02.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-11-16T20:29:02.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-11-16T20:29:02.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-11-16T20:29:02.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-11-16T20:29:02.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-11-16T20:29:02.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-11-16T20:29:02.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-11-16T20:29:02.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-11-16T20:29:02.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-11-16T20:29:02.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-11-16T20:29:02.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-11-16T20:29:02.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-11-16T20:29:02.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-11-16T20:29:02.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-11-16T20:29:02.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-11-16T20:29:02.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-11-16T20:29:02.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-11-16T20:29:02.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-11-16T20:29:02.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:02.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-11-16T20:29:03.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:03.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-11-16T20:29:03.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:03.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-11-16T20:29:03.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:03.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-11-16T20:29:03.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:03.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-11-16T20:29:03.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:03.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-11-16T20:29:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-11-16T20:29:03.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:03.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-11-16T20:29:04.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:04.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-11-16T20:29:04.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:04.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-11-16T20:29:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-11-16T20:29:04.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:04.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-11-16T20:29:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:04.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-11-16T20:29:04.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:04.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-11-16T20:29:05.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:05.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-11-16T20:29:05.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:05.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-11-16T20:29:05.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:05.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-11-16T20:29:05.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:05.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-11-16T20:29:05.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:05.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-11-16T20:29:05.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:05.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-11-16T20:29:05.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:05.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-11-16T20:29:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-11-16T20:29:06.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:06.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-11-16T20:29:06.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:06.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-11-16T20:29:06.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:06.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-11-16T20:29:06.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:06.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-11-16T20:29:06.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:06.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-11-16T20:29:07.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:07.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-11-16T20:29:07.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:07.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-11-16T20:29:07.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:07.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-11-16T20:29:07.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:07.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-11-16T20:29:07.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:07.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-11-16T20:29:07.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:07.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-11-16T20:29:07.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:07.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-11-16T20:29:08.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:08.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-11-16T20:29:08.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:08.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-11-16T20:29:08.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:08.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-11-16T20:29:08.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:08.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-11-16T20:29:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:08.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-11-16T20:29:08.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:08.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-11-16T20:29:08.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:08.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-11-16T20:29:09.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:09.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-11-16T20:29:09.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:09.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-11-16T20:29:09.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:09.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-11-16T20:29:09.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:09.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-11-16T20:29:09.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:09.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-11-16T20:29:09.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:09.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-11-16T20:29:10.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:10.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-11-16T20:29:10.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:10.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-11-16T20:29:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-11-16T20:29:10.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:10.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-11-16T20:29:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-11-16T20:29:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-11-16T20:29:14.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-11-16T20:29:14.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-11-16T20:29:14.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-11-16T20:29:14.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-11-16T20:29:14.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-11-16T20:29:14.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-11-16T20:29:14.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-11-16T20:29:14.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-11-16T20:29:14.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-11-16T20:29:14.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-11-16T20:29:14.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-11-16T20:29:14.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-11-16T20:29:14.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-11-16T20:29:14.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-11-16T20:29:14.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-11-16T20:29:14.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-11-16T20:29:14.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:29:14.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-11-16T20:29:14.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-11-16T20:29:14.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-11-16T20:29:14.870 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:29:14.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-11-16T20:29:14.871 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-11-16T20:29:14.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-11-16T20:29:14.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-11-16T20:29:15.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-11-16T20:29:15.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:29:15.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:29:15.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:29:15.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:29:15.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:29:15.471 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:29:15.471 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:29:15.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:29:15.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:29:15.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:29:15.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:29:15.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:29:15.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:29:15.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:29:15.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:29:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:29:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:29:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:29:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:29:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T20:29:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T20:29:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:29:16.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:16.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:29:16.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836496 2020-11-16T20:29:16.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836496 2020-11-16T20:29:16.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496' 2020-11-16T20:29:16.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:16.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:29:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672975 2020-11-16T20:29:16.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672975 2020-11-16T20:29:16.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672975' 2020-11-16T20:29:16.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:16.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:29:16.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542157 2020-11-16T20:29:16.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542157 2020-11-16T20:29:16.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-60129542157' 2020-11-16T20:29:16.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:16.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:29:16.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378637 2020-11-16T20:29:16.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378637 2020-11-16T20:29:16.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-60129542157 3-81604378637' 2020-11-16T20:29:16.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:16.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:29:16.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215115 2020-11-16T20:29:16.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215115 2020-11-16T20:29:16.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-60129542157 3-81604378637 4-103079215115' 2020-11-16T20:29:16.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:16.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T20:29:17.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051592 2020-11-16T20:29:17.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051592 2020-11-16T20:29:17.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-60129542157 3-81604378637 4-103079215115 5-124554051592' 2020-11-16T20:29:17.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:29:17.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:29:17.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836496 2020-11-16T20:29:17.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:29:17.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:29:17.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2020-11-16T20:29:17.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836496 2020-11-16T20:29:17.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2020-11-16T20:29:17.155 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836496 2020-11-16T20:29:17.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:29:17.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836496 2020-11-16T20:29:17.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:29:17.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672975 2020-11-16T20:29:17.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:29:17.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:29:17.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:29:17.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672975 2020-11-16T20:29:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672975 2020-11-16T20:29:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2020-11-16T20:29:17.563 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672975 2020-11-16T20:29:17.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:29:17.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672975 -lt 42949672975 2020-11-16T20:29:17.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:29:17.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542157 2020-11-16T20:29:17.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:29:17.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:29:17.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542157 2020-11-16T20:29:17.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:29:17.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542157 2020-11-16T20:29:17.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2020-11-16T20:29:17.950 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542157 2020-11-16T20:29:17.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:29:18.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542157 -lt 60129542157 2020-11-16T20:29:18.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:29:18.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378637 2020-11-16T20:29:18.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:29:18.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:29:18.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378637 2020-11-16T20:29:18.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:29:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378637 2020-11-16T20:29:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378637' 2020-11-16T20:29:18.342 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378637 2020-11-16T20:29:18.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:29:18.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378637 -lt 81604378637 2020-11-16T20:29:18.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:29:18.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215115 2020-11-16T20:29:18.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:29:18.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:29:18.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:29:18.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215115 2020-11-16T20:29:18.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215115 2020-11-16T20:29:18.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215115' 2020-11-16T20:29:18.817 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215115 2020-11-16T20:29:18.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:29:19.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215115 -lt 103079215115 2020-11-16T20:29:19.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:29:19.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051592 2020-11-16T20:29:19.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:29:19.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T20:29:19.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051592 2020-11-16T20:29:19.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:29:19.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051592 2020-11-16T20:29:19.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051592' 2020-11-16T20:29:19.260 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051592 2020-11-16T20:29:19.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T20:29:19.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051592 -lt 124554051592 2020-11-16T20:29:19.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:29:19.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:19.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:20.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:29:20.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:20.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:20.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:20.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:20.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:20.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:20.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:20.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:20.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:20.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:20.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:20.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:20.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:29:20.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:29:20.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:29:20.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:20.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:21.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:21.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:21.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:21.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:21.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:21.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:21.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:21.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:21.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:21.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:21.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:21.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:21.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:21.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:21.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:21.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:21.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:21.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:21.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:21.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:21.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:22.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6553 2020-11-16T20:29:22.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6553 '!=' null 2020-11-16T20:29:22.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:22.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:22.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:22.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:22.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:22.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:22.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:22.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:22.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:22.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:22.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:22.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:22.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:22.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:22.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:23.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:23.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:23.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:23.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:23.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:23.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:23.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:23.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:23.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:23.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24726 2020-11-16T20:29:23.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24726 '!=' null 2020-11-16T20:29:23.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:23.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:23.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:23.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:23.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:23.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:23.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:23.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:23.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:23.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:24.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:24.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:24.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:24.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:24.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:24.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:24.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:24.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:24.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:24.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:24.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:24.822 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:24.822 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20104 2020-11-16T20:29:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20104 '!=' null 2020-11-16T20:29:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:25.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:25.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:25.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:25.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:25.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:25.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:25.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:25.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:25.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:25.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:25.737 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:25.737 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:26.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:26.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:26.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:26.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:26.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:26.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:26.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:26.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:26.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:26.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20104 2020-11-16T20:29:26.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20104 '!=' null 2020-11-16T20:29:26.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:26.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:26.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:26.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:26.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:26.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:26.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:26.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:26.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:26.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:26.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:27.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:27.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:27.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:27.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:27.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:27.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:27.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:27.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:27.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:27.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:27.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:27.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:27.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:28.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17480 2020-11-16T20:29:28.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17480 '!=' null 2020-11-16T20:29:28.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:28.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:28.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:28.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:28.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:28.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:28.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:28.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:28.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:28.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:28.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:28.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:28.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:28.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:28.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:29.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:29.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:29.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:29.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:29.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:29.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:29.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:29.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:29.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:29.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=29671 2020-11-16T20:29:29.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 29671 '!=' null 2020-11-16T20:29:29.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:29.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:29.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:29.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:29.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:29.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:29.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:29.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:29.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:29.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:29.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:30.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:30.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:30.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:30.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:30.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:30.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:30.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:30.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:30.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:30.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:30.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:30.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:30.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:31.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26375 2020-11-16T20:29:31.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26375 '!=' null 2020-11-16T20:29:31.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:31.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:31.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:31.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:31.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:31.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:31.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:31.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:31.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:31.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:31.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:31.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:31.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:31.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:31.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:32.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:32.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:32.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26375 2020-11-16T20:29:32.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26375 '!=' null 2020-11-16T20:29:32.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:32.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:32.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:32.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:32.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:32.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:32.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:32.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:32.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:32.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:32.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:32.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:32.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:32.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:32.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:33.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:33.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:33.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:33.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:33.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:33.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:33.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:33.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:33.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:34.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=39563 2020-11-16T20:29:34.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 39563 '!=' null 2020-11-16T20:29:34.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:34.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:34.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:34.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:34.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:34.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:34.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:34.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:34.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:34.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:34.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:34.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:34.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:34.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:34.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:34.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:34.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:34.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:34.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:34.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:34.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:34.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:35.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:35.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:35.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26375 2020-11-16T20:29:35.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26375 '!=' null 2020-11-16T20:29:35.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:35.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:35.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:35.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:35.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:35.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:35.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:35.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:35.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:35.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:35.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:36.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:36.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:36.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:36.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:36.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:36.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:36.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:36.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:36.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:36.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:36.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:36.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:36.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:36.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26375 2020-11-16T20:29:36.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26375 '!=' null 2020-11-16T20:29:36.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:36.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:36.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:37.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:37.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:37.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:37.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:37.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:37.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:37.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:37.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:37.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:37.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:37.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:37.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:37.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:37.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:37.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:38.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26376 2020-11-16T20:29:38.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26376 '!=' null 2020-11-16T20:29:42.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:42.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:42.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:42.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:42.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:42.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:42.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:42.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:42.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:42.590 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:42.590 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:42.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:42.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:42.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:42.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:42.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:42.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:42.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:42.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:42.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:42.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:42.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:42.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:42.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:42.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=38904 2020-11-16T20:29:42.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 38904 '!=' null 2020-11-16T20:29:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:42.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:42.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:42.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:42.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:42.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:42.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:42.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:42.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:42.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:42.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:42.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:42.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:42.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:42.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:42.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:42.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25717 2020-11-16T20:29:42.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25717 '!=' null 2020-11-16T20:29:42.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:42.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:42.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:42.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:42.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:42.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:42.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:42.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:42.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:42.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:42.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:42.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:42.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:42.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:42.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:42.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:42.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:42.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:42.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:42.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:42.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:42.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:42.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25717 2020-11-16T20:29:42.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25717 '!=' null 2020-11-16T20:29:42.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:42.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:42.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:42.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:42.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:42.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:42.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:42.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:42.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:42.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:42.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:43.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:43.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:43.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:43.230 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:43.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:43.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:43.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:43.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:43.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:43.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:43.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:43.698 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:43.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:44.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=39563 2020-11-16T20:29:48.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 39563 '!=' null 2020-11-16T20:29:48.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:48.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:48.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:48.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:48.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:48.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:48.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:48.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:48.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:48.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:48.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:48.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:48.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:48.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:48.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:48.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:48.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:48.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:48.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:48.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:48.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:48.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:48.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:48.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:48.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26375 2020-11-16T20:29:48.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26375 '!=' null 2020-11-16T20:29:48.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:48.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:48.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:48.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:48.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:48.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:48.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:48.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:48.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:48.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:48.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:48.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:48.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:48.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:48.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:48.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:48.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:48.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:48.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:48.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:48.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:48.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:48.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:48.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:48.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26375 2020-11-16T20:29:48.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26375 '!=' null 2020-11-16T20:29:48.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:48.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:48.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:48.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:48.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:48.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:48.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:48.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:48.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:48.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:48.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:48.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:48.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:48.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:48.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:48.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:48.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:48.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:48.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:48.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:48.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:48.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:48.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:48.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26375 2020-11-16T20:29:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26375 '!=' null 2020-11-16T20:29:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:48.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:48.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:48.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:48.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:48.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:48.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:48.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:49.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:49.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:49.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:49.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:49.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:50.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40223 2020-11-16T20:29:54.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40223 '!=' null 2020-11-16T20:29:54.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:54.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:54.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:54.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:54.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:54.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:54.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:54.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:54.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:54.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:54.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:54.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:54.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:54.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:54.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:54.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:54.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:54.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:54.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:54.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27695 2020-11-16T20:29:54.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27695 '!=' null 2020-11-16T20:29:54.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:54.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:54.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:54.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:54.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:54.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:54.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:54.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:54.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:54.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:54.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:54.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:54.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:54.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:54.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:54.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:54.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:54.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:54.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:54.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:54.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:54.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:54.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:54.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:54.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27695 2020-11-16T20:29:54.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27695 '!=' null 2020-11-16T20:29:54.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:54.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:54.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:54.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:54.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:54.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:54.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:54.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:54.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:54.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:54.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:54.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:29:54.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:54.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:54.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:54.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:29:54.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:29:54.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:29:54.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:29:54.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:29:54.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:29:54.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:29:54.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:29:54.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:29:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41542 2020-11-16T20:29:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41542 '!=' null 2020-11-16T20:29:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:29:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:29:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:29:54.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:29:54.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:29:54.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:29:54.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:29:54.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:29:54.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:29:54.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:29:54.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:29:54.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:29:54.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:29:54.882 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:29:54.882 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:29:55.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:29:55.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:29:55.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:29:55.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2020-11-16T20:29:55.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:29:55.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:29:55.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:29:55.717 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:29:55.718 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:29:55.718 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:29:55.718 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T20:29:55.718 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T20:29:55.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:29:55.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:55.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:29:55.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836506 2020-11-16T20:29:55.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836506 2020-11-16T20:29:55.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506' 2020-11-16T20:29:55.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:55.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:29:56.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672985 2020-11-16T20:29:56.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672985 2020-11-16T20:29:56.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-42949672985' 2020-11-16T20:29:56.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:56.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:29:56.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542167 2020-11-16T20:29:56.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542167 2020-11-16T20:29:56.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542167' 2020-11-16T20:29:56.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:56.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:29:56.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378646 2020-11-16T20:29:56.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378646 2020-11-16T20:29:56.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542167 3-81604378646' 2020-11-16T20:29:56.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:56.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:29:56.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215125 2020-11-16T20:29:56.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215125 2020-11-16T20:29:56.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542167 3-81604378646 4-103079215125' 2020-11-16T20:29:56.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:29:56.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T20:29:56.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051602 2020-11-16T20:29:56.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051602 2020-11-16T20:29:56.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542167 3-81604378646 4-103079215125 5-124554051602' 2020-11-16T20:29:56.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:29:56.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836506 2020-11-16T20:29:56.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:29:56.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:29:56.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836506 2020-11-16T20:29:56.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:29:56.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836506 2020-11-16T20:29:56.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2020-11-16T20:29:56.714 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836506 2020-11-16T20:29:56.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:29:57.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836506 -lt 21474836506 2020-11-16T20:29:57.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:29:57.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672985 2020-11-16T20:29:57.098 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:29:57.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:29:57.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672985 2020-11-16T20:29:57.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:29:57.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672985 2020-11-16T20:29:57.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2020-11-16T20:29:57.105 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672985 2020-11-16T20:29:57.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:29:57.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672985 -lt 42949672985 2020-11-16T20:29:57.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:29:57.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542167 2020-11-16T20:29:57.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:29:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:29:57.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542167 2020-11-16T20:29:57.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:29:57.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542167 2020-11-16T20:29:57.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542167' 2020-11-16T20:29:57.517 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542167 2020-11-16T20:29:57.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:29:57.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542167 -lt 60129542167 2020-11-16T20:29:57.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:29:57.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378646 2020-11-16T20:29:57.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:29:57.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:29:57.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378646 2020-11-16T20:29:57.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:30:02.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378646 2020-11-16T20:30:02.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378646' 2020-11-16T20:30:02.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:30:02.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378647 -lt 81604378646 2020-11-16T20:30:02.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:30:02.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215125 2020-11-16T20:30:02.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:30:02.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:30:02.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:30:02.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215125 2020-11-16T20:30:02.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215125 2020-11-16T20:30:02.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215125' 2020-11-16T20:30:02.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:30:02.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215125 -lt 103079215125 2020-11-16T20:30:02.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:30:02.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051602 2020-11-16T20:30:02.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:30:02.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T20:30:02.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051602 2020-11-16T20:30:02.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:30:02.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051602 2020-11-16T20:30:02.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051602' 2020-11-16T20:30:02.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T20:30:02.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051602 -lt 124554051602 2020-11-16T20:30:02.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2020-11-16T20:30:02.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2020-11-16T20:30:02.139 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:30:02.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2020-11-16T20:30:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2020-11-16T20:30:02.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:30:02.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:30:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:30:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:30:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:30:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:30:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2020-11-16T20:30:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-11-16T20:30:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-11-16T20:30:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T20:30:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=4 2020-11-16T20:30:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T20:30:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-11-16T20:30:02.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-11-16T20:30:02.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-11-16T20:30:02.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=2 2020-11-16T20:30:02.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:85: _common_test: '[' 2 '!=' 2 ']' 2020-11-16T20:30:02.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-11-16T20:30:02.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=8 2020-11-16T20:30:02.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:90: _common_test: '[' 8 '!=' 8 ']' 2020-11-16T20:30:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-11-16T20:30:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-11-16T20:30:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2020-11-16T20:30:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-11-16T20:30:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:30:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-11-16T20:30:02.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:30:02.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:30:02.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:30:02.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:30:02.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:30:02.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:30:02.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:30:02.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:30:02.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:30:02.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:30:02.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:30:02.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:30:02.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:30:02.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:30:02.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:30:02.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:30:02.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:30:02.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:30:02.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-11-16T20:30:02.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:30:02.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:02.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90694 2020-11-16T20:30:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:30:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:30:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-11-16T20:30:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-11-16T20:30:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-11-16T20:30:02.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:02.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:30:02.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-11-16T20:30:02.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:30:02.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:30:02.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:30:02.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:30:02.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:30:02.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:30:02.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:30:02.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:30:02.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:30:02.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:30:02.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:30:02.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:30:02.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:30:02.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:30:02.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:30:02.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:30:02.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:30:02.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:30:02.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-11-16T20:30:02.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:30:02.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:02.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:02.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90694 2020-11-16T20:30:02.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:30:02.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:30:02.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-11-16T20:30:02.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:30:02.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:02.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:02.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.90694 2020-11-16T20:30:02.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:30:02.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T20:30:02.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:30:02.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-11-16T20:30:02.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_2 td/osd-backfill-recovery-log 2020-11-16T20:30:02.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:113: TEST_backfill_log_2: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:02.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:115: TEST_backfill_log_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2 148 150 2020-11-16T20:30:02.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:02.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-11-16T20:30:02.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-11-16T20:30:02.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=148 2020-11-16T20:30:02.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-11-16T20:30:02.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-11-16T20:30:02.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-11-16T20:30:02.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-11-16T20:30:02.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:02.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:30:02.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:30:02.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:30:02.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-11-16T20:30:02.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-11-16T20:30:02.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:30:02.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:30:02.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:30:02.155 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:30:02.155 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:02.156 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:02.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:02.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:30:02.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:30:02.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:30:02.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:30:02.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:30:02.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:30:02.157 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:30:02.157 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:30:02.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:30:02.157 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:30:02.157 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:30:02.157 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:02.158 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:02.158 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90694/ceph-mon.a.asok 2020-11-16T20:30:02.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:30:02.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.90694/ceph-mon.a.asok config get fsid 2020-11-16T20:30:02.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:30:02.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:30:02.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:30:02.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:30:02.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:30:02.159 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:30:02.159 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:30:02.159 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:30:02.159 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:30:02.159 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:02.159 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:02.160 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90694/ceph-mon.a.asok 2020-11-16T20:30:02.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:30:02.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.90694/ceph-mon.a.asok config get mon_host 2020-11-16T20:30:02.160 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378646 2020-11-16T20:30:02.160 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215125 2020-11-16T20:30:02.160 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051602 2020-11-16T20:30:02.160 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:30:02.161 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_log_t": { 2020-11-16T20:30:02.161 INFO:tasks.workunit.client.0.smithi189.stdout: "head": "32'150", 2020-11-16T20:30:02.161 INFO:tasks.workunit.client.0.smithi189.stdout: "tail": "32'148", 2020-11-16T20:30:02.161 INFO:tasks.workunit.client.0.smithi189.stdout: "log": [ 2020-11-16T20:30:02.161 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:30:02.161 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:30:02.161 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-11-16T20:30:02.162 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'149", 2020-11-16T20:30:02.162 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:30:02.162 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4633.0:1", 2020-11-16T20:30:02.162 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:30:02.162 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:29:13.240387+0000", 2020-11-16T20:30:02.162 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:30:02.162 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:30:02.162 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:30:02.163 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:30:02.163 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:30:02.163 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:30:02.163 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:30:02.163 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:30:02.163 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:30:02.163 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:30:02.164 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:30:02.164 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:30:02.164 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:30:02.164 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:30:02.164 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:30:02.164 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:30:02.164 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:30:02.164 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:30:02.165 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-11-16T20:30:02.165 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'150", 2020-11-16T20:30:02.165 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:30:02.165 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4636.0:1", 2020-11-16T20:30:02.165 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:30:02.165 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:29:13.405930+0000", 2020-11-16T20:30:02.165 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:30:02.166 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:30:02.166 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:30:02.166 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:30:02.166 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:30:02.166 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:30:02.166 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:30:02.166 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:30:02.166 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:30:02.167 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:30:02.167 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:30:02.167 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:30:02.167 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:30:02.167 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:30:02.167 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:30:02.167 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:30:02.168 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:30:02.168 INFO:tasks.workunit.client.0.smithi189.stdout: "dups": [ 2020-11-16T20:30:02.168 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:30:02.168 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4609.0:1", 2020-11-16T20:30:02.168 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'141", 2020-11-16T20:30:02.168 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "141", 2020-11-16T20:30:02.168 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:30:02.168 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:30:02.169 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:30:02.169 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4612.0:1", 2020-11-16T20:30:02.169 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'142", 2020-11-16T20:30:02.169 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "142", 2020-11-16T20:30:02.169 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:30:02.169 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:30:02.169 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:30:02.170 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4615.0:1", 2020-11-16T20:30:02.170 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'143", 2020-11-16T20:30:02.170 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "143", 2020-11-16T20:30:02.170 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:30:02.170 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:30:02.170 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:30:02.170 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4618.0:1", 2020-11-16T20:30:02.170 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'144", 2020-11-16T20:30:02.171 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "144", 2020-11-16T20:30:02.171 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:30:02.171 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:30:02.171 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:30:02.171 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4621.0:1", 2020-11-16T20:30:02.171 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'145", 2020-11-16T20:30:02.171 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "145", 2020-11-16T20:30:02.172 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:30:02.172 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:30:02.172 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:30:02.172 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4624.0:1", 2020-11-16T20:30:02.172 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'146", 2020-11-16T20:30:02.172 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "146", 2020-11-16T20:30:02.172 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:30:02.172 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:30:02.173 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:30:02.173 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4627.0:1", 2020-11-16T20:30:02.173 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'147", 2020-11-16T20:30:02.173 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "147", 2020-11-16T20:30:02.173 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:30:02.173 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:30:02.173 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:30:02.174 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4630.0:1", 2020-11-16T20:30:02.174 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'148", 2020-11-16T20:30:02.174 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "148", 2020-11-16T20:30:02.174 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:30:02.174 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:30:02.174 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:30:02.174 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:30:02.175 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_missing_t": { 2020-11-16T20:30:02.175 INFO:tasks.workunit.client.0.smithi189.stdout: "missing": [], 2020-11-16T20:30:02.175 INFO:tasks.workunit.client.0.smithi189.stdout: "may_include_deletes": true 2020-11-16T20:30:02.175 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:30:02.175 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:30:02.175 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to max 2 2020-11-16T20:30:02.175 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to copy log version 32'150 2020-11-16T20:30:02.176 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to copy log version 32'149 2020-11-16T20:30:02.176 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after earliest_dup_version 141 2020-11-16T20:30:02.176 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'141 2020-11-16T20:30:02.176 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'142 2020-11-16T20:30:02.176 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'143 2020-11-16T20:30:02.176 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'144 2020-11-16T20:30:02.176 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'145 2020-11-16T20:30:02.177 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'146 2020-11-16T20:30:02.177 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'147 2020-11-16T20:30:02.177 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'148 2020-11-16T20:30:02.177 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to max 2 2020-11-16T20:30:02.177 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to copy log version 32'150 2020-11-16T20:30:02.177 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to copy log version 32'149 2020-11-16T20:30:02.177 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after earliest_dup_version 141 2020-11-16T20:30:02.177 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'141 2020-11-16T20:30:02.178 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'142 2020-11-16T20:30:02.178 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'143 2020-11-16T20:30:02.178 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'144 2020-11-16T20:30:02.178 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'145 2020-11-16T20:30:02.178 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'146 2020-11-16T20:30:02.178 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'147 2020-11-16T20:30:02.178 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'148 2020-11-16T20:30:02.179 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to max 2 2020-11-16T20:30:02.179 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to copy log version 32'150 2020-11-16T20:30:02.179 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to copy log version 32'149 2020-11-16T20:30:02.179 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after earliest_dup_version 141 2020-11-16T20:30:02.179 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'141 2020-11-16T20:30:02.179 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'142 2020-11-16T20:30:02.179 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'143 2020-11-16T20:30:02.180 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'144 2020-11-16T20:30:02.180 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'145 2020-11-16T20:30:02.180 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'146 2020-11-16T20:30:02.180 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'147 2020-11-16T20:30:02.180 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:15.498+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'148 2020-11-16T20:30:02.180 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_after v 32'150 2020-11-16T20:30:02.180 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_up_to/copy_after earliest_dup_version 141 2020-11-16T20:30:02.181 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'141 2020-11-16T20:30:02.181 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'142 2020-11-16T20:30:02.181 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'143 2020-11-16T20:30:02.181 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'144 2020-11-16T20:30:02.181 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'145 2020-11-16T20:30:02.181 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'146 2020-11-16T20:30:02.181 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'147 2020-11-16T20:30:02.181 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'148 2020-11-16T20:30:02.182 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'149 2020-11-16T20:30:02.182 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:29:53.950+0000 7f12a08a8700 20 copy_up_to/copy_after copy dup from log version 32'150 2020-11-16T20:30:02.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-11-16T20:30:02.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:02.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:30:02.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:30:02.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:30:02.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-11-16T20:30:02.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:30:03.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:30:03.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:30:03.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:30:03.084 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:30:03.084 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:03.084 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:03.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:03.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:30:03.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:30:03.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-11-16T20:30:03.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-11-16T20:30:03.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-11-16T20:30:03.112 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-11-16T20:30:03.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-11-16T20:30:03.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:30:03.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-11-16T20:30:03.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:03.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:30:03.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:30:03.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:30:03.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-11-16T20:30:03.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:30:03.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:30:03.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:30:03.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:30:03.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-11-16T20:30:03.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-11-16T20:30:03.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:30:03.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-11-16T20:30:03.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:30:03.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:30:03.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:30:03.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:30:03.119 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:30:03.119 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:03.119 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:03.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:03.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:03.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:30:03.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:30:03.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:30:03.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:30:03.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:30:03.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:30:03.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:30:03.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:03.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:30:03.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:30:03.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-11-16T20:30:03.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:30:03.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e4ecd23d-cf29-48c0-9158-6f9b674d27b3 2020-11-16T20:30:03.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 e4ecd23d-cf29-48c0-9158-6f9b674d27b3' 2020-11-16T20:30:03.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:30:03.122 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 e4ecd23d-cf29-48c0-9158-6f9b674d27b3 2020-11-16T20:30:03.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBL4bJfxp45CBAACjq433brhu7DHl8lXkF6iw== 2020-11-16T20:30:03.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBL4bJfxp45CBAACjq433brhu7DHl8lXkF6iw=="}' 2020-11-16T20:30:03.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e4ecd23d-cf29-48c0-9158-6f9b674d27b3 -i td/osd-backfill-recovery-log/0/new.json 2020-11-16T20:30:03.478 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:30:03.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-11-16T20:30:03.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBL4bJfxp45CBAACjq433brhu7DHl8lXkF6iw== --osd-uuid e4ecd23d-cf29-48c0-9158-6f9b674d27b3 2020-11-16T20:30:03.523 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:03.517+0000 7f78dc058d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:03.526 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:03.521+0000 7f78dc058d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:03.527 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:03.521+0000 7f78dc058d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:03.527 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:03.525+0000 7f78dc058d80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:30:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:03.525+0000 7f78dc058d80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:30:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:03.525+0000 7f78dc058d80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:30:03.585 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:03.581+0000 7f78dc058d80 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-11-16T20:30:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-11-16T20:30:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:30:08.163 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:30:08.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:30:08.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:30:08.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:30:08.650 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:30:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:30:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:30:08.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:30:08.680 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:08.677+0000 7fda9c645d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:08.707 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:08.701+0000 7fda9c645d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:08.708 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:08.705+0000 7fda9c645d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:09.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:30:13.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:30:13.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:30:13.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:30:13.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:30:13.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:13.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:30:13.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:13.183 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:30:13.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:30:13.188 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:09.253+0000 7fda9c645d80 -1 Falling back to public interface 2020-11-16T20:30:13.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:30:13.188 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:10.133+0000 7fda9c645d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:30:13.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:30:13.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:13.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:30:13.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:13.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:30:13.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:30:13.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:30:13.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:30:13.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:30:13.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-11-16T20:30:13.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:13.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:30:13.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:30:13.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:30:13.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-11-16T20:30:13.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:30:13.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:30:13.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:30:13.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:30:13.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-11-16T20:30:13.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-11-16T20:30:13.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:30:13.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-11-16T20:30:13.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:30:13.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:30:13.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:30:13.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:30:13.192 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:30:13.192 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:13.192 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:13.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:13.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:13.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:30:13.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:30:13.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:30:13.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:30:13.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:30:13.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:30:13.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:30:13.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:13.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:30:13.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:30:13.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-11-16T20:30:13.194 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:30:13.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7d7b9c45-74db-482e-b4d8-91828c99b015 2020-11-16T20:30:13.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 7d7b9c45-74db-482e-b4d8-91828c99b015' 2020-11-16T20:30:13.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:30:13.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBS4bJfIArcNBAAvBB0DS7zQYiS2ASi8MuUrQ== 2020-11-16T20:30:13.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBS4bJfIArcNBAAvBB0DS7zQYiS2ASi8MuUrQ=="}' 2020-11-16T20:30:13.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7d7b9c45-74db-482e-b4d8-91828c99b015 -i td/osd-backfill-recovery-log/1/new.json 2020-11-16T20:30:13.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-11-16T20:30:13.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBS4bJfIArcNBAAvBB0DS7zQYiS2ASi8MuUrQ== --osd-uuid 7d7b9c45-74db-482e-b4d8-91828c99b015 2020-11-16T20:30:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:11.533+0000 7f84f5568d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:11.537+0000 7f84f5568d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:11.537+0000 7f84f5568d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:11.537+0000 7f84f5568d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:30:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:11.537+0000 7f84f5568d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:30:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:11.537+0000 7f84f5568d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:30:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:11.589+0000 7f84f5568d80 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-11-16T20:30:13.197 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:30:13.197 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/103981,v1:127.0.0.1:6803/103981] [v2:127.0.0.1:6804/103981,v1:127.0.0.1:6805/103981] exists,up e4ecd23d-cf29-48c0-9158-6f9b674d27b3 2020-11-16T20:30:13.197 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 7d7b9c45-74db-482e-b4d8-91828c99b015 2020-11-16T20:30:13.197 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:30:15.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-11-16T20:30:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:30:15.986 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:30:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:30:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:30:16.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:30:16.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:30:16.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:30:16.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:30:16.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:16.483 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:30:16.505 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:16.501+0000 7fa87c34dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:16.508 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:16.505+0000 7fa87c34dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:16.509 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:16.505+0000 7fa87c34dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:16.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:30:16.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:30:16.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:30:16.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:30:16.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:30:16.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:16.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:30:16.859 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:30:16.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:16.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:30:17.053 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:17.049+0000 7fa87c34dd80 -1 Falling back to public interface 2020-11-16T20:30:17.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:30:17.966 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:17.957+0000 7fa87c34dd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:30:18.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:30:18.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:18.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:30:18.252 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:30:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:30:18.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:30:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:30:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:30:19.656 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:30:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:30:20.062 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/104975,v1:127.0.0.1:6811/104975] [v2:127.0.0.1:6812/104975,v1:127.0.0.1:6813/104975] exists,up 7d7b9c45-74db-482e-b4d8-91828c99b015 2020-11-16T20:30:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:30:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:30:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:30:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:30:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-11-16T20:30:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:30:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:30:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:30:20.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-11-16T20:30:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:30:20.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:30:24.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:30:24.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:30:24.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-11-16T20:30:24.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-11-16T20:30:24.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:30:24.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-11-16T20:30:24.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:30:24.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:30:24.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:30:24.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:30:24.198 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:30:24.198 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:24.198 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:24.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:24.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:24.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:30:24.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:30:24.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:30:24.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:30:24.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:30:24.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:30:24.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:30:24.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:24.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:30:24.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:30:24.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-11-16T20:30:24.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:30:24.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fad13ab8-e2c9-4978-a5bc-122dbb18f3cc 2020-11-16T20:30:24.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 fad13ab8-e2c9-4978-a5bc-122dbb18f3cc' 2020-11-16T20:30:24.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:30:24.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBc4bJfTW8DBRAARqqovwZOG5VMrDs/P2fiDw== 2020-11-16T20:30:24.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBc4bJfTW8DBRAARqqovwZOG5VMrDs/P2fiDw=="}' 2020-11-16T20:30:24.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fad13ab8-e2c9-4978-a5bc-122dbb18f3cc -i td/osd-backfill-recovery-log/2/new.json 2020-11-16T20:30:24.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-11-16T20:30:24.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBc4bJfTW8DBRAARqqovwZOG5VMrDs/P2fiDw== --osd-uuid fad13ab8-e2c9-4978-a5bc-122dbb18f3cc 2020-11-16T20:30:24.201 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:20.605+0000 7f2adf94ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:24.202 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:20.605+0000 7f2adf94ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:24.202 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:20.609+0000 7f2adf94ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:24.202 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:20.609+0000 7f2adf94ad80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:30:24.202 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:20.609+0000 7f2adf94ad80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:30:24.202 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:20.609+0000 7f2adf94ad80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:30:24.202 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:20.709+0000 7f2adf94ad80 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-11-16T20:30:24.203 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 fad13ab8-e2c9-4978-a5bc-122dbb18f3cc 2020-11-16T20:30:24.203 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:30:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-11-16T20:30:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:30:25.335 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:30:25.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:30:25.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:30:25.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:30:25.834 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:30:25.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:30:25.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:25.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:30:25.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:30:25.861 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:25.857+0000 7f55243a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:25.864 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:25.861+0000 7f55243a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:25.865 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:25.861+0000 7f55243a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:26.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:30:26.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:30:26.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:30:26.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:30:26.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:30:26.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:26.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:30:26.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:26.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:30:26.217 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:30:26.405 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:26.401+0000 7f55243a3d80 -1 Falling back to public interface 2020-11-16T20:30:26.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:30:27.386 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:27.373+0000 7f55243a3d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:30:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:30:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:30:27.597 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:30:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:30:28.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:30:29.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:30:29.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:29.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:30:29.100 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:30:29.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:29.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:30:29.492 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/105995,v1:127.0.0.1:6819/105995] [v2:127.0.0.1:6820/105995,v1:127.0.0.1:6821/105995] exists,up fad13ab8-e2c9-4978-a5bc-122dbb18f3cc 2020-11-16T20:30:29.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:30:29.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:30:29.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:30:29.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:30:29.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-11-16T20:30:29.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:29.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:30:29.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T20:30:29.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:30:29.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-11-16T20:30:29.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:30:29.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:30:29.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:30:29.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:30:29.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-11-16T20:30:29.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-11-16T20:30:29.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:30:29.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-11-16T20:30:29.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:30:29.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:30:29.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:30:29.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:30:29.499 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:30:29.499 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:29.499 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:29.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:29.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:29.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:30:29.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:30:29.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:30:29.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:30:29.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:30:29.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:30:29.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:30:29.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:29.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:30:29.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:30:29.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-11-16T20:30:29.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:30:29.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0cdf473a-45a8-40d9-99fd-dc3b9724d50d 2020-11-16T20:30:29.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 0cdf473a-45a8-40d9-99fd-dc3b9724d50d' 2020-11-16T20:30:29.502 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 0cdf473a-45a8-40d9-99fd-dc3b9724d50d 2020-11-16T20:30:29.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:30:29.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBl4bJfQ3XNHhAAjzBjP5bgkrO41P+f80FJFg== 2020-11-16T20:30:29.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBl4bJfQ3XNHhAAjzBjP5bgkrO41P+f80FJFg=="}' 2020-11-16T20:30:29.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0cdf473a-45a8-40d9-99fd-dc3b9724d50d -i td/osd-backfill-recovery-log/3/new.json 2020-11-16T20:30:29.971 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T20:30:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-11-16T20:30:29.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBl4bJfQ3XNHhAAjzBjP5bgkrO41P+f80FJFg== --osd-uuid 0cdf473a-45a8-40d9-99fd-dc3b9724d50d 2020-11-16T20:30:30.015 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:30.009+0000 7f8470256d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:30.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:30.013+0000 7f8470256d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:30.019 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:30.013+0000 7f8470256d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:30.020 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:30.017+0000 7f8470256d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:30:30.020 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:30.017+0000 7f8470256d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:30:30.020 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:30.017+0000 7f8470256d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:30:30.124 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:30.121+0000 7f8470256d80 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-11-16T20:30:34.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-11-16T20:30:34.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:30:34.810 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T20:30:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T20:30:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:30:35.304 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T20:30:35.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T20:30:35.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:30:35.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:30:35.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:35.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:30:35.333 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:35.329+0000 7f5187b1ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:35.335 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:35.329+0000 7f5187b1ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:35.336 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:35.333+0000 7f5187b1ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:35.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T20:30:35.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:30:35.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T20:30:35.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:30:35.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:30:35.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:35.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:30:35.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:35.689 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:30:35.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:30:35.898 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:35.893+0000 7f5187b1ad80 -1 Falling back to public interface 2020-11-16T20:30:36.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:30:36.753 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:36.749+0000 7f5187b1ad80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T20:30:37.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:30:37.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:37.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:30:37.092 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:30:37.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:37.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:30:37.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:30:38.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:30:38.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:38.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:30:38.519 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:30:38.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:38.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:30:38.911 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/107015,v1:127.0.0.1:6827/107015] [v2:127.0.0.1:6828/107015,v1:127.0.0.1:6829/107015] exists,up 0cdf473a-45a8-40d9-99fd-dc3b9724d50d 2020-11-16T20:30:38.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:30:38.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:30:38.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:30:38.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:30:38.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-11-16T20:30:38.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:38.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:30:38.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T20:30:38.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:30:38.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-11-16T20:30:38.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:30:38.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:30:38.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:30:38.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:30:38.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-11-16T20:30:38.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-11-16T20:30:38.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:30:38.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-11-16T20:30:38.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:30:38.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:30:38.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:30:38.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:30:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:30:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:38.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:30:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:30:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:30:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:30:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:30:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:30:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:30:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:30:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:30:38.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-11-16T20:30:38.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:30:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=12038b61-a117-43ce-a029-7c3c18347c88 2020-11-16T20:30:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 12038b61-a117-43ce-a029-7c3c18347c88' 2020-11-16T20:30:38.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:30:38.919 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 12038b61-a117-43ce-a029-7c3c18347c88 2020-11-16T20:30:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBu4bJfI3eGNxAAUSy+1KC4LmJ2TuWvzBAUFA== 2020-11-16T20:30:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBu4bJfI3eGNxAAUSy+1KC4LmJ2TuWvzBAUFA=="}' 2020-11-16T20:30:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 12038b61-a117-43ce-a029-7c3c18347c88 -i td/osd-backfill-recovery-log/4/new.json 2020-11-16T20:30:39.441 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T20:30:39.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-11-16T20:30:39.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBu4bJfI3eGNxAAUSy+1KC4LmJ2TuWvzBAUFA== --osd-uuid 12038b61-a117-43ce-a029-7c3c18347c88 2020-11-16T20:30:39.484 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:39.481+0000 7ff897b07d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:39.487 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:39.481+0000 7ff897b07d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:39.488 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:39.485+0000 7ff897b07d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:39.489 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:39.485+0000 7ff897b07d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:30:39.489 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:39.485+0000 7ff897b07d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:30:39.489 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:39.485+0000 7ff897b07d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:30:39.527 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:39.525+0000 7ff897b07d80 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-11-16T20:30:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-11-16T20:30:44.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:30:44.135 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T20:30:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T20:30:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:30:44.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T20:30:44.633 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T20:30:44.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:44.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:30:44.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:30:44.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:30:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:44.661+0000 7f2ff859fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:44.666 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:44.661+0000 7f2ff859fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:44.667 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:44.665+0000 7f2ff859fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:45.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T20:30:45.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:30:45.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T20:30:45.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:30:45.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:30:45.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:45.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:30:45.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:45.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:30:45.028 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:30:45.222 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:45.217+0000 7f2ff859fd80 -1 Falling back to public interface 2020-11-16T20:30:45.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:30:46.075 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:46.065+0000 7f2ff859fd80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T20:30:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:30:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:46.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:30:46.441 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:30:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:46.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:30:46.857 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/108035,v1:127.0.0.1:6835/108035] [v2:127.0.0.1:6836/108035,v1:127.0.0.1:6837/108035] exists,up 12038b61-a117-43ce-a029-7c3c18347c88 2020-11-16T20:30:46.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:30:46.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:30:46.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:30:46.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:30:46.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-11-16T20:30:46.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:30:46.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:30:46.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T20:30:46.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:30:46.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-11-16T20:30:46.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:30:46.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:30:46.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:30:46.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:30:46.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-11-16T20:30:46.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-11-16T20:30:46.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:30:46.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-11-16T20:30:46.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:30:46.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:30:46.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:30:46.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:30:46.862 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:30:46.862 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:30:46.862 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:30:46.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:46.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:30:46.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:30:46.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:30:46.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:30:46.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:30:46.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:30:46.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:30:46.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:30:46.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:46.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:30:46.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:30:46.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-11-16T20:30:46.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:30:46.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=348ca718-d219-47eb-a571-c428b52e086c 2020-11-16T20:30:46.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 348ca718-d219-47eb-a571-c428b52e086c' 2020-11-16T20:30:46.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:30:46.866 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 348ca718-d219-47eb-a571-c428b52e086c 2020-11-16T20:30:46.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB24bJfl2tHNBAAggblKpBnUeorROsDALNwJA== 2020-11-16T20:30:46.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB24bJfl2tHNBAAggblKpBnUeorROsDALNwJA=="}' 2020-11-16T20:30:46.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 348ca718-d219-47eb-a571-c428b52e086c -i td/osd-backfill-recovery-log/5/new.json 2020-11-16T20:30:47.501 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T20:30:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-11-16T20:30:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB24bJfl2tHNBAAggblKpBnUeorROsDALNwJA== --osd-uuid 348ca718-d219-47eb-a571-c428b52e086c 2020-11-16T20:30:47.546 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:47.541+0000 7fd7b6ecfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:47.549 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:47.545+0000 7fd7b6ecfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:47.551 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:47.549+0000 7fd7b6ecfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:47.552 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:47.549+0000 7fd7b6ecfd80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:30:47.552 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:47.549+0000 7fd7b6ecfd80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:30:47.552 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:47.549+0000 7fd7b6ecfd80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:30:47.593 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:47.589+0000 7fd7b6ecfd80 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-11-16T20:30:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-11-16T20:30:52.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:30:52.251 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T20:30:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T20:30:52.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:30:52.755 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T20:30:52.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T20:30:52.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:30:52.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:30:52.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:30:52.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:30:52.785 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:52.781+0000 7fba18736d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:52.787 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:52.785+0000 7fba18736d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:52.789 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:52.785+0000 7fba18736d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:30:53.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T20:30:53.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:30:53.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T20:30:53.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:30:53.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:30:53.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:53.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:30:53.116 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:30:53.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:53.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T20:30:53.337 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:53.333+0000 7fba18736d80 -1 Falling back to public interface 2020-11-16T20:30:53.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:30:54.218 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:30:54.209+0000 7fba18736d80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T20:30:54.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:30:54.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:30:54.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:30:54.493 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:30:54.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:30:54.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T20:30:54.955 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/109029,v1:127.0.0.1:6843/109029] [v2:127.0.0.1:6844/109029,v1:127.0.0.1:6845/109029] exists,up 348ca718-d219-47eb-a571-c428b52e086c 2020-11-16T20:30:54.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:30:54.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:30:54.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:30:54.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-11-16T20:30:54.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T20:30:55.461 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T20:30:55.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:30:56.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-11-16T20:30:56.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:56.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-11-16T20:30:56.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:56.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-11-16T20:30:57.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:57.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-11-16T20:30:57.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:57.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-11-16T20:30:57.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:57.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-11-16T20:30:57.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:57.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-11-16T20:30:57.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:57.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-11-16T20:30:58.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:58.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-11-16T20:30:58.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:58.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-11-16T20:30:58.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:58.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-11-16T20:30:58.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:58.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-11-16T20:30:58.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:58.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-11-16T20:30:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:58.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-11-16T20:30:59.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:59.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-11-16T20:30:59.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:59.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-11-16T20:30:59.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:59.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-11-16T20:30:59.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:59.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-11-16T20:30:59.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:59.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-11-16T20:30:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:30:59.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-11-16T20:31:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-11-16T20:31:00.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:00.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-11-16T20:31:00.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:00.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-11-16T20:31:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:00.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-11-16T20:31:00.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:00.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-11-16T20:31:00.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:00.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-11-16T20:31:01.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:01.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-11-16T20:31:01.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:01.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-11-16T20:31:01.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:01.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-11-16T20:31:01.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:01.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-11-16T20:31:01.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:01.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-11-16T20:31:01.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:01.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-11-16T20:31:02.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:02.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-11-16T20:31:02.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:02.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-11-16T20:31:02.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:02.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-11-16T20:31:02.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:02.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-11-16T20:31:02.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:02.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-11-16T20:31:02.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:02.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-11-16T20:31:03.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:03.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-11-16T20:31:03.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:03.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-11-16T20:31:03.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:03.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-11-16T20:31:03.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:03.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-11-16T20:31:03.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:03.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-11-16T20:31:03.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:03.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-11-16T20:31:04.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:04.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-11-16T20:31:04.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:04.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-11-16T20:31:04.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:04.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-11-16T20:31:04.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:04.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-11-16T20:31:04.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:04.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-11-16T20:31:04.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:04.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-11-16T20:31:05.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:05.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-11-16T20:31:05.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:05.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-11-16T20:31:05.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:05.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-11-16T20:31:05.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:05.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-11-16T20:31:05.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:05.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-11-16T20:31:05.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:05.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-11-16T20:31:05.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:05.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-11-16T20:31:06.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:06.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-11-16T20:31:06.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:06.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-11-16T20:31:06.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:06.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-11-16T20:31:06.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:06.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-11-16T20:31:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-11-16T20:31:06.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:06.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-11-16T20:31:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-11-16T20:31:07.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:07.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-11-16T20:31:07.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:07.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-11-16T20:31:07.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:07.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-11-16T20:31:07.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:07.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-11-16T20:31:08.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:08.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-11-16T20:31:08.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:08.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-11-16T20:31:08.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:08.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-11-16T20:31:08.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:08.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-11-16T20:31:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-11-16T20:31:08.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:08.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-11-16T20:31:08.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:08.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-11-16T20:31:09.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:09.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-11-16T20:31:09.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:09.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-11-16T20:31:09.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:09.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-11-16T20:31:09.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:09.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-11-16T20:31:09.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:09.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-11-16T20:31:09.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:09.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-11-16T20:31:10.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:10.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-11-16T20:31:10.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:10.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-11-16T20:31:10.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:10.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-11-16T20:31:10.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:10.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-11-16T20:31:10.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:10.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-11-16T20:31:10.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:10.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-11-16T20:31:11.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:11.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-11-16T20:31:11.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:11.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-11-16T20:31:11.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:11.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-11-16T20:31:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-11-16T20:31:11.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:11.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-11-16T20:31:11.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:11.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-11-16T20:31:12.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:12.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-11-16T20:31:12.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:12.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-11-16T20:31:12.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:12.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-11-16T20:31:12.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:12.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-11-16T20:31:12.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:12.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-11-16T20:31:12.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:12.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-11-16T20:31:13.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:13.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-11-16T20:31:13.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:13.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-11-16T20:31:13.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:13.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-11-16T20:31:13.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-11-16T20:31:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-11-16T20:31:13.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:13.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-11-16T20:31:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-11-16T20:31:14.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:14.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-11-16T20:31:14.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:14.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-11-16T20:31:14.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:14.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-11-16T20:31:14.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:14.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-11-16T20:31:14.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:14.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-11-16T20:31:15.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:15.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-11-16T20:31:15.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:15.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-11-16T20:31:15.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:15.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-11-16T20:31:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-11-16T20:31:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-11-16T20:31:15.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:15.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-11-16T20:31:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-11-16T20:31:16.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-11-16T20:31:16.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:16.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-11-16T20:31:16.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:16.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-11-16T20:31:16.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:16.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-11-16T20:31:20.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-11-16T20:31:20.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-11-16T20:31:20.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-11-16T20:31:20.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-11-16T20:31:20.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-11-16T20:31:20.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-11-16T20:31:20.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-11-16T20:31:20.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-11-16T20:31:20.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-11-16T20:31:20.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-11-16T20:31:20.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-11-16T20:31:20.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-11-16T20:31:20.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-11-16T20:31:20.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-11-16T20:31:20.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-11-16T20:31:20.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-11-16T20:31:20.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-11-16T20:31:20.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-11-16T20:31:20.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-11-16T20:31:20.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-11-16T20:31:20.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-11-16T20:31:20.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-11-16T20:31:20.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-11-16T20:31:20.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-11-16T20:31:20.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-11-16T20:31:20.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:20.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-11-16T20:31:21.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:21.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-11-16T20:31:21.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:21.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-11-16T20:31:21.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:31:21.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-11-16T20:31:21.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-11-16T20:31:21.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-11-16T20:31:21.863 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:31:21.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-11-16T20:31:22.496 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-11-16T20:31:22.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-11-16T20:31:22.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-11-16T20:31:23.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-11-16T20:31:23.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:31:23.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:31:23.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:31:23.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:31:23.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:31:23.509 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:31:23.509 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:31:23.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:31:23.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:31:23.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:31:23.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:31:23.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:31:23.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:31:23.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:31:23.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:31:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:31:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:31:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:31:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:31:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T20:31:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T20:31:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:31:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:31:24.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:31:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836497 2020-11-16T20:31:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836497 2020-11-16T20:31:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497' 2020-11-16T20:31:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:31:24.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:31:24.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672976 2020-11-16T20:31:24.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672976 2020-11-16T20:31:24.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672976' 2020-11-16T20:31:24.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:31:24.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:31:24.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509454 2020-11-16T20:31:24.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509454 2020-11-16T20:31:24.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509454' 2020-11-16T20:31:24.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:31:24.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:31:24.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345932 2020-11-16T20:31:24.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345932 2020-11-16T20:31:24.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509454 3-85899345932' 2020-11-16T20:31:24.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:31:24.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:31:24.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182410 2020-11-16T20:31:24.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182410 2020-11-16T20:31:24.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509454 3-85899345932 4-107374182410' 2020-11-16T20:31:24.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:31:24.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T20:31:25.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018888 2020-11-16T20:31:25.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018888 2020-11-16T20:31:25.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509454 3-85899345932 4-107374182410 5-128849018888' 2020-11-16T20:31:25.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:31:25.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836497 2020-11-16T20:31:25.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:31:25.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:31:25.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:31:25.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836497 2020-11-16T20:31:25.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836497 2020-11-16T20:31:25.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2020-11-16T20:31:25.096 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836497 2020-11-16T20:31:25.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:31:25.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836497 -lt 21474836497 2020-11-16T20:31:25.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:31:25.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672976 2020-11-16T20:31:25.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:31:25.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:31:25.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672976 2020-11-16T20:31:25.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:31:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672976 2020-11-16T20:31:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2020-11-16T20:31:25.511 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672976 2020-11-16T20:31:25.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:31:25.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672976 -lt 42949672976 2020-11-16T20:31:25.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:31:25.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509454 2020-11-16T20:31:25.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:31:25.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:31:25.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509454 2020-11-16T20:31:25.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:31:25.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509454 2020-11-16T20:31:25.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2020-11-16T20:31:25.960 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509454 2020-11-16T20:31:25.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:31:26.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509454 -lt 64424509454 2020-11-16T20:31:26.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:31:26.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345932 2020-11-16T20:31:26.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:31:26.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:31:26.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345932 2020-11-16T20:31:26.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:31:26.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345932 2020-11-16T20:31:26.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2020-11-16T20:31:26.347 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 85899345932 2020-11-16T20:31:26.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:31:26.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345932 -lt 85899345932 2020-11-16T20:31:26.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:31:26.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182410 2020-11-16T20:31:26.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:31:26.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:31:26.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182410 2020-11-16T20:31:26.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:31:26.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182410 2020-11-16T20:31:26.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2020-11-16T20:31:26.715 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 107374182410 2020-11-16T20:31:26.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:31:27.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182410 -lt 107374182410 2020-11-16T20:31:27.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:31:27.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018888 2020-11-16T20:31:27.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:31:27.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T20:31:27.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018888 2020-11-16T20:31:27.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:31:27.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018888 2020-11-16T20:31:27.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2020-11-16T20:31:27.150 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 128849018888 2020-11-16T20:31:27.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T20:31:27.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018889 -lt 128849018888 2020-11-16T20:31:27.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:31:27.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:27.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:28.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:31:28.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:28.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:28.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:28.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:28.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:28.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:28.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:28.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:28.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:28.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:28.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:28.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:28.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:31:28.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:31:28.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:31:28.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:28.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:28.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:28.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:28.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:28.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:28.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:28.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:28.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:28.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:29.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:29.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:29.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:29.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:29.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:29.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:29.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:30.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=10536 2020-11-16T20:31:30.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 10536 '!=' null 2020-11-16T20:31:30.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:30.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:30.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:30.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:30.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:30.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:30.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:30.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:30.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:30.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:30.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:30.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:30.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:30.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:30.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:31.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:31.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:31.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:31.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:31.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:31.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:31.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:31.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:31.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:31.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7912 2020-11-16T20:31:31.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7912 '!=' null 2020-11-16T20:31:31.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:31.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:31.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:31.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:31.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:31.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:31.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:31.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:31.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:31.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:31.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:32.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:32.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:32.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:32.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:32.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:32.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:32.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:32.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:32.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:32.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:32.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:32.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:32.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21195 2020-11-16T20:31:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21195 '!=' null 2020-11-16T20:31:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:33.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:33.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:33.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:33.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:33.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:33.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:33.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:33.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:33.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:33.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:33.782 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:33.782 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:34.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:34.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:34.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:34.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:34.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:34.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:34.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:34.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:34.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:38.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21195 2020-11-16T20:31:38.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21195 '!=' null 2020-11-16T20:31:38.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:38.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:38.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:38.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:38.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:38.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:38.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:38.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:38.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:38.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:38.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:38.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:38.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:38.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:38.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:38.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:38.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18106 2020-11-16T20:31:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18106 '!=' null 2020-11-16T20:31:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:38.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:38.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:38.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:38.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:38.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:38.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:38.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:38.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:38.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:38.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:38.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=29673 2020-11-16T20:31:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 29673 '!=' null 2020-11-16T20:31:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:38.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:38.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:38.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:38.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:38.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:38.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:38.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:38.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:38.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:38.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:38.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:38.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:38.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:39.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24398 2020-11-16T20:31:39.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24398 '!=' null 2020-11-16T20:31:39.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:39.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:39.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:39.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:39.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:39.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:39.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:39.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:39.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:39.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:39.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:39.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:39.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:39.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:39.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:40.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:40.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:40.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:40.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:40.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:40.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:40.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:40.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:40.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:40.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24398 2020-11-16T20:31:40.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24398 '!=' null 2020-11-16T20:31:40.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:40.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:40.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:40.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:40.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:40.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:40.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:40.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:40.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:40.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:40.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:41.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:41.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:41.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:41.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:41.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:41.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:41.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:41.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:41.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:41.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:41.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:41.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:41.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:41.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24398 2020-11-16T20:31:41.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24398 '!=' null 2020-11-16T20:31:41.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:41.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:41.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:42.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:42.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:42.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:42.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:42.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:42.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:42.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:42.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:42.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:42.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:42.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:42.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:42.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:42.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:42.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:42.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:42.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:42.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:42.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:42.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:42.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:43.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=36267 2020-11-16T20:31:43.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 36267 '!=' null 2020-11-16T20:31:43.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:43.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:43.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:43.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:43.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:43.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:43.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:43.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:43.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:43.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:43.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:44.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:44.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:44.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:44.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:44.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:44.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:44.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:44.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:44.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:44.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:44.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:44.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:44.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:45.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24397 2020-11-16T20:31:45.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24397 '!=' null 2020-11-16T20:31:45.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:45.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:45.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:45.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:45.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:45.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:45.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:45.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:45.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:45.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:45.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:45.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:45.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:45.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:45.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:45.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:45.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:45.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:45.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:45.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:45.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:45.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:45.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:46.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24397 2020-11-16T20:31:46.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24397 '!=' null 2020-11-16T20:31:46.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:46.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:46.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:46.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:46.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:46.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:46.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:46.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:46.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:46.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:46.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:46.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:46.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:46.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:46.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:47.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:47.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:47.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:47.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:47.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:47.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:47.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:47.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:47.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:47.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=36925 2020-11-16T20:31:47.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 36925 '!=' null 2020-11-16T20:31:47.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:47.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:47.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:47.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:47.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:47.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:47.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:47.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:47.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:47.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:47.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:48.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:48.361 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:48.361 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:48.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:48.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:48.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:48.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:48.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:48.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:48.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:48.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:48.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:48.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:49.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24397 2020-11-16T20:31:49.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24397 '!=' null 2020-11-16T20:31:49.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:49.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:49.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:49.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:49.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:49.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:49.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:49.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:49.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:49.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:49.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:49.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:49.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:49.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:50.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:50.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:50.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:50.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24397 2020-11-16T20:31:50.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24397 '!=' null 2020-11-16T20:31:50.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:50.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:50.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:50.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:50.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:50.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:50.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:50.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:50.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:50.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:50.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:51.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:51.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:51.239 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:51.239 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:51.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:51.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:51.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:51.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:51.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:51.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:51.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:51.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:51.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:52.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24397 2020-11-16T20:31:52.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24397 '!=' null 2020-11-16T20:31:52.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:52.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:52.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:52.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:52.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:52.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:52.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:52.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:52.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:52.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:52.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:52.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:52.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:52.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:52.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:53.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:53.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:53.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:53.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:53.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:53.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:53.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:53.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:53.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:53.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=37584 2020-11-16T20:31:53.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 37584 '!=' null 2020-11-16T20:31:53.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:53.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:53.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:53.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:53.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:53.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:53.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:53.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:53.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:53.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:53.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:54.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:54.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:54.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:54.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:54.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:54.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:54.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:54.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:54.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:54.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:54.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:54.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:54.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25716 2020-11-16T20:31:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25716 '!=' null 2020-11-16T20:31:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:55.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:55.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:55.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:55.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:55.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:55.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:55.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:55.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:55.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:55.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:55.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:56.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:56.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:56.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:56.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:56.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:56.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:56.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:56.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:56.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:56.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25716 2020-11-16T20:31:56.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25716 '!=' null 2020-11-16T20:31:56.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:56.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:56.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:56.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:56.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:56.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:56.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:56.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:56.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:56.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:56.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:57.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:57.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:57.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:57.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:57.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:57.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:57.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:57.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:57.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:57.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:57.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:57.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:57.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:57.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=38904 2020-11-16T20:31:57.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 38904 '!=' null 2020-11-16T20:31:57.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:57.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:57.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:58.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:58.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:58.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:58.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:58.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:58.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:58.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:58.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:58.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:58.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:58.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:31:58.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:31:58.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:31:58.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:31:58.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:31:58.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:31:58.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:31:58.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:31:58.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:31:58.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:31:59.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26376 2020-11-16T20:31:59.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26376 '!=' null 2020-11-16T20:31:59.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:31:59.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:31:59.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:31:59.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:31:59.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:31:59.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:31:59.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:31:59.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:31:59.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:31:59.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:31:59.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:31:59.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:31:59.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:31:59.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:31:59.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:32:00.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:32:00.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:32:00.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:32:00.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:32:00.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:32:00.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:32:00.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:32:00.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:32:00.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:32:00.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26376 2020-11-16T20:32:00.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26376 '!=' null 2020-11-16T20:32:00.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:32:00.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:32:00.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:32:00.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:32:00.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:32:00.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:32:00.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:32:00.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:32:00.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:32:00.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:32:00.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:32:01.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:32:01.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:32:01.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:32:01.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:32:01.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:32:01.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:32:01.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:32:01.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:32:01.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:32:01.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:32:01.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:32:01.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:32:01.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:32:02.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26375 2020-11-16T20:32:02.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26375 '!=' null 2020-11-16T20:32:02.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:32:02.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:32:02.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:32:02.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:32:02.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:32:02.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:32:02.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:32:02.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:32:02.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:32:02.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:32:02.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:32:02.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:32:02.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:32:02.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:32:02.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:32:03.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:32:03.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:32:03.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:32:03.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:32:03.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:32:03.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:32:03.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:32:03.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:32:03.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:32:03.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=38245 2020-11-16T20:32:03.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 38245 '!=' null 2020-11-16T20:32:03.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:32:03.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:32:03.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:32:03.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:32:03.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:32:03.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:32:03.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:32:03.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:32:03.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:32:03.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:32:03.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:32:04.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:32:04.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:32:04.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:32:04.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:32:04.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:32:04.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:32:04.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:32:04.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:32:04.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:32:04.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:32:04.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:32:04.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:32:04.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:32:05.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17804 2020-11-16T20:32:05.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17804 '!=' null 2020-11-16T20:32:05.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:32:05.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:32:05.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:32:05.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:32:05.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:32:05.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:32:05.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:32:05.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:32:05.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:32:05.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:32:05.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:32:05.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:32:05.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:32:05.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:32:05.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:32:06.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:32:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:32:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:32:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:32:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:32:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:32:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:32:06.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:32:06.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:32:06.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17804 2020-11-16T20:32:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17804 '!=' null 2020-11-16T20:32:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:32:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:32:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:32:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:32:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:32:06.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:32:06.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:32:06.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:32:06.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:32:06.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:32:06.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:32:07.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:32:07.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:32:07.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:32:07.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:32:07.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:32:07.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:32:07.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:32:07.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:32:07.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:32:07.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:32:07.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:32:07.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:32:07.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:32:08.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17804 2020-11-16T20:32:08.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17804 '!=' null 2020-11-16T20:32:12.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:32:12.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:32:12.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:32:12.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:32:12.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:32:12.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:32:12.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:32:12.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:32:12.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:32:12.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:32:12.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:32:12.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:32:12.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:32:12.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:32:12.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:32:12.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:32:12.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:32:12.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:32:12.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:32:12.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:32:12.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:32:12.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:32:12.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:32:12.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:32:12.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27695 2020-11-16T20:32:12.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27695 '!=' null 2020-11-16T20:32:12.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:32:12.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:32:12.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:32:12.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:32:12.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:32:12.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:32:12.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:32:12.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:32:12.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:32:12.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:32:12.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:32:12.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:32:12.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:32:12.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:32:12.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:32:12.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:32:12.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:32:12.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:32:12.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2020-11-16T20:32:12.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:32:12.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:32:12.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:32:12.262 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:32:12.262 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:32:12.262 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:32:12.262 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T20:32:12.262 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T20:32:12.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:32:12.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:32:12.263 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:32:12.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836508 2020-11-16T20:32:12.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836508 2020-11-16T20:32:12.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508' 2020-11-16T20:32:12.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:32:12.263 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:32:12.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672987 2020-11-16T20:32:12.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672987 2020-11-16T20:32:12.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672987' 2020-11-16T20:32:12.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:32:12.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:32:12.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509465 2020-11-16T20:32:12.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509465 2020-11-16T20:32:12.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509465' 2020-11-16T20:32:12.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:32:12.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:32:12.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345943 2020-11-16T20:32:12.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345943 2020-11-16T20:32:12.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509465 3-85899345943' 2020-11-16T20:32:12.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:32:12.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:32:12.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182421 2020-11-16T20:32:12.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182421 2020-11-16T20:32:12.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509465 3-85899345943 4-107374182421' 2020-11-16T20:32:12.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:32:12.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T20:32:12.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018900 2020-11-16T20:32:12.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018900 2020-11-16T20:32:12.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509465 3-85899345943 4-107374182421 5-128849018900' 2020-11-16T20:32:12.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:32:12.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836508 2020-11-16T20:32:12.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:32:12.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:32:12.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836508 2020-11-16T20:32:12.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:32:12.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836508 2020-11-16T20:32:12.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2020-11-16T20:32:12.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:32:12.268 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836508 2020-11-16T20:32:12.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836506 -lt 21474836508 2020-11-16T20:32:12.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:32:13.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:32:13.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:32:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836509 -lt 21474836508 2020-11-16T20:32:13.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:32:13.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672987 2020-11-16T20:32:13.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:32:13.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:32:13.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672987 2020-11-16T20:32:13.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:32:13.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672987 2020-11-16T20:32:13.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2020-11-16T20:32:13.696 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672987 2020-11-16T20:32:13.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:32:14.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672987 -lt 42949672987 2020-11-16T20:32:14.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:32:14.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509465 2020-11-16T20:32:14.098 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:32:14.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:32:14.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509465 2020-11-16T20:32:14.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:32:14.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509465 2020-11-16T20:32:14.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2020-11-16T20:32:14.102 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509465 2020-11-16T20:32:14.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:32:14.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509465 -lt 64424509465 2020-11-16T20:32:14.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:32:14.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345943 2020-11-16T20:32:14.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:32:14.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:32:14.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345943 2020-11-16T20:32:14.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:32:14.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345943 2020-11-16T20:32:14.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345943' 2020-11-16T20:32:14.494 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 85899345943 2020-11-16T20:32:14.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:32:14.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345943 -lt 85899345943 2020-11-16T20:32:14.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:32:14.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:32:14.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182421 2020-11-16T20:32:14.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:32:14.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:32:14.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182421 2020-11-16T20:32:14.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182421 2020-11-16T20:32:14.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182421' 2020-11-16T20:32:14.896 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 107374182421 2020-11-16T20:32:14.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:32:15.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182422 -lt 107374182421 2020-11-16T20:32:15.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:32:15.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018900 2020-11-16T20:32:15.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:32:15.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T20:32:15.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018900 2020-11-16T20:32:15.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:32:15.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018900 2020-11-16T20:32:15.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018900' 2020-11-16T20:32:15.298 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 128849018900 2020-11-16T20:32:15.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T20:32:15.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018900 -lt 128849018900 2020-11-16T20:32:15.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2020-11-16T20:32:15.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2020-11-16T20:32:16.039 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:32:16.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2020-11-16T20:32:16.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2020-11-16T20:32:16.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:32:16.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:32:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:32:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:32:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:32:16.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:32:16.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2020-11-16T20:32:16.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-11-16T20:32:16.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-11-16T20:32:16.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:16.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T20:32:16.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=4 2020-11-16T20:32:16.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T20:32:16.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-11-16T20:32:16.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-11-16T20:32:17.425 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:32:17.425 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_log_t": { 2020-11-16T20:32:17.425 INFO:tasks.workunit.client.0.smithi189.stdout: "head": "33'150", 2020-11-16T20:32:17.426 INFO:tasks.workunit.client.0.smithi189.stdout: "tail": "33'148", 2020-11-16T20:32:17.426 INFO:tasks.workunit.client.0.smithi189.stdout: "log": [ 2020-11-16T20:32:17.426 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.426 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:32:17.426 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-11-16T20:32:17.426 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'149", 2020-11-16T20:32:17.426 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:32:17.427 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4637.0:1", 2020-11-16T20:32:17.427 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:32:17.427 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:31:21.252010+0000", 2020-11-16T20:32:17.427 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:32:17.427 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:32:17.427 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:32:17.427 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:32:17.428 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:32:17.428 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:32:17.428 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:32:17.428 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.428 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:32:17.428 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:32:17.428 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:32:17.429 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:32:17.429 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:32:17.429 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:32:17.429 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:32:17.429 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.429 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.429 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:32:17.429 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-11-16T20:32:17.430 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'150", 2020-11-16T20:32:17.430 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:32:17.430 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4640.0:1", 2020-11-16T20:32:17.430 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:32:17.430 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:31:21.382149+0000", 2020-11-16T20:32:17.430 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:32:17.430 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:32:17.431 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:32:17.431 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:32:17.431 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:32:17.431 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:32:17.431 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:32:17.431 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.431 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:32:17.431 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:32:17.432 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:32:17.432 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:32:17.432 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:32:17.432 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:32:17.432 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:32:17.432 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:32:17.432 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:32:17.433 INFO:tasks.workunit.client.0.smithi189.stdout: "dups": [ 2020-11-16T20:32:17.433 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.433 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4193.0:1", 2020-11-16T20:32:17.433 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'1", 2020-11-16T20:32:17.433 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "1", 2020-11-16T20:32:17.433 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.433 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.433 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.434 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4196.0:1", 2020-11-16T20:32:17.434 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'2", 2020-11-16T20:32:17.434 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "2", 2020-11-16T20:32:17.434 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.434 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.434 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.434 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4199.0:1", 2020-11-16T20:32:17.435 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'3", 2020-11-16T20:32:17.435 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "3", 2020-11-16T20:32:17.435 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.435 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.435 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.435 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4202.0:1", 2020-11-16T20:32:17.435 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'4", 2020-11-16T20:32:17.435 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "4", 2020-11-16T20:32:17.436 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.436 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.436 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.436 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4205.0:1", 2020-11-16T20:32:17.436 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'5", 2020-11-16T20:32:17.436 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "5", 2020-11-16T20:32:17.436 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.437 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.437 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.437 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4208.0:1", 2020-11-16T20:32:17.437 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'6", 2020-11-16T20:32:17.437 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "6", 2020-11-16T20:32:17.437 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.437 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.437 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.438 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4211.0:1", 2020-11-16T20:32:17.438 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'7", 2020-11-16T20:32:17.438 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "7", 2020-11-16T20:32:17.438 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.438 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.438 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.438 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4214.0:1", 2020-11-16T20:32:17.439 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'8", 2020-11-16T20:32:17.439 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "8", 2020-11-16T20:32:17.439 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.439 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.439 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.439 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4217.0:1", 2020-11-16T20:32:17.439 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'9", 2020-11-16T20:32:17.439 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "9", 2020-11-16T20:32:17.440 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.440 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.440 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.440 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4220.0:1", 2020-11-16T20:32:17.440 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'10", 2020-11-16T20:32:17.440 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "10", 2020-11-16T20:32:17.440 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.441 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.441 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.441 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4223.0:1", 2020-11-16T20:32:17.441 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'11", 2020-11-16T20:32:17.441 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "11", 2020-11-16T20:32:17.441 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.441 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.441 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.442 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4226.0:1", 2020-11-16T20:32:17.442 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'12", 2020-11-16T20:32:17.442 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "12", 2020-11-16T20:32:17.442 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.442 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.442 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.442 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4229.0:1", 2020-11-16T20:32:17.443 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'13", 2020-11-16T20:32:17.443 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "13", 2020-11-16T20:32:17.443 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.443 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.443 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.443 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4232.0:1", 2020-11-16T20:32:17.443 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'14", 2020-11-16T20:32:17.443 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "14", 2020-11-16T20:32:17.444 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.444 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.444 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.444 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4235.0:1", 2020-11-16T20:32:17.444 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'15", 2020-11-16T20:32:17.444 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "15", 2020-11-16T20:32:17.444 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.444 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.445 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.445 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4238.0:1", 2020-11-16T20:32:17.445 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'16", 2020-11-16T20:32:17.445 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "16", 2020-11-16T20:32:17.445 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.445 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.445 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.446 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4241.0:1", 2020-11-16T20:32:17.446 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'17", 2020-11-16T20:32:17.446 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "17", 2020-11-16T20:32:17.446 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.446 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.446 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.446 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4244.0:1", 2020-11-16T20:32:17.446 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'18", 2020-11-16T20:32:17.447 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "18", 2020-11-16T20:32:17.447 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.447 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.447 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.447 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4247.0:1", 2020-11-16T20:32:17.447 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'19", 2020-11-16T20:32:17.447 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "19", 2020-11-16T20:32:17.448 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.448 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.448 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.448 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4250.0:1", 2020-11-16T20:32:17.448 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'20", 2020-11-16T20:32:17.448 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "20", 2020-11-16T20:32:17.448 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.448 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.449 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.449 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4253.0:1", 2020-11-16T20:32:17.449 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'21", 2020-11-16T20:32:17.449 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "21", 2020-11-16T20:32:17.449 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.449 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.449 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.449 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4256.0:1", 2020-11-16T20:32:17.450 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'22", 2020-11-16T20:32:17.450 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "22", 2020-11-16T20:32:17.450 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.450 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.450 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.450 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4259.0:1", 2020-11-16T20:32:17.450 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'23", 2020-11-16T20:32:17.451 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "23", 2020-11-16T20:32:17.451 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.451 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.451 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.451 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4262.0:1", 2020-11-16T20:32:17.451 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'24", 2020-11-16T20:32:17.451 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "24", 2020-11-16T20:32:17.451 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.452 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.452 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.452 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4265.0:1", 2020-11-16T20:32:17.452 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'25", 2020-11-16T20:32:17.452 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "25", 2020-11-16T20:32:17.452 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.452 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.453 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.453 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4268.0:1", 2020-11-16T20:32:17.453 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'26", 2020-11-16T20:32:17.453 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "26", 2020-11-16T20:32:17.453 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.453 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.453 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.453 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4271.0:1", 2020-11-16T20:32:17.454 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'27", 2020-11-16T20:32:17.454 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "27", 2020-11-16T20:32:17.454 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.454 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.454 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.454 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4274.0:1", 2020-11-16T20:32:17.454 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'28", 2020-11-16T20:32:17.454 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "28", 2020-11-16T20:32:17.455 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.455 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.455 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.455 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4277.0:1", 2020-11-16T20:32:17.455 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'29", 2020-11-16T20:32:17.455 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "29", 2020-11-16T20:32:17.455 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.456 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.456 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.456 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4280.0:1", 2020-11-16T20:32:17.456 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'30", 2020-11-16T20:32:17.456 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "30", 2020-11-16T20:32:17.456 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.456 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.456 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.457 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4283.0:1", 2020-11-16T20:32:17.457 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'31", 2020-11-16T20:32:17.457 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "31", 2020-11-16T20:32:17.457 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.457 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.457 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.457 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4286.0:1", 2020-11-16T20:32:17.457 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'32", 2020-11-16T20:32:17.458 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "32", 2020-11-16T20:32:17.458 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.458 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.458 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.458 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4289.0:1", 2020-11-16T20:32:17.458 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'33", 2020-11-16T20:32:17.458 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "33", 2020-11-16T20:32:17.459 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.459 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.459 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.459 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4292.0:1", 2020-11-16T20:32:17.459 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'34", 2020-11-16T20:32:17.459 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "34", 2020-11-16T20:32:17.459 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.459 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.460 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.460 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4295.0:1", 2020-11-16T20:32:17.460 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'35", 2020-11-16T20:32:17.460 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "35", 2020-11-16T20:32:17.460 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.460 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.460 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.461 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4298.0:1", 2020-11-16T20:32:17.461 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'36", 2020-11-16T20:32:17.461 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "36", 2020-11-16T20:32:17.461 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.461 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.461 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.461 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4301.0:1", 2020-11-16T20:32:17.461 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'37", 2020-11-16T20:32:17.462 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "37", 2020-11-16T20:32:17.462 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.462 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.462 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.462 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4304.0:1", 2020-11-16T20:32:17.462 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'38", 2020-11-16T20:32:17.462 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "38", 2020-11-16T20:32:17.463 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.463 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.463 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.463 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4307.0:1", 2020-11-16T20:32:17.463 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'39", 2020-11-16T20:32:17.463 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "39", 2020-11-16T20:32:17.463 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.463 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.464 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.464 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4310.0:1", 2020-11-16T20:32:17.464 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'40", 2020-11-16T20:32:17.464 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "40", 2020-11-16T20:32:17.464 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.464 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.464 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.464 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4313.0:1", 2020-11-16T20:32:17.465 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'41", 2020-11-16T20:32:17.465 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "41", 2020-11-16T20:32:17.465 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.465 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.465 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.465 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4316.0:1", 2020-11-16T20:32:17.465 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'42", 2020-11-16T20:32:17.466 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "42", 2020-11-16T20:32:17.466 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.466 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.466 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.466 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4319.0:1", 2020-11-16T20:32:17.466 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'43", 2020-11-16T20:32:17.466 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "43", 2020-11-16T20:32:17.466 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.467 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.467 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.467 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4322.0:1", 2020-11-16T20:32:17.467 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'44", 2020-11-16T20:32:17.467 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "44", 2020-11-16T20:32:17.467 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.467 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.468 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.468 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4325.0:1", 2020-11-16T20:32:17.468 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'45", 2020-11-16T20:32:17.468 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "45", 2020-11-16T20:32:17.468 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.468 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.468 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.468 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4328.0:1", 2020-11-16T20:32:17.469 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'46", 2020-11-16T20:32:17.469 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "46", 2020-11-16T20:32:17.469 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.469 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.469 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.469 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4331.0:1", 2020-11-16T20:32:17.469 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'47", 2020-11-16T20:32:17.469 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "47", 2020-11-16T20:32:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4334.0:1", 2020-11-16T20:32:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'48", 2020-11-16T20:32:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "48", 2020-11-16T20:32:17.470 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4337.0:1", 2020-11-16T20:32:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'49", 2020-11-16T20:32:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "49", 2020-11-16T20:32:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.471 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4340.0:1", 2020-11-16T20:32:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'50", 2020-11-16T20:32:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "50", 2020-11-16T20:32:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.472 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4343.0:1", 2020-11-16T20:32:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'51", 2020-11-16T20:32:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "51", 2020-11-16T20:32:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4346.0:1", 2020-11-16T20:32:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'52", 2020-11-16T20:32:17.473 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "52", 2020-11-16T20:32:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4349.0:1", 2020-11-16T20:32:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'53", 2020-11-16T20:32:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "53", 2020-11-16T20:32:17.474 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.475 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.475 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.475 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4352.0:1", 2020-11-16T20:32:17.475 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'54", 2020-11-16T20:32:17.475 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "54", 2020-11-16T20:32:17.475 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.475 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.475 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.476 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4355.0:1", 2020-11-16T20:32:17.476 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'55", 2020-11-16T20:32:17.476 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "55", 2020-11-16T20:32:17.476 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.476 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.476 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.476 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4358.0:1", 2020-11-16T20:32:17.477 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'56", 2020-11-16T20:32:17.477 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "56", 2020-11-16T20:32:17.477 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.477 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.477 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.477 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4361.0:1", 2020-11-16T20:32:17.477 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'57", 2020-11-16T20:32:17.477 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "57", 2020-11-16T20:32:17.478 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.478 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.478 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.478 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4364.0:1", 2020-11-16T20:32:17.478 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'58", 2020-11-16T20:32:17.478 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "58", 2020-11-16T20:32:17.478 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.478 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.479 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.479 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4367.0:1", 2020-11-16T20:32:17.479 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'59", 2020-11-16T20:32:17.479 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "59", 2020-11-16T20:32:17.479 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.479 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.479 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.480 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4370.0:1", 2020-11-16T20:32:17.480 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'60", 2020-11-16T20:32:17.480 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "60", 2020-11-16T20:32:17.480 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.480 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.480 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.480 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4373.0:1", 2020-11-16T20:32:17.480 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'61", 2020-11-16T20:32:17.481 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "61", 2020-11-16T20:32:17.481 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.481 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.481 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.481 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4376.0:1", 2020-11-16T20:32:17.481 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'62", 2020-11-16T20:32:17.481 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "62", 2020-11-16T20:32:17.481 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.482 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.482 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.482 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4379.0:1", 2020-11-16T20:32:17.482 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'63", 2020-11-16T20:32:17.482 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "63", 2020-11-16T20:32:17.482 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.482 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4382.0:1", 2020-11-16T20:32:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'64", 2020-11-16T20:32:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "64", 2020-11-16T20:32:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.483 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4385.0:1", 2020-11-16T20:32:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'65", 2020-11-16T20:32:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "65", 2020-11-16T20:32:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4388.0:1", 2020-11-16T20:32:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'66", 2020-11-16T20:32:17.484 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "66", 2020-11-16T20:32:17.485 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.485 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.485 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.485 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4391.0:1", 2020-11-16T20:32:17.485 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'67", 2020-11-16T20:32:17.485 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "67", 2020-11-16T20:32:17.485 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.486 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.486 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.486 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4394.0:1", 2020-11-16T20:32:17.486 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'68", 2020-11-16T20:32:17.486 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "68", 2020-11-16T20:32:17.486 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.486 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.486 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.487 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4397.0:1", 2020-11-16T20:32:17.487 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'69", 2020-11-16T20:32:17.487 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "69", 2020-11-16T20:32:17.487 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.487 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.487 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.487 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4400.0:1", 2020-11-16T20:32:17.488 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'70", 2020-11-16T20:32:17.488 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "70", 2020-11-16T20:32:17.488 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.488 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.488 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.488 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4403.0:1", 2020-11-16T20:32:17.488 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'71", 2020-11-16T20:32:17.488 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "71", 2020-11-16T20:32:17.489 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.489 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.489 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.489 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4406.0:1", 2020-11-16T20:32:17.489 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'72", 2020-11-16T20:32:17.489 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "72", 2020-11-16T20:32:17.489 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.490 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.490 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.490 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4409.0:1", 2020-11-16T20:32:17.490 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'73", 2020-11-16T20:32:17.490 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "73", 2020-11-16T20:32:17.490 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.490 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.490 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.491 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4412.0:1", 2020-11-16T20:32:17.491 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'74", 2020-11-16T20:32:17.491 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "74", 2020-11-16T20:32:17.491 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.491 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.491 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.491 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4415.0:1", 2020-11-16T20:32:17.491 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'75", 2020-11-16T20:32:17.492 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "75", 2020-11-16T20:32:17.492 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.492 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.492 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.492 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4418.0:1", 2020-11-16T20:32:17.492 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'76", 2020-11-16T20:32:17.492 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "76", 2020-11-16T20:32:17.493 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.493 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.493 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.493 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4421.0:1", 2020-11-16T20:32:17.493 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'77", 2020-11-16T20:32:17.493 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "77", 2020-11-16T20:32:17.493 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.493 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.494 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.494 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4424.0:1", 2020-11-16T20:32:17.494 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'78", 2020-11-16T20:32:17.494 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "78", 2020-11-16T20:32:17.494 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.494 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.494 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.494 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4427.0:1", 2020-11-16T20:32:17.495 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'79", 2020-11-16T20:32:17.495 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "79", 2020-11-16T20:32:17.495 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.495 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.495 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.495 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4430.0:1", 2020-11-16T20:32:17.495 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'80", 2020-11-16T20:32:17.496 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "80", 2020-11-16T20:32:17.496 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.496 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.496 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.496 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4433.0:1", 2020-11-16T20:32:17.496 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'81", 2020-11-16T20:32:17.496 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "81", 2020-11-16T20:32:17.496 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.497 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.497 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.497 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4436.0:1", 2020-11-16T20:32:17.497 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'82", 2020-11-16T20:32:17.497 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "82", 2020-11-16T20:32:17.497 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.497 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.498 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.498 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4439.0:1", 2020-11-16T20:32:17.498 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'83", 2020-11-16T20:32:17.498 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "83", 2020-11-16T20:32:17.498 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.498 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.498 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.498 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4442.0:1", 2020-11-16T20:32:17.499 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'84", 2020-11-16T20:32:17.499 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "84", 2020-11-16T20:32:17.499 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.499 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.499 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.499 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4445.0:1", 2020-11-16T20:32:17.499 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'85", 2020-11-16T20:32:17.500 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "85", 2020-11-16T20:32:17.500 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.500 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.500 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.500 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4448.0:1", 2020-11-16T20:32:17.500 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'86", 2020-11-16T20:32:17.500 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "86", 2020-11-16T20:32:17.500 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.501 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.501 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.501 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4451.0:1", 2020-11-16T20:32:17.501 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'87", 2020-11-16T20:32:17.501 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "87", 2020-11-16T20:32:17.501 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.501 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.502 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.502 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4454.0:1", 2020-11-16T20:32:17.502 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'88", 2020-11-16T20:32:17.502 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "88", 2020-11-16T20:32:17.502 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.502 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.502 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.502 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4457.0:1", 2020-11-16T20:32:17.503 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'89", 2020-11-16T20:32:17.503 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "89", 2020-11-16T20:32:17.503 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.503 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.503 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.503 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4460.0:1", 2020-11-16T20:32:17.503 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'90", 2020-11-16T20:32:17.504 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "90", 2020-11-16T20:32:17.504 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.504 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.504 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.504 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4463.0:1", 2020-11-16T20:32:17.504 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'91", 2020-11-16T20:32:17.504 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "91", 2020-11-16T20:32:17.504 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.505 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.505 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.505 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4466.0:1", 2020-11-16T20:32:17.505 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'92", 2020-11-16T20:32:17.505 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "92", 2020-11-16T20:32:17.505 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.505 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.505 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.506 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4469.0:1", 2020-11-16T20:32:17.506 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'93", 2020-11-16T20:32:17.506 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "93", 2020-11-16T20:32:17.506 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.506 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.506 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.506 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4472.0:1", 2020-11-16T20:32:17.507 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'94", 2020-11-16T20:32:17.507 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "94", 2020-11-16T20:32:17.507 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.507 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.507 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.507 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4475.0:1", 2020-11-16T20:32:17.507 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'95", 2020-11-16T20:32:17.507 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "95", 2020-11-16T20:32:17.508 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.508 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.508 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.508 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4478.0:1", 2020-11-16T20:32:17.508 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'96", 2020-11-16T20:32:17.508 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "96", 2020-11-16T20:32:17.508 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.509 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.509 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.509 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4481.0:1", 2020-11-16T20:32:17.509 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'97", 2020-11-16T20:32:17.509 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "97", 2020-11-16T20:32:17.509 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.509 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.509 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.510 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4484.0:1", 2020-11-16T20:32:17.510 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'98", 2020-11-16T20:32:17.510 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "98", 2020-11-16T20:32:17.510 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.510 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.510 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.510 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4487.0:1", 2020-11-16T20:32:17.510 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'99", 2020-11-16T20:32:17.511 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "99", 2020-11-16T20:32:17.511 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.511 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.511 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.511 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4490.0:1", 2020-11-16T20:32:17.511 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'100", 2020-11-16T20:32:17.511 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "100", 2020-11-16T20:32:17.512 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.512 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.512 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.512 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4493.0:1", 2020-11-16T20:32:17.512 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'101", 2020-11-16T20:32:17.512 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "101", 2020-11-16T20:32:17.512 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.512 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.513 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.513 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4496.0:1", 2020-11-16T20:32:17.513 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'102", 2020-11-16T20:32:17.513 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "102", 2020-11-16T20:32:17.513 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.513 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.513 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.513 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4499.0:1", 2020-11-16T20:32:17.514 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'103", 2020-11-16T20:32:17.514 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "103", 2020-11-16T20:32:17.514 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.514 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.514 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.514 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4502.0:1", 2020-11-16T20:32:17.514 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'104", 2020-11-16T20:32:17.515 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "104", 2020-11-16T20:32:17.515 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.515 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.515 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.515 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4505.0:1", 2020-11-16T20:32:17.515 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'105", 2020-11-16T20:32:17.515 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "105", 2020-11-16T20:32:17.515 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.516 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.516 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.516 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4508.0:1", 2020-11-16T20:32:17.516 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'106", 2020-11-16T20:32:17.516 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "106", 2020-11-16T20:32:17.516 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.516 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.516 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.517 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4511.0:1", 2020-11-16T20:32:17.517 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'107", 2020-11-16T20:32:17.517 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "107", 2020-11-16T20:32:17.517 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.517 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.517 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.517 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4514.0:1", 2020-11-16T20:32:17.518 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'108", 2020-11-16T20:32:17.518 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "108", 2020-11-16T20:32:17.518 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.518 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.518 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.518 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4517.0:1", 2020-11-16T20:32:17.518 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'109", 2020-11-16T20:32:17.518 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "109", 2020-11-16T20:32:17.519 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.519 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.519 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.519 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4520.0:1", 2020-11-16T20:32:17.519 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'110", 2020-11-16T20:32:17.519 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "110", 2020-11-16T20:32:17.519 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.520 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.520 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.520 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4523.0:1", 2020-11-16T20:32:17.520 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'111", 2020-11-16T20:32:17.520 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "111", 2020-11-16T20:32:17.520 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.520 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.520 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.521 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4526.0:1", 2020-11-16T20:32:17.521 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'112", 2020-11-16T20:32:17.521 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "112", 2020-11-16T20:32:17.521 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.521 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.521 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.521 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4529.0:1", 2020-11-16T20:32:17.521 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'113", 2020-11-16T20:32:17.522 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "113", 2020-11-16T20:32:17.522 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.522 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.522 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.522 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4532.0:1", 2020-11-16T20:32:17.522 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'114", 2020-11-16T20:32:17.522 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "114", 2020-11-16T20:32:17.523 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.523 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.523 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.523 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4535.0:1", 2020-11-16T20:32:17.523 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'115", 2020-11-16T20:32:17.523 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "115", 2020-11-16T20:32:17.523 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.523 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.524 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.524 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4538.0:1", 2020-11-16T20:32:17.524 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'116", 2020-11-16T20:32:17.524 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "116", 2020-11-16T20:32:17.524 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.524 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.524 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.524 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4541.0:1", 2020-11-16T20:32:17.525 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'117", 2020-11-16T20:32:17.525 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "117", 2020-11-16T20:32:17.525 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.525 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.525 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.525 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4544.0:1", 2020-11-16T20:32:17.525 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'118", 2020-11-16T20:32:17.526 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "118", 2020-11-16T20:32:17.526 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.526 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.526 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.526 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4547.0:1", 2020-11-16T20:32:17.526 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'119", 2020-11-16T20:32:17.526 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "119", 2020-11-16T20:32:17.526 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.527 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.527 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.527 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4550.0:1", 2020-11-16T20:32:17.527 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'120", 2020-11-16T20:32:17.527 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "120", 2020-11-16T20:32:17.527 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.527 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.527 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.528 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4553.0:1", 2020-11-16T20:32:17.528 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'121", 2020-11-16T20:32:17.528 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "121", 2020-11-16T20:32:17.528 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.528 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.528 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.528 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4556.0:1", 2020-11-16T20:32:17.529 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'122", 2020-11-16T20:32:17.529 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "122", 2020-11-16T20:32:17.529 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.529 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.529 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.529 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4559.0:1", 2020-11-16T20:32:17.529 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'123", 2020-11-16T20:32:17.530 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "123", 2020-11-16T20:32:17.530 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.530 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.530 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.530 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4562.0:1", 2020-11-16T20:32:17.530 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'124", 2020-11-16T20:32:17.530 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "124", 2020-11-16T20:32:17.530 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.531 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.531 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.531 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4565.0:1", 2020-11-16T20:32:17.531 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'125", 2020-11-16T20:32:17.531 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "125", 2020-11-16T20:32:17.531 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.531 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.532 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.532 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4568.0:1", 2020-11-16T20:32:17.532 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'126", 2020-11-16T20:32:17.532 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "126", 2020-11-16T20:32:17.532 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.532 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.532 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.532 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4571.0:1", 2020-11-16T20:32:17.533 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'127", 2020-11-16T20:32:17.533 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "127", 2020-11-16T20:32:17.533 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.533 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.533 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.533 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4574.0:1", 2020-11-16T20:32:17.533 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'128", 2020-11-16T20:32:17.533 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "128", 2020-11-16T20:32:17.534 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.534 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.534 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.534 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4577.0:1", 2020-11-16T20:32:17.534 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'129", 2020-11-16T20:32:17.534 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "129", 2020-11-16T20:32:17.534 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.535 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.535 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.535 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4580.0:1", 2020-11-16T20:32:17.535 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'130", 2020-11-16T20:32:17.535 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "130", 2020-11-16T20:32:17.535 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.535 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.535 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.536 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4583.0:1", 2020-11-16T20:32:17.536 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'131", 2020-11-16T20:32:17.536 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "131", 2020-11-16T20:32:17.536 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.536 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.536 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.536 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4586.0:1", 2020-11-16T20:32:17.536 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'132", 2020-11-16T20:32:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "132", 2020-11-16T20:32:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4589.0:1", 2020-11-16T20:32:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'133", 2020-11-16T20:32:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "133", 2020-11-16T20:32:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4592.0:1", 2020-11-16T20:32:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'134", 2020-11-16T20:32:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "134", 2020-11-16T20:32:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4595.0:1", 2020-11-16T20:32:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'135", 2020-11-16T20:32:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "135", 2020-11-16T20:32:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4598.0:1", 2020-11-16T20:32:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'136", 2020-11-16T20:32:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "136", 2020-11-16T20:32:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4601.0:1", 2020-11-16T20:32:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'137", 2020-11-16T20:32:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "137", 2020-11-16T20:32:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4604.0:1", 2020-11-16T20:32:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'138", 2020-11-16T20:32:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "138", 2020-11-16T20:32:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.542 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.542 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.542 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4607.0:1", 2020-11-16T20:32:17.542 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'139", 2020-11-16T20:32:17.542 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "139", 2020-11-16T20:32:17.542 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.542 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.542 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.543 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4610.0:1", 2020-11-16T20:32:17.543 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'140", 2020-11-16T20:32:17.543 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "140", 2020-11-16T20:32:17.543 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.543 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.543 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.543 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4613.0:1", 2020-11-16T20:32:17.544 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'141", 2020-11-16T20:32:17.544 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "141", 2020-11-16T20:32:17.544 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.544 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.544 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.544 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4616.0:1", 2020-11-16T20:32:17.544 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'142", 2020-11-16T20:32:17.544 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "142", 2020-11-16T20:32:17.545 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.545 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.545 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.545 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4619.0:1", 2020-11-16T20:32:17.545 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'143", 2020-11-16T20:32:17.545 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "143", 2020-11-16T20:32:17.545 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.545 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.546 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.546 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4622.0:1", 2020-11-16T20:32:17.546 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'144", 2020-11-16T20:32:17.546 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "144", 2020-11-16T20:32:17.546 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.546 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.546 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.547 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4625.0:1", 2020-11-16T20:32:17.547 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'145", 2020-11-16T20:32:17.547 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "145", 2020-11-16T20:32:17.547 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.547 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.547 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.547 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4628.0:1", 2020-11-16T20:32:17.547 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'146", 2020-11-16T20:32:17.548 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "146", 2020-11-16T20:32:17.548 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.548 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.548 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.548 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4631.0:1", 2020-11-16T20:32:17.548 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'147", 2020-11-16T20:32:17.957 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "147", 2020-11-16T20:32:17.958 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.958 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.958 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:32:17.958 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4634.0:1", 2020-11-16T20:32:17.958 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "33'148", 2020-11-16T20:32:17.958 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "148", 2020-11-16T20:32:17.959 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:32:17.959 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:32:17.959 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:32:17.959 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:32:17.959 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_missing_t": { 2020-11-16T20:32:17.960 INFO:tasks.workunit.client.0.smithi189.stdout: "missing": [], 2020-11-16T20:32:17.960 INFO:tasks.workunit.client.0.smithi189.stdout: "may_include_deletes": true 2020-11-16T20:32:17.960 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:32:17.960 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:32:17.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-11-16T20:32:17.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=2 2020-11-16T20:32:17.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:85: _common_test: '[' 2 '!=' 2 ']' 2020-11-16T20:32:17.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-11-16T20:32:17.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=148 2020-11-16T20:32:17.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:90: _common_test: '[' 148 '!=' 148 ']' 2020-11-16T20:32:17.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-11-16T20:32:17.981 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to max 2 2020-11-16T20:32:17.981 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to copy log version 33'150 2020-11-16T20:32:17.981 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to copy log version 33'149 2020-11-16T20:32:17.981 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after earliest_dup_version 0 2020-11-16T20:32:17.981 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 32'1 2020-11-16T20:32:17.981 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'2 2020-11-16T20:32:17.982 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'3 2020-11-16T20:32:17.982 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'4 2020-11-16T20:32:17.982 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'5 2020-11-16T20:32:17.982 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'6 2020-11-16T20:32:17.982 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'7 2020-11-16T20:32:17.982 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'8 2020-11-16T20:32:17.982 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'9 2020-11-16T20:32:17.983 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'10 2020-11-16T20:32:17.983 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'11 2020-11-16T20:32:17.983 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'12 2020-11-16T20:32:17.983 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'13 2020-11-16T20:32:17.983 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'14 2020-11-16T20:32:17.983 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.612+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'15 2020-11-16T20:32:17.983 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'16 2020-11-16T20:32:17.984 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'17 2020-11-16T20:32:17.984 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'18 2020-11-16T20:32:17.984 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'19 2020-11-16T20:32:17.984 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'20 2020-11-16T20:32:17.984 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'21 2020-11-16T20:32:17.984 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'22 2020-11-16T20:32:17.984 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'23 2020-11-16T20:32:17.985 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'24 2020-11-16T20:32:17.985 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'25 2020-11-16T20:32:17.985 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'26 2020-11-16T20:32:17.985 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'27 2020-11-16T20:32:17.985 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'28 2020-11-16T20:32:17.985 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'29 2020-11-16T20:32:17.985 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'30 2020-11-16T20:32:17.985 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'31 2020-11-16T20:32:17.986 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'32 2020-11-16T20:32:17.986 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'33 2020-11-16T20:32:17.986 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'34 2020-11-16T20:32:17.986 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'35 2020-11-16T20:32:17.986 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'36 2020-11-16T20:32:17.986 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'37 2020-11-16T20:32:17.986 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'38 2020-11-16T20:32:17.987 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'39 2020-11-16T20:32:17.987 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'40 2020-11-16T20:32:17.987 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'41 2020-11-16T20:32:17.987 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'42 2020-11-16T20:32:17.987 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'43 2020-11-16T20:32:17.987 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'44 2020-11-16T20:32:17.987 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'45 2020-11-16T20:32:17.987 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'46 2020-11-16T20:32:17.988 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'47 2020-11-16T20:32:17.988 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'48 2020-11-16T20:32:17.988 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'49 2020-11-16T20:32:17.988 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'50 2020-11-16T20:32:17.988 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'51 2020-11-16T20:32:17.988 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'52 2020-11-16T20:32:17.988 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'53 2020-11-16T20:32:17.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'54 2020-11-16T20:32:17.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'55 2020-11-16T20:32:17.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'56 2020-11-16T20:32:17.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'57 2020-11-16T20:32:17.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'58 2020-11-16T20:32:17.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'59 2020-11-16T20:32:17.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'60 2020-11-16T20:32:17.989 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'61 2020-11-16T20:32:17.990 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'62 2020-11-16T20:32:17.990 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'63 2020-11-16T20:32:17.990 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'64 2020-11-16T20:32:17.990 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'65 2020-11-16T20:32:17.990 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'66 2020-11-16T20:32:17.990 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'67 2020-11-16T20:32:17.990 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'68 2020-11-16T20:32:17.991 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'69 2020-11-16T20:32:17.991 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'70 2020-11-16T20:32:17.991 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'71 2020-11-16T20:32:17.991 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'72 2020-11-16T20:32:17.991 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'73 2020-11-16T20:32:17.991 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'74 2020-11-16T20:32:17.991 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'75 2020-11-16T20:32:17.992 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'76 2020-11-16T20:32:17.992 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'77 2020-11-16T20:32:17.992 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'78 2020-11-16T20:32:17.992 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'79 2020-11-16T20:32:17.992 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'80 2020-11-16T20:32:17.992 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'81 2020-11-16T20:32:17.992 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'82 2020-11-16T20:32:17.993 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'83 2020-11-16T20:32:17.993 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'84 2020-11-16T20:32:17.993 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'85 2020-11-16T20:32:17.993 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'86 2020-11-16T20:32:17.993 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'87 2020-11-16T20:32:17.993 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'88 2020-11-16T20:32:17.993 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'89 2020-11-16T20:32:17.993 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'90 2020-11-16T20:32:17.994 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'91 2020-11-16T20:32:17.994 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'92 2020-11-16T20:32:17.994 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'93 2020-11-16T20:32:17.994 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'94 2020-11-16T20:32:17.994 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'95 2020-11-16T20:32:17.994 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'96 2020-11-16T20:32:17.994 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'97 2020-11-16T20:32:17.995 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'98 2020-11-16T20:32:17.995 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'99 2020-11-16T20:32:17.995 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'100 2020-11-16T20:32:17.995 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'101 2020-11-16T20:32:17.995 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'102 2020-11-16T20:32:17.995 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'103 2020-11-16T20:32:17.995 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'104 2020-11-16T20:32:17.995 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'105 2020-11-16T20:32:17.996 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'106 2020-11-16T20:32:17.996 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'107 2020-11-16T20:32:17.996 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'108 2020-11-16T20:32:17.996 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'109 2020-11-16T20:32:17.996 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'110 2020-11-16T20:32:17.996 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'111 2020-11-16T20:32:17.996 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'112 2020-11-16T20:32:17.997 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'113 2020-11-16T20:32:17.997 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'114 2020-11-16T20:32:17.997 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'115 2020-11-16T20:32:17.997 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'116 2020-11-16T20:32:17.997 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'117 2020-11-16T20:32:17.997 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'118 2020-11-16T20:32:17.997 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'119 2020-11-16T20:32:17.997 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'120 2020-11-16T20:32:17.998 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'121 2020-11-16T20:32:17.998 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'122 2020-11-16T20:32:17.998 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'123 2020-11-16T20:32:17.998 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'124 2020-11-16T20:32:17.998 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'125 2020-11-16T20:32:17.998 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'126 2020-11-16T20:32:17.998 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'127 2020-11-16T20:32:17.999 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'128 2020-11-16T20:32:17.999 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'129 2020-11-16T20:32:17.999 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'130 2020-11-16T20:32:17.999 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'131 2020-11-16T20:32:17.999 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'132 2020-11-16T20:32:17.999 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'133 2020-11-16T20:32:17.999 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'134 2020-11-16T20:32:18.000 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'135 2020-11-16T20:32:18.000 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'136 2020-11-16T20:32:18.000 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'137 2020-11-16T20:32:18.000 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'138 2020-11-16T20:32:18.000 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'139 2020-11-16T20:32:18.000 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'140 2020-11-16T20:32:18.000 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-11-16T20:32:18.000 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-11-16T20:32:18.001 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-11-16T20:32:18.001 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-11-16T20:32:18.001 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-11-16T20:32:18.001 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-11-16T20:32:18.001 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-11-16T20:32:18.001 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-11-16T20:32:18.001 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to max 2 2020-11-16T20:32:18.002 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to copy log version 33'150 2020-11-16T20:32:18.002 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to copy log version 33'149 2020-11-16T20:32:18.002 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after earliest_dup_version 0 2020-11-16T20:32:18.002 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 32'1 2020-11-16T20:32:18.002 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'2 2020-11-16T20:32:18.002 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'3 2020-11-16T20:32:18.002 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'4 2020-11-16T20:32:18.002 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'5 2020-11-16T20:32:18.003 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'6 2020-11-16T20:32:18.003 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'7 2020-11-16T20:32:18.003 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'8 2020-11-16T20:32:18.003 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'9 2020-11-16T20:32:18.003 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'10 2020-11-16T20:32:18.003 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'11 2020-11-16T20:32:18.003 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'12 2020-11-16T20:32:18.004 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'13 2020-11-16T20:32:18.004 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'14 2020-11-16T20:32:18.004 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'15 2020-11-16T20:32:18.004 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'16 2020-11-16T20:32:18.004 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'17 2020-11-16T20:32:18.004 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'18 2020-11-16T20:32:18.004 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'19 2020-11-16T20:32:18.004 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'20 2020-11-16T20:32:18.005 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'21 2020-11-16T20:32:18.005 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'22 2020-11-16T20:32:18.005 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'23 2020-11-16T20:32:18.005 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'24 2020-11-16T20:32:18.005 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'25 2020-11-16T20:32:18.005 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'26 2020-11-16T20:32:18.005 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'27 2020-11-16T20:32:18.006 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'28 2020-11-16T20:32:18.006 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'29 2020-11-16T20:32:18.006 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'30 2020-11-16T20:32:18.006 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'31 2020-11-16T20:32:18.006 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'32 2020-11-16T20:32:18.006 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'33 2020-11-16T20:32:18.006 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'34 2020-11-16T20:32:18.007 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'35 2020-11-16T20:32:18.007 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'36 2020-11-16T20:32:18.007 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'37 2020-11-16T20:32:18.007 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'38 2020-11-16T20:32:18.007 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'39 2020-11-16T20:32:18.007 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'40 2020-11-16T20:32:18.007 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'41 2020-11-16T20:32:18.007 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'42 2020-11-16T20:32:18.008 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'43 2020-11-16T20:32:18.008 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'44 2020-11-16T20:32:18.008 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'45 2020-11-16T20:32:18.008 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'46 2020-11-16T20:32:18.008 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'47 2020-11-16T20:32:18.008 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'48 2020-11-16T20:32:18.008 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'49 2020-11-16T20:32:18.008 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'50 2020-11-16T20:32:18.009 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'51 2020-11-16T20:32:18.009 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'52 2020-11-16T20:32:18.009 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'53 2020-11-16T20:32:18.009 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'54 2020-11-16T20:32:18.009 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'55 2020-11-16T20:32:18.009 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'56 2020-11-16T20:32:18.009 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'57 2020-11-16T20:32:18.010 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'58 2020-11-16T20:32:18.010 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'59 2020-11-16T20:32:18.010 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'60 2020-11-16T20:32:18.010 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'61 2020-11-16T20:32:18.010 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'62 2020-11-16T20:32:18.010 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'63 2020-11-16T20:32:18.010 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'64 2020-11-16T20:32:18.011 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'65 2020-11-16T20:32:18.011 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'66 2020-11-16T20:32:18.011 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'67 2020-11-16T20:32:18.011 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'68 2020-11-16T20:32:18.011 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'69 2020-11-16T20:32:18.011 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'70 2020-11-16T20:32:18.011 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'71 2020-11-16T20:32:18.011 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'72 2020-11-16T20:32:18.012 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'73 2020-11-16T20:32:18.012 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'74 2020-11-16T20:32:18.012 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'75 2020-11-16T20:32:18.012 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'76 2020-11-16T20:32:18.012 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'77 2020-11-16T20:32:18.012 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'78 2020-11-16T20:32:18.012 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'79 2020-11-16T20:32:18.013 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'80 2020-11-16T20:32:18.013 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'81 2020-11-16T20:32:18.013 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'82 2020-11-16T20:32:18.013 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'83 2020-11-16T20:32:18.013 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'84 2020-11-16T20:32:18.014 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'85 2020-11-16T20:32:18.014 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'86 2020-11-16T20:32:18.014 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'87 2020-11-16T20:32:18.014 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'88 2020-11-16T20:32:18.014 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'89 2020-11-16T20:32:18.014 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'90 2020-11-16T20:32:18.014 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'91 2020-11-16T20:32:18.015 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'92 2020-11-16T20:32:18.015 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'93 2020-11-16T20:32:18.015 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'94 2020-11-16T20:32:18.015 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'95 2020-11-16T20:32:18.015 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'96 2020-11-16T20:32:18.015 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'97 2020-11-16T20:32:18.015 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'98 2020-11-16T20:32:18.016 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'99 2020-11-16T20:32:18.016 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'100 2020-11-16T20:32:18.016 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'101 2020-11-16T20:32:18.016 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'102 2020-11-16T20:32:18.016 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'103 2020-11-16T20:32:18.016 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'104 2020-11-16T20:32:18.016 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'105 2020-11-16T20:32:18.016 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'106 2020-11-16T20:32:18.017 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'107 2020-11-16T20:32:18.017 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'108 2020-11-16T20:32:18.017 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'109 2020-11-16T20:32:18.017 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'110 2020-11-16T20:32:18.017 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'111 2020-11-16T20:32:18.017 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'112 2020-11-16T20:32:18.017 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'113 2020-11-16T20:32:18.018 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'114 2020-11-16T20:32:18.018 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'115 2020-11-16T20:32:18.018 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'116 2020-11-16T20:32:18.018 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'117 2020-11-16T20:32:18.018 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'118 2020-11-16T20:32:18.018 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'119 2020-11-16T20:32:18.018 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'120 2020-11-16T20:32:18.018 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'121 2020-11-16T20:32:18.019 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'122 2020-11-16T20:32:18.019 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'123 2020-11-16T20:32:18.019 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'124 2020-11-16T20:32:18.019 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'125 2020-11-16T20:32:18.019 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'126 2020-11-16T20:32:18.019 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'127 2020-11-16T20:32:18.019 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'128 2020-11-16T20:32:18.020 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'129 2020-11-16T20:32:18.020 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'130 2020-11-16T20:32:18.020 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'131 2020-11-16T20:32:18.020 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'132 2020-11-16T20:32:18.020 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'133 2020-11-16T20:32:18.020 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'134 2020-11-16T20:32:18.020 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'135 2020-11-16T20:32:18.020 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'136 2020-11-16T20:32:18.021 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'137 2020-11-16T20:32:18.021 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'138 2020-11-16T20:32:18.021 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'139 2020-11-16T20:32:18.021 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'140 2020-11-16T20:32:18.021 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-11-16T20:32:18.021 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-11-16T20:32:18.021 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-11-16T20:32:18.022 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-11-16T20:32:18.022 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-11-16T20:32:18.022 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-11-16T20:32:18.022 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-11-16T20:32:18.022 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-11-16T20:32:18.022 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to max 2 2020-11-16T20:32:18.022 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to copy log version 33'150 2020-11-16T20:32:18.022 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to copy log version 33'149 2020-11-16T20:32:18.023 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after earliest_dup_version 0 2020-11-16T20:32:18.023 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 32'1 2020-11-16T20:32:18.023 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'2 2020-11-16T20:32:18.023 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'3 2020-11-16T20:32:18.023 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'4 2020-11-16T20:32:18.023 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'5 2020-11-16T20:32:18.023 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'6 2020-11-16T20:32:18.024 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'7 2020-11-16T20:32:18.024 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'8 2020-11-16T20:32:18.024 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'9 2020-11-16T20:32:18.024 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'10 2020-11-16T20:32:18.024 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'11 2020-11-16T20:32:18.024 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'12 2020-11-16T20:32:18.024 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'13 2020-11-16T20:32:18.025 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'14 2020-11-16T20:32:18.025 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'15 2020-11-16T20:32:18.025 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'16 2020-11-16T20:32:18.025 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'17 2020-11-16T20:32:18.025 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'18 2020-11-16T20:32:18.025 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'19 2020-11-16T20:32:18.025 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'20 2020-11-16T20:32:18.025 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'21 2020-11-16T20:32:18.026 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'22 2020-11-16T20:32:18.026 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'23 2020-11-16T20:32:18.026 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'24 2020-11-16T20:32:18.026 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'25 2020-11-16T20:32:18.026 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'26 2020-11-16T20:32:18.026 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'27 2020-11-16T20:32:18.026 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'28 2020-11-16T20:32:18.027 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'29 2020-11-16T20:32:18.027 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'30 2020-11-16T20:32:18.027 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'31 2020-11-16T20:32:18.027 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'32 2020-11-16T20:32:18.027 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'33 2020-11-16T20:32:18.027 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'34 2020-11-16T20:32:18.027 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'35 2020-11-16T20:32:18.027 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'36 2020-11-16T20:32:18.028 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'37 2020-11-16T20:32:18.028 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'38 2020-11-16T20:32:18.028 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'39 2020-11-16T20:32:18.028 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'40 2020-11-16T20:32:18.028 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'41 2020-11-16T20:32:18.028 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'42 2020-11-16T20:32:18.028 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'43 2020-11-16T20:32:18.029 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'44 2020-11-16T20:32:18.029 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'45 2020-11-16T20:32:18.029 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'46 2020-11-16T20:32:18.029 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'47 2020-11-16T20:32:18.029 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'48 2020-11-16T20:32:18.029 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'49 2020-11-16T20:32:18.029 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'50 2020-11-16T20:32:18.029 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'51 2020-11-16T20:32:18.030 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'52 2020-11-16T20:32:18.030 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'53 2020-11-16T20:32:18.030 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'54 2020-11-16T20:32:18.030 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'55 2020-11-16T20:32:18.030 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'56 2020-11-16T20:32:18.030 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'57 2020-11-16T20:32:18.030 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'58 2020-11-16T20:32:18.031 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'59 2020-11-16T20:32:18.031 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'60 2020-11-16T20:32:18.031 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'61 2020-11-16T20:32:18.031 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'62 2020-11-16T20:32:18.031 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'63 2020-11-16T20:32:18.031 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'64 2020-11-16T20:32:18.031 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'65 2020-11-16T20:32:18.031 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'66 2020-11-16T20:32:18.032 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'67 2020-11-16T20:32:18.032 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'68 2020-11-16T20:32:18.032 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'69 2020-11-16T20:32:18.032 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'70 2020-11-16T20:32:18.032 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'71 2020-11-16T20:32:18.032 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'72 2020-11-16T20:32:18.032 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'73 2020-11-16T20:32:18.033 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'74 2020-11-16T20:32:18.033 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'75 2020-11-16T20:32:18.033 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'76 2020-11-16T20:32:18.033 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'77 2020-11-16T20:32:18.033 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'78 2020-11-16T20:32:18.033 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'79 2020-11-16T20:32:18.033 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'80 2020-11-16T20:32:18.033 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'81 2020-11-16T20:32:18.034 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'82 2020-11-16T20:32:18.034 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'83 2020-11-16T20:32:18.034 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'84 2020-11-16T20:32:18.034 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'85 2020-11-16T20:32:18.034 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'86 2020-11-16T20:32:18.034 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'87 2020-11-16T20:32:18.034 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'88 2020-11-16T20:32:18.035 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'89 2020-11-16T20:32:18.035 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'90 2020-11-16T20:32:18.035 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'91 2020-11-16T20:32:18.035 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'92 2020-11-16T20:32:18.035 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'93 2020-11-16T20:32:18.035 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'94 2020-11-16T20:32:18.035 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'95 2020-11-16T20:32:18.035 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'96 2020-11-16T20:32:18.036 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'97 2020-11-16T20:32:18.036 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'98 2020-11-16T20:32:18.036 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'99 2020-11-16T20:32:18.036 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'100 2020-11-16T20:32:18.036 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'101 2020-11-16T20:32:18.036 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'102 2020-11-16T20:32:18.036 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'103 2020-11-16T20:32:18.036 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'104 2020-11-16T20:32:18.037 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'105 2020-11-16T20:32:18.037 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'106 2020-11-16T20:32:18.037 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'107 2020-11-16T20:32:18.037 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'108 2020-11-16T20:32:18.037 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'109 2020-11-16T20:32:18.037 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'110 2020-11-16T20:32:18.037 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'111 2020-11-16T20:32:18.038 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'112 2020-11-16T20:32:18.038 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'113 2020-11-16T20:32:18.038 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'114 2020-11-16T20:32:18.038 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'115 2020-11-16T20:32:18.038 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'116 2020-11-16T20:32:18.038 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'117 2020-11-16T20:32:18.038 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'118 2020-11-16T20:32:18.039 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'119 2020-11-16T20:32:18.039 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'120 2020-11-16T20:32:18.039 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'121 2020-11-16T20:32:18.039 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'122 2020-11-16T20:32:18.039 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'123 2020-11-16T20:32:18.039 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'124 2020-11-16T20:32:18.039 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'125 2020-11-16T20:32:18.039 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'126 2020-11-16T20:32:18.040 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'127 2020-11-16T20:32:18.040 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'128 2020-11-16T20:32:18.042 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'129 2020-11-16T20:32:18.042 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'130 2020-11-16T20:32:18.042 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'131 2020-11-16T20:32:18.042 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'132 2020-11-16T20:32:18.043 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'133 2020-11-16T20:32:18.043 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'134 2020-11-16T20:32:18.043 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'135 2020-11-16T20:32:18.043 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'136 2020-11-16T20:32:18.043 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'137 2020-11-16T20:32:18.043 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'138 2020-11-16T20:32:18.043 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'139 2020-11-16T20:32:18.043 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'140 2020-11-16T20:32:18.044 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-11-16T20:32:18.044 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-11-16T20:32:18.044 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-11-16T20:32:18.044 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-11-16T20:32:18.044 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-11-16T20:32:18.044 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-11-16T20:32:18.044 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-11-16T20:32:18.045 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:31:23.616+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-11-16T20:32:18.045 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_after v 33'150 2020-11-16T20:32:18.045 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after earliest_dup_version 0 2020-11-16T20:32:18.045 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 32'1 2020-11-16T20:32:18.045 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'2 2020-11-16T20:32:18.045 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'3 2020-11-16T20:32:18.045 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'4 2020-11-16T20:32:18.046 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'5 2020-11-16T20:32:18.046 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'6 2020-11-16T20:32:18.046 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'7 2020-11-16T20:32:18.046 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'8 2020-11-16T20:32:18.046 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'9 2020-11-16T20:32:18.046 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'10 2020-11-16T20:32:18.046 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'11 2020-11-16T20:32:18.047 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'12 2020-11-16T20:32:18.047 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'13 2020-11-16T20:32:18.047 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'14 2020-11-16T20:32:18.047 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'15 2020-11-16T20:32:18.047 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'16 2020-11-16T20:32:18.047 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'17 2020-11-16T20:32:18.047 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'18 2020-11-16T20:32:18.047 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'19 2020-11-16T20:32:18.048 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'20 2020-11-16T20:32:18.048 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'21 2020-11-16T20:32:18.048 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'22 2020-11-16T20:32:18.048 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'23 2020-11-16T20:32:18.048 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'24 2020-11-16T20:32:18.048 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'25 2020-11-16T20:32:18.048 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'26 2020-11-16T20:32:18.049 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'27 2020-11-16T20:32:18.049 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'28 2020-11-16T20:32:18.049 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'29 2020-11-16T20:32:18.049 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'30 2020-11-16T20:32:18.049 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'31 2020-11-16T20:32:18.049 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'32 2020-11-16T20:32:18.049 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'33 2020-11-16T20:32:18.049 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'34 2020-11-16T20:32:18.050 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'35 2020-11-16T20:32:18.050 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'36 2020-11-16T20:32:18.050 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'37 2020-11-16T20:32:18.050 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'38 2020-11-16T20:32:18.050 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'39 2020-11-16T20:32:18.050 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'40 2020-11-16T20:32:18.050 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'41 2020-11-16T20:32:18.051 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'42 2020-11-16T20:32:18.051 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'43 2020-11-16T20:32:18.051 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'44 2020-11-16T20:32:18.051 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'45 2020-11-16T20:32:18.051 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'46 2020-11-16T20:32:18.051 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'47 2020-11-16T20:32:18.051 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'48 2020-11-16T20:32:18.051 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'49 2020-11-16T20:32:18.052 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'50 2020-11-16T20:32:18.052 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'51 2020-11-16T20:32:18.052 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'52 2020-11-16T20:32:18.052 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'53 2020-11-16T20:32:18.052 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'54 2020-11-16T20:32:18.052 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'55 2020-11-16T20:32:18.052 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'56 2020-11-16T20:32:18.053 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'57 2020-11-16T20:32:18.053 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'58 2020-11-16T20:32:18.053 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'59 2020-11-16T20:32:18.053 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'60 2020-11-16T20:32:18.053 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'61 2020-11-16T20:32:18.053 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'62 2020-11-16T20:32:18.053 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'63 2020-11-16T20:32:18.053 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'64 2020-11-16T20:32:18.054 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'65 2020-11-16T20:32:18.054 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'66 2020-11-16T20:32:18.054 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'67 2020-11-16T20:32:18.054 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'68 2020-11-16T20:32:18.054 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'69 2020-11-16T20:32:18.054 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'70 2020-11-16T20:32:18.054 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'71 2020-11-16T20:32:18.055 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'72 2020-11-16T20:32:18.055 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'73 2020-11-16T20:32:18.055 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'74 2020-11-16T20:32:18.055 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'75 2020-11-16T20:32:18.055 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'76 2020-11-16T20:32:18.055 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'77 2020-11-16T20:32:18.055 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'78 2020-11-16T20:32:18.055 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'79 2020-11-16T20:32:18.056 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'80 2020-11-16T20:32:18.056 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'81 2020-11-16T20:32:18.056 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'82 2020-11-16T20:32:18.056 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'83 2020-11-16T20:32:18.056 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'84 2020-11-16T20:32:18.056 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'85 2020-11-16T20:32:18.056 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'86 2020-11-16T20:32:18.057 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'87 2020-11-16T20:32:18.057 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'88 2020-11-16T20:32:18.057 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'89 2020-11-16T20:32:18.057 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'90 2020-11-16T20:32:18.057 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'91 2020-11-16T20:32:18.057 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'92 2020-11-16T20:32:18.057 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'93 2020-11-16T20:32:18.057 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'94 2020-11-16T20:32:18.058 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'95 2020-11-16T20:32:18.058 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'96 2020-11-16T20:32:18.058 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'97 2020-11-16T20:32:18.058 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'98 2020-11-16T20:32:18.058 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'99 2020-11-16T20:32:18.058 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup version 33'100 2020-11-16T20:32:18.058 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'101 2020-11-16T20:32:18.059 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'102 2020-11-16T20:32:18.059 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'103 2020-11-16T20:32:18.059 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'104 2020-11-16T20:32:18.059 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'105 2020-11-16T20:32:18.059 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'106 2020-11-16T20:32:18.059 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'107 2020-11-16T20:32:18.059 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'108 2020-11-16T20:32:18.060 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'109 2020-11-16T20:32:18.060 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'110 2020-11-16T20:32:18.060 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'111 2020-11-16T20:32:18.060 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'112 2020-11-16T20:32:18.060 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'113 2020-11-16T20:32:18.060 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'114 2020-11-16T20:32:18.060 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'115 2020-11-16T20:32:18.060 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'116 2020-11-16T20:32:18.061 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'117 2020-11-16T20:32:18.061 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'118 2020-11-16T20:32:18.061 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'119 2020-11-16T20:32:18.061 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'120 2020-11-16T20:32:18.061 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'121 2020-11-16T20:32:18.061 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'122 2020-11-16T20:32:18.061 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'123 2020-11-16T20:32:18.062 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'124 2020-11-16T20:32:18.062 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'125 2020-11-16T20:32:18.062 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'126 2020-11-16T20:32:18.062 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'127 2020-11-16T20:32:18.062 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'128 2020-11-16T20:32:18.062 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'129 2020-11-16T20:32:18.062 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'130 2020-11-16T20:32:18.062 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'131 2020-11-16T20:32:18.063 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'132 2020-11-16T20:32:18.063 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'133 2020-11-16T20:32:18.063 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'134 2020-11-16T20:32:18.063 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'135 2020-11-16T20:32:18.063 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'136 2020-11-16T20:32:18.063 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'137 2020-11-16T20:32:18.063 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'138 2020-11-16T20:32:18.064 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'139 2020-11-16T20:32:18.064 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'140 2020-11-16T20:32:18.064 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-11-16T20:32:18.064 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-11-16T20:32:18.064 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-11-16T20:32:18.064 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-11-16T20:32:18.064 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-11-16T20:32:18.064 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-11-16T20:32:18.065 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-11-16T20:32:18.065 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-11-16T20:32:18.065 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'149 2020-11-16T20:32:18.065 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:32:04.648+0000 7fda7aad1700 20 copy_up_to/copy_after copy dup from log version 33'150 2020-11-16T20:32:18.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-11-16T20:32:18.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2020-11-16T20:32:18.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-11-16T20:32:18.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:18.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:32:18.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-11-16T20:32:18.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:32:18.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:32:18.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:32:18.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:32:18.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:32:18.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:32:18.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:32:18.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:32:18.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:32:18.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:32:18.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:32:18.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:32:18.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:32:18.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:32:18.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:32:18.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:32:18.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:32:18.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:32:18.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-11-16T20:32:18.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:32:18.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:18.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90694 2020-11-16T20:32:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:32:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:32:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-11-16T20:32:18.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-11-16T20:32:18.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:18.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-11-16T20:32:18.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:18.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:32:18.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-11-16T20:32:18.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:32:18.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:32:18.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:32:18.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:32:18.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:32:18.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:32:18.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:32:18.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:32:18.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:32:18.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:32:18.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:32:18.074 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:32:18.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:32:18.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:32:18.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:32:18.074 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:32:18.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:32:18.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:32:18.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-11-16T20:32:18.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:32:18.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:18.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:18.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90694 2020-11-16T20:32:18.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:32:18.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:32:18.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-11-16T20:32:18.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:32:18.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:18.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:18.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.90694 2020-11-16T20:32:18.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:32:18.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T20:32:18.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:32:18.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-11-16T20:32:18.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_recovery_1 td/osd-backfill-recovery-log 2020-11-16T20:32:18.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:121: TEST_recovery_1: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:18.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:123: TEST_recovery_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 40 0 40 2020-11-16T20:32:18.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:18.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-11-16T20:32:18.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=40 2020-11-16T20:32:18.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=0 2020-11-16T20:32:18.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=40 2020-11-16T20:32:18.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-11-16T20:32:18.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-11-16T20:32:18.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-11-16T20:32:18.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:18.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:32:18.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:32:18.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:32:18.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-11-16T20:32:18.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-11-16T20:32:18.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:32:18.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:32:18.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:32:18.384 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:32:18.384 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:18.384 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:18.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:18.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:32:18.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:32:18.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:32:18.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:32:18.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:32:18.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:32:18.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:32:18.617 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:32:18.618 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:32:18.618 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:32:18.618 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:32:18.618 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:18.619 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:18.620 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90694/ceph-mon.a.asok 2020-11-16T20:32:18.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:32:18.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.90694/ceph-mon.a.asok config get fsid 2020-11-16T20:32:18.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:32:18.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:32:18.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:32:18.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:32:18.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:32:18.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:32:18.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:32:18.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:32:18.731 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:32:18.731 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:18.731 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:18.732 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90694/ceph-mon.a.asok 2020-11-16T20:32:18.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:32:18.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.90694/ceph-mon.a.asok config get mon_host 2020-11-16T20:32:18.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-11-16T20:32:18.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:18.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:32:18.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:32:18.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:32:18.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-11-16T20:32:18.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:32:19.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:32:19.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:32:19.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:32:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:32:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:19.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:19.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:32:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:32:19.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-11-16T20:32:19.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-11-16T20:32:19.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-11-16T20:32:19.786 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-11-16T20:32:19.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-11-16T20:32:19.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:32:19.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-11-16T20:32:19.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:19.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:32:19.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:32:19.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:32:19.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-11-16T20:32:19.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:32:19.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:32:19.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:32:19.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:32:19.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-11-16T20:32:19.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-11-16T20:32:19.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:32:19.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-11-16T20:32:19.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:32:19.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:32:19.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:32:19.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:32:19.793 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:32:19.794 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:19.794 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:19.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:19.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:19.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:32:19.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:32:19.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:32:19.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:32:19.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:32:19.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:32:19.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:32:19.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:32:19.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:32:19.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:32:19.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-11-16T20:32:19.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:32:19.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=40a42080-572e-4cb6-af60-63ad226ab0e4 2020-11-16T20:32:19.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 40a42080-572e-4cb6-af60-63ad226ab0e4' 2020-11-16T20:32:19.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:32:19.797 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 40a42080-572e-4cb6-af60-63ad226ab0e4 2020-11-16T20:32:19.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDT4bJfCNpcMBAAvMVouefneO9Pr6dkNt45Mw== 2020-11-16T20:32:19.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDT4bJfCNpcMBAAvMVouefneO9Pr6dkNt45Mw=="}' 2020-11-16T20:32:19.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 40a42080-572e-4cb6-af60-63ad226ab0e4 -i td/osd-backfill-recovery-log/0/new.json 2020-11-16T20:32:20.139 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:32:20.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-11-16T20:32:20.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDT4bJfCNpcMBAAvMVouefneO9Pr6dkNt45Mw== --osd-uuid 40a42080-572e-4cb6-af60-63ad226ab0e4 2020-11-16T20:32:20.185 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:20.183+0000 7f01e5e00d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:20.187 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:20.187+0000 7f01e5e00d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:20.189 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:20.187+0000 7f01e5e00d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:20.189 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:20.187+0000 7f01e5e00d80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:32:20.190 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:20.187+0000 7f01e5e00d80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:32:20.190 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:20.187+0000 7f01e5e00d80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:32:24.338 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:20.251+0000 7f01e5e00d80 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-11-16T20:32:24.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-11-16T20:32:24.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:32:24.634 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:32:24.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:32:24.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:32:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:32:25.120 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:32:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:32:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:32:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:32:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:32:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:25.147+0000 7f860bcb2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:25.152 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:25.151+0000 7f860bcb2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:25.154 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:25.151+0000 7f860bcb2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:25.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:32:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:32:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:32:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:32:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:32:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:32:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:32:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:32:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:32:25.535 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:32:25.693 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:25.691+0000 7f860bcb2d80 -1 Falling back to public interface 2020-11-16T20:32:25.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:32:26.533 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:26.531+0000 7f860bcb2d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:32:26.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:32:26.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:32:26.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:32:26.897 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:32:26.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:32:26.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:32:27.314 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/116882,v1:127.0.0.1:6803/116882] [v2:127.0.0.1:6804/116882,v1:127.0.0.1:6805/116882] exists,up 40a42080-572e-4cb6-af60-63ad226ab0e4 2020-11-16T20:32:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:32:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:32:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:32:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:32:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-11-16T20:32:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:32:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:32:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:32:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-11-16T20:32:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:32:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:32:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:32:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:32:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-11-16T20:32:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-11-16T20:32:27.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:32:27.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-11-16T20:32:27.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:32:27.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:32:27.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:32:27.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:32:27.319 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:32:27.319 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:27.319 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:27.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:27.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:32:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:32:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:32:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:32:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:32:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:32:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:32:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:32:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:32:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:32:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-11-16T20:32:27.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:32:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=41b2b906-2ada-4abd-a810-77280c4bb64b 2020-11-16T20:32:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 41b2b906-2ada-4abd-a810-77280c4bb64b' 2020-11-16T20:32:27.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:32:27.322 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 41b2b906-2ada-4abd-a810-77280c4bb64b 2020-11-16T20:32:27.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDb4bJf85T8ExAAu8MSieJ/MHA5zQ1d+qrWFw== 2020-11-16T20:32:27.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDb4bJf85T8ExAAu8MSieJ/MHA5zQ1d+qrWFw=="}' 2020-11-16T20:32:27.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 41b2b906-2ada-4abd-a810-77280c4bb64b -i td/osd-backfill-recovery-log/1/new.json 2020-11-16T20:32:27.950 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:32:27.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-11-16T20:32:27.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDb4bJf85T8ExAAu8MSieJ/MHA5zQ1d+qrWFw== --osd-uuid 41b2b906-2ada-4abd-a810-77280c4bb64b 2020-11-16T20:32:28.001 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:27.999+0000 7fe734545d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:28.004 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:28.003+0000 7fe734545d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:28.005 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:28.003+0000 7fe734545d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:28.006 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:28.003+0000 7fe734545d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:32:28.006 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:28.003+0000 7fe734545d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:32:28.006 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:28.003+0000 7fe734545d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:32:28.080 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:28.079+0000 7fe734545d80 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-11-16T20:32:32.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-11-16T20:32:32.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:32:32.582 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:32:32.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:32:32.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:32:33.091 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:32:33.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:32:33.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:32:33.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:32:33.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:32:33.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:32:33.118 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:33.115+0000 7f1aaedf9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:33.121 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:33.119+0000 7f1aaedf9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:33.122 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:33.119+0000 7f1aaedf9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:33.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:32:33.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:32:33.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:32:33.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:32:33.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:32:33.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:32:33.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:32:33.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:32:33.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:32:33.464 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:32:33.669 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:33.667+0000 7f1aaedf9d80 -1 Falling back to public interface 2020-11-16T20:32:33.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:32:34.500 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:34.499+0000 7f1aaedf9d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:32:34.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:32:34.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:32:34.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:32:34.865 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:32:34.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:32:34.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:32:35.314 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/117876,v1:127.0.0.1:6811/117876] [v2:127.0.0.1:6812/117876,v1:127.0.0.1:6813/117876] exists,up 41b2b906-2ada-4abd-a810-77280c4bb64b 2020-11-16T20:32:35.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:32:35.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:32:35.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:32:35.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:32:35.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-11-16T20:32:35.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:35.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:32:35.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:32:35.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:32:35.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-11-16T20:32:35.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:32:35.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:32:35.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:32:35.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:32:35.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-11-16T20:32:35.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-11-16T20:32:35.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:32:35.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-11-16T20:32:35.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:32:35.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:32:35.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:32:35.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:32:35.320 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:32:35.321 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:35.321 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:35.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:35.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:35.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:32:35.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:32:35.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:32:35.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:32:35.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:32:35.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:32:35.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:32:35.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:32:35.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:32:35.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:32:35.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-11-16T20:32:35.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:32:35.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ef2d4740-dacd-4c69-9080-4ef53a155aff 2020-11-16T20:32:35.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 ef2d4740-dacd-4c69-9080-4ef53a155aff' 2020-11-16T20:32:35.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:32:35.324 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 ef2d4740-dacd-4c69-9080-4ef53a155aff 2020-11-16T20:32:35.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDj4bJfgFNQFBAA0byAaohayPJf3OL+XWCD4w== 2020-11-16T20:32:35.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDj4bJfgFNQFBAA0byAaohayPJf3OL+XWCD4w=="}' 2020-11-16T20:32:35.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ef2d4740-dacd-4c69-9080-4ef53a155aff -i td/osd-backfill-recovery-log/2/new.json 2020-11-16T20:32:35.911 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:32:35.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-11-16T20:32:35.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDj4bJfgFNQFBAA0byAaohayPJf3OL+XWCD4w== --osd-uuid ef2d4740-dacd-4c69-9080-4ef53a155aff 2020-11-16T20:32:35.959 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:35.959+0000 7fae5e15cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:35.961 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:35.959+0000 7fae5e15cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:35.963 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:35.963+0000 7fae5e15cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:35.963 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:35.963+0000 7fae5e15cd80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:32:35.964 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:35.963+0000 7fae5e15cd80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:32:35.964 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:35.963+0000 7fae5e15cd80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:32:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:36.023+0000 7fae5e15cd80 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-11-16T20:32:40.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-11-16T20:32:40.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:32:40.477 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:32:40.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:32:40.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:32:40.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:32:40.979 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:32:40.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:32:40.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:32:40.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:32:40.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:32:41.008 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:41.007+0000 7f65e5517d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:41.010 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:41.007+0000 7f65e5517d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:41.011 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:41.011+0000 7f65e5517d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:41.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:32:41.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:32:41.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:32:41.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:32:41.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:32:41.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:32:41.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:32:41.366 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:32:41.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:32:41.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:32:41.549 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:41.547+0000 7f65e5517d80 -1 Falling back to public interface 2020-11-16T20:32:41.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:32:42.436 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:42.427+0000 7f65e5517d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:32:42.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:32:42.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:32:42.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:32:42.753 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:32:42.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:32:42.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:32:43.176 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/118870,v1:127.0.0.1:6819/118870] [v2:127.0.0.1:6820/118870,v1:127.0.0.1:6821/118870] exists,up ef2d4740-dacd-4c69-9080-4ef53a155aff 2020-11-16T20:32:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:32:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:32:43.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:32:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:32:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-11-16T20:32:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:32:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T20:32:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:32:43.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-11-16T20:32:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:32:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:32:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:32:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:32:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-11-16T20:32:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-11-16T20:32:43.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:32:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-11-16T20:32:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:32:43.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:32:43.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:32:43.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:32:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:32:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:43.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:32:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:32:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:32:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:32:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:32:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:32:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:32:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:32:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:32:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:32:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-11-16T20:32:43.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:32:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9b5c9dcc-e8fd-4549-86b2-9152276c9203 2020-11-16T20:32:43.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 9b5c9dcc-e8fd-4549-86b2-9152276c9203' 2020-11-16T20:32:43.185 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:32:43.185 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 9b5c9dcc-e8fd-4549-86b2-9152276c9203 2020-11-16T20:32:43.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDr4bJfrHD8CxAA7PfR4Ymz5+IAkBREWx/FEw== 2020-11-16T20:32:43.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDr4bJfrHD8CxAA7PfR4Ymz5+IAkBREWx/FEw=="}' 2020-11-16T20:32:43.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9b5c9dcc-e8fd-4549-86b2-9152276c9203 -i td/osd-backfill-recovery-log/3/new.json 2020-11-16T20:32:43.828 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T20:32:43.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-11-16T20:32:43.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDr4bJfrHD8CxAA7PfR4Ymz5+IAkBREWx/FEw== --osd-uuid 9b5c9dcc-e8fd-4549-86b2-9152276c9203 2020-11-16T20:32:43.879 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:43.879+0000 7f1c9495dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:43.883 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:43.883+0000 7f1c9495dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:43.884 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:43.883+0000 7f1c9495dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:43.883+0000 7f1c9495dd80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:32:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:43.883+0000 7f1c9495dd80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:32:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:43.883+0000 7f1c9495dd80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:32:43.937 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:43.935+0000 7f1c9495dd80 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-11-16T20:32:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-11-16T20:32:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:32:48.633 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T20:32:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T20:32:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:32:49.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T20:32:49.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:32:49.133 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T20:32:49.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:32:49.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:32:53.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:32:53.336 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:49.155+0000 7faeb08ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:53.336 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:49.159+0000 7faeb08ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:53.336 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:49.159+0000 7faeb08ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:53.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T20:32:53.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:32:53.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T20:32:53.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:32:53.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:32:53.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:32:53.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:32:53.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:32:53.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:32:53.338 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:49.715+0000 7faeb08ccd80 -1 Falling back to public interface 2020-11-16T20:32:53.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:32:53.338 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:50.587+0000 7faeb08ccd80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T20:32:53.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:32:53.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:32:53.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:32:53.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:32:53.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:32:53.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:32:53.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:32:53.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:32:53.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:32:53.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-11-16T20:32:53.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:32:53.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:32:53.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T20:32:53.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:32:53.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-11-16T20:32:53.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:32:53.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:32:53.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:32:53.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:32:53.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-11-16T20:32:53.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-11-16T20:32:53.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:32:53.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-11-16T20:32:53.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:32:53.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:32:53.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:32:53.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:32:53.342 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:32:53.342 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:32:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:32:53.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:32:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:32:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:32:53.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:32:53.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:32:53.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:32:53.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:32:53.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:32:53.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:32:53.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:32:53.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:32:53.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-11-16T20:32:53.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:32:53.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=89bc0690-02cd-428f-bcd1-3a0fe04fdf1e 2020-11-16T20:32:53.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 89bc0690-02cd-428f-bcd1-3a0fe04fdf1e' 2020-11-16T20:32:53.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:32:53.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDz4bJfjblPExAAvbeUwcdxoRNSFjZMbTqOzw== 2020-11-16T20:32:53.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDz4bJfjblPExAAvbeUwcdxoRNSFjZMbTqOzw=="}' 2020-11-16T20:32:53.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 89bc0690-02cd-428f-bcd1-3a0fe04fdf1e -i td/osd-backfill-recovery-log/4/new.json 2020-11-16T20:32:53.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-11-16T20:32:53.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDz4bJfjblPExAAvbeUwcdxoRNSFjZMbTqOzw== --osd-uuid 89bc0690-02cd-428f-bcd1-3a0fe04fdf1e 2020-11-16T20:32:53.346 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:51.935+0000 7f44c98d9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:53.346 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:51.939+0000 7f44c98d9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:53.346 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:51.939+0000 7f44c98d9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:53.347 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:51.943+0000 7f44c98d9d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:32:53.347 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:51.943+0000 7f44c98d9d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:32:53.347 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:51.943+0000 7f44c98d9d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:32:53.347 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:51.991+0000 7f44c98d9d80 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-11-16T20:32:53.347 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:32:53.347 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:32:53.347 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/119864,v1:127.0.0.1:6827/119864] [v2:127.0.0.1:6828/119864,v1:127.0.0.1:6829/119864] exists,up 9b5c9dcc-e8fd-4549-86b2-9152276c9203 2020-11-16T20:32:53.348 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 89bc0690-02cd-428f-bcd1-3a0fe04fdf1e 2020-11-16T20:32:53.348 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T20:32:57.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-11-16T20:32:57.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:32:57.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T20:32:57.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:32:57.022 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T20:32:57.561 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T20:32:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T20:32:57.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:32:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:32:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:32:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:32:57.590 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:57.587+0000 7f2976c32d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:57.609 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:57.607+0000 7f2976c32d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:57.610 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:57.607+0000 7f2976c32d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:32:57.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T20:32:57.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:32:57.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T20:32:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:32:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:32:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:32:57.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:32:57.944 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:32:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:32:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:32:58.148 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:58.147+0000 7f2976c32d80 -1 Falling back to public interface 2020-11-16T20:32:58.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:32:59.059 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:32:59.059+0000 7f2976c32d80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T20:32:59.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:32:59.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:32:59.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:32:59.331 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:32:59.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:32:59.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:32:59.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:33:00.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:33:00.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:33:00.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:33:00.794 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:33:00.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:33:00.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:33:01.187 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/120858,v1:127.0.0.1:6835/120858] [v2:127.0.0.1:6836/120858,v1:127.0.0.1:6837/120858] exists,up 89bc0690-02cd-428f-bcd1-3a0fe04fdf1e 2020-11-16T20:33:01.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:33:01.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:33:01.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:33:01.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:33:01.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-11-16T20:33:01.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:33:01.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:33:01.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T20:33:01.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:33:01.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-11-16T20:33:01.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:33:01.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:33:01.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:33:01.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:33:01.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-11-16T20:33:01.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-11-16T20:33:01.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:33:01.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-11-16T20:33:01.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:33:01.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:33:01.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:33:01.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:33:01.193 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:33:01.193 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:33:01.193 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:33:01.193 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:33:01.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:33:01.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:33:01.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:33:01.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:33:01.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:33:01.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:33:01.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:33:01.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:33:01.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:33:01.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:33:01.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:33:01.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-11-16T20:33:01.196 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:33:01.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2eb9a00f-5333-47c4-8b6d-bdbde12d3b5e 2020-11-16T20:33:01.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 2eb9a00f-5333-47c4-8b6d-bdbde12d3b5e' 2020-11-16T20:33:01.196 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 2eb9a00f-5333-47c4-8b6d-bdbde12d3b5e 2020-11-16T20:33:01.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:33:01.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD94bJfj/q2DBAAqdSRJqLkKviiOD6VUKwi5Q== 2020-11-16T20:33:01.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD94bJfj/q2DBAAqdSRJqLkKviiOD6VUKwi5Q=="}' 2020-11-16T20:33:01.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2eb9a00f-5333-47c4-8b6d-bdbde12d3b5e -i td/osd-backfill-recovery-log/5/new.json 2020-11-16T20:33:01.793 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T20:33:01.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-11-16T20:33:01.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD94bJfj/q2DBAAqdSRJqLkKviiOD6VUKwi5Q== --osd-uuid 2eb9a00f-5333-47c4-8b6d-bdbde12d3b5e 2020-11-16T20:33:01.832 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:01.831+0000 7fe2d5d01d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:01.835 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:01.835+0000 7fe2d5d01d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:01.837 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:01.835+0000 7fe2d5d01d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:01.837 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:01.835+0000 7fe2d5d01d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:33:01.837 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:01.835+0000 7fe2d5d01d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:33:01.838 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:01.835+0000 7fe2d5d01d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:33:01.888 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:01.887+0000 7fe2d5d01d80 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-11-16T20:33:06.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-11-16T20:33:06.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:33:06.750 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T20:33:06.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T20:33:06.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:33:07.238 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T20:33:07.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T20:33:07.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:33:07.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:33:07.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:33:07.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:33:07.269 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:07.266+0000 7fd09ce1ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:07.272 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:07.270+0000 7fd09ce1ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:07.273 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:07.270+0000 7fd09ce1ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T20:33:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:33:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T20:33:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:33:07.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:33:07.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:33:07.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:33:07.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:33:07.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T20:33:07.659 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:33:07.816 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:07.814+0000 7fd09ce1ad80 -1 Falling back to public interface 2020-11-16T20:33:08.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:33:08.717 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:08.714+0000 7fd09ce1ad80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T20:33:09.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:33:09.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:33:09.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:33:09.075 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:33:09.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:33:09.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T20:33:09.521 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/121878,v1:127.0.0.1:6843/121878] [v2:127.0.0.1:6844/121878,v1:127.0.0.1:6845/121878] exists,up 2eb9a00f-5333-47c4-8b6d-bdbde12d3b5e 2020-11-16T20:33:09.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:33:09.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:33:09.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:33:09.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-11-16T20:33:09.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T20:33:10.281 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T20:33:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:33:11.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 40 2020-11-16T20:33:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-11-16T20:33:11.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:11.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-11-16T20:33:12.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:12.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-11-16T20:33:12.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:12.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-11-16T20:33:12.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:12.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-11-16T20:33:12.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:12.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-11-16T20:33:13.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:13.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-11-16T20:33:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:13.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-11-16T20:33:13.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:13.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-11-16T20:33:13.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:13.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-11-16T20:33:13.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:13.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-11-16T20:33:13.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:13.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-11-16T20:33:14.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:14.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-11-16T20:33:14.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:14.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-11-16T20:33:14.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:14.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-11-16T20:33:14.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:14.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-11-16T20:33:14.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:14.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-11-16T20:33:14.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:14.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-11-16T20:33:15.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:15.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-11-16T20:33:15.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:15.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-11-16T20:33:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:15.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-11-16T20:33:15.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:15.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-11-16T20:33:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-11-16T20:33:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-11-16T20:33:16.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:16.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-11-16T20:33:16.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:16.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-11-16T20:33:16.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:16.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-11-16T20:33:16.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:16.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-11-16T20:33:16.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:16.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-11-16T20:33:17.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:17.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-11-16T20:33:17.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:17.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-11-16T20:33:17.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:17.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-11-16T20:33:17.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-11-16T20:33:17.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:17.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-11-16T20:33:17.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:17.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-11-16T20:33:18.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:18.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-11-16T20:33:18.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:18.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-11-16T20:33:18.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:18.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-11-16T20:33:18.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:18.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-11-16T20:33:18.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:33:18.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-11-16T20:33:18.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-11-16T20:33:18.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-11-16T20:33:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:33:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-11-16T20:33:19.946 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-11-16T20:33:19.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-11-16T20:33:19.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-11-16T20:33:20.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-11-16T20:33:20.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:33:20.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:33:20.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:33:20.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:33:20.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:33:20.960 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:33:20.960 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:33:20.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:33:20.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:33:20.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:33:21.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:33:21.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:33:21.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:33:21.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:33:21.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:33:21.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:33:21.606 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:33:21.606 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:33:21.606 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:33:21.606 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T20:33:21.607 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T20:33:21.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:33:21.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:21.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:33:21.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836493 2020-11-16T20:33:21.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836493 2020-11-16T20:33:21.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493' 2020-11-16T20:33:21.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:21.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:33:21.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672972 2020-11-16T20:33:21.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672972 2020-11-16T20:33:21.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-11-16T20:33:21.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:21.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:33:22.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509451 2020-11-16T20:33:22.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509451 2020-11-16T20:33:22.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451' 2020-11-16T20:33:22.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:22.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:33:22.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378633 2020-11-16T20:33:22.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378633 2020-11-16T20:33:22.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451 3-81604378633' 2020-11-16T20:33:22.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:22.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:33:22.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247815 2020-11-16T20:33:22.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247815 2020-11-16T20:33:22.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451 3-81604378633 4-98784247815' 2020-11-16T20:33:22.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:22.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T20:33:22.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084293 2020-11-16T20:33:22.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084293 2020-11-16T20:33:22.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509451 3-81604378633 4-98784247815 5-120259084293' 2020-11-16T20:33:22.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:22.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836493 2020-11-16T20:33:22.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:22.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:33:22.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:22.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836493 2020-11-16T20:33:22.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836493 2020-11-16T20:33:22.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-11-16T20:33:22.599 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836493 2020-11-16T20:33:22.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:33:22.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 0 -lt 21474836493 2020-11-16T20:33:22.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:33:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:33:23.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:33:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836494 -lt 21474836493 2020-11-16T20:33:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:24.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672972 2020-11-16T20:33:24.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:33:24.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:24.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672972 2020-11-16T20:33:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672972 2020-11-16T20:33:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-11-16T20:33:24.402 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672972 2020-11-16T20:33:24.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:33:24.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672972 -lt 42949672972 2020-11-16T20:33:24.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:24.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509451 2020-11-16T20:33:24.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:24.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:33:24.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509451 2020-11-16T20:33:24.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:24.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509451 2020-11-16T20:33:24.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-11-16T20:33:24.788 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509451 2020-11-16T20:33:24.789 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:33:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509451 -lt 64424509451 2020-11-16T20:33:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:25.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378633 2020-11-16T20:33:25.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:33:25.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378633 2020-11-16T20:33:25.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378633 2020-11-16T20:33:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378633' 2020-11-16T20:33:25.165 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378633 2020-11-16T20:33:25.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:33:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378633 -lt 81604378633 2020-11-16T20:33:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:25.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247815 2020-11-16T20:33:25.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:33:25.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247815 2020-11-16T20:33:25.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:25.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247815 2020-11-16T20:33:25.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247815' 2020-11-16T20:33:25.559 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247815 2020-11-16T20:33:25.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:33:25.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247815 -lt 98784247815 2020-11-16T20:33:25.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:25.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084293 2020-11-16T20:33:25.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:25.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T20:33:25.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084293 2020-11-16T20:33:25.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:25.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084293 2020-11-16T20:33:25.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084293' 2020-11-16T20:33:25.954 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 120259084293 2020-11-16T20:33:25.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T20:33:26.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084294 -lt 120259084293 2020-11-16T20:33:26.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:33:26.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:33:26.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:33:26.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:33:26.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:33:26.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:33:26.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:33:26.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:33:26.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:33:26.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:33:26.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:33:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:33:27.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:33:27.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:33:27.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:33:27.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:33:27.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:33:27.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:33:27.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:33:27.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:33:27.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:33:27.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:33:27.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:33:27.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:33:27.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:33:27.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:33:27.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:33:27.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:33:27.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:33:28.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:33:28.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:33:28.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:33:28.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:33:28.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:33:28.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:33:28.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:33:28.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:33:28.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:33:28.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:33:28.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:33:28.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:33:28.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:33:29.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14835 2020-11-16T20:33:29.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14835 '!=' null 2020-11-16T20:33:29.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:33:29.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:33:29.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:33:29.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:33:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:33:29.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:33:29.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:33:29.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:33:29.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:33:29.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:33:29.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:33:29.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:33:29.708 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:33:29.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:33:29.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:33:30.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:33:30.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:33:30.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:33:30.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:33:30.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:33:30.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:33:30.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:33:30.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:33:30.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:33:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12365 2020-11-16T20:33:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12365 '!=' null 2020-11-16T20:33:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:33:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:33:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:33:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:33:30.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:33:30.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:33:30.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:33:30.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:33:30.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:33:30.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:33:30.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:33:31.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:33:31.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:33:31.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:33:31.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:33:31.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:33:31.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:33:31.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:33:31.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:33:31.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:33:31.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:33:31.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:33:31.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:33:31.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:33:32.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23344 2020-11-16T20:33:32.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23344 '!=' null 2020-11-16T20:33:32.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:33:32.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:33:32.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:33:32.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:33:32.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:33:32.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:33:32.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:33:32.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:33:32.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:33:32.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:33:32.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:33:32.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:33:32.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:33:32.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:33:32.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:33:33.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:33:33.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:33:33.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:33:33.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:33:33.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:33:33.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:33:33.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:33:33.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:33:33.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:33:33.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21759 2020-11-16T20:33:33.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21759 '!=' null 2020-11-16T20:33:33.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:33:33.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:33:33.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:33:33.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:33:33.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:33:33.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:33:33.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:33:33.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:33:33.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:33:33.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:33:33.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:33:34.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:33:34.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:33:34.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:33:34.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:33:34.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:33:34.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:33:34.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:33:34.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:33:34.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:33:34.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:33:34.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:33:34.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:33:34.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:33:35.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21759 2020-11-16T20:33:35.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21759 '!=' null 2020-11-16T20:33:35.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:33:35.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:33:35.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:33:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:33:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:33:35.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:33:35.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:33:35.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:33:35.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:33:35.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:33:35.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:33:35.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:33:35.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:33:35.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:33:35.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:33:36.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:33:36.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:33:36.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:33:36.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:33:36.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:33:36.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:33:36.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:33:36.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:33:36.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:33:36.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19121 2020-11-16T20:33:36.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19121 '!=' null 2020-11-16T20:33:36.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:33:36.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:33:36.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:33:36.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:33:36.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:33:36.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:33:36.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:33:36.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:33:36.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:33:36.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:33:36.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:33:37.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:33:37.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:33:37.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:33:37.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:33:37.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:33:37.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:33:37.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:33:37.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:33:37.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:33:37.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:33:37.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:33:37.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:33:37.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:33:38.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23737 2020-11-16T20:33:38.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23737 '!=' null 2020-11-16T20:33:38.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:33:38.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:33:38.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:33:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:33:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:33:38.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:33:38.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:33:38.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:33:38.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:33:38.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:33:38.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:33:38.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:33:38.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:33:38.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:33:38.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:33:39.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:33:39.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:33:39.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:33:39.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2020-11-16T20:33:39.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:33:39.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:33:39.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:33:39.477 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:33:39.477 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:33:39.477 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:33:39.477 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T20:33:39.478 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T20:33:39.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:33:39.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:39.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:33:39.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836499 2020-11-16T20:33:39.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836499 2020-11-16T20:33:39.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499' 2020-11-16T20:33:39.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:39.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:33:39.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672977 2020-11-16T20:33:39.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672977 2020-11-16T20:33:39.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672977' 2020-11-16T20:33:39.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:39.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:33:40.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509457 2020-11-16T20:33:40.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509457 2020-11-16T20:33:40.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509457' 2020-11-16T20:33:40.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:40.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:33:40.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378638 2020-11-16T20:33:40.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378638 2020-11-16T20:33:40.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509457 3-81604378638' 2020-11-16T20:33:40.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:40.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:33:40.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247820 2020-11-16T20:33:40.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247820 2020-11-16T20:33:40.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509457 3-81604378638 4-98784247820' 2020-11-16T20:33:40.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:33:40.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T20:33:40.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084299 2020-11-16T20:33:40.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084299 2020-11-16T20:33:40.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509457 3-81604378638 4-98784247820 5-120259084299' 2020-11-16T20:33:40.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:40.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836499 2020-11-16T20:33:40.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:40.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:33:40.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836499 2020-11-16T20:33:40.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:40.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836499 2020-11-16T20:33:40.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2020-11-16T20:33:40.501 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836499 2020-11-16T20:33:40.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:33:40.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836497 -lt 21474836499 2020-11-16T20:33:40.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:33:41.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:33:41.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:33:42.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836499 -lt 21474836499 2020-11-16T20:33:42.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:42.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672977 2020-11-16T20:33:42.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:42.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:33:42.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672977 2020-11-16T20:33:42.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:42.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672977 2020-11-16T20:33:42.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2020-11-16T20:33:42.295 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672977 2020-11-16T20:33:42.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:33:42.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672978 -lt 42949672977 2020-11-16T20:33:42.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:42.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509457 2020-11-16T20:33:42.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:42.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:33:42.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:42.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509457 2020-11-16T20:33:42.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509457 2020-11-16T20:33:42.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2020-11-16T20:33:42.685 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509457 2020-11-16T20:33:42.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:33:43.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509457 -lt 64424509457 2020-11-16T20:33:43.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:43.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378638 2020-11-16T20:33:43.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:43.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:33:43.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:43.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378638 2020-11-16T20:33:43.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378638 2020-11-16T20:33:43.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378638' 2020-11-16T20:33:43.113 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378638 2020-11-16T20:33:43.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:33:43.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378638 -lt 81604378638 2020-11-16T20:33:43.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:43.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247820 2020-11-16T20:33:43.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:43.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:33:43.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247820 2020-11-16T20:33:43.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:43.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247820 2020-11-16T20:33:43.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247820' 2020-11-16T20:33:43.500 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247820 2020-11-16T20:33:43.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:33:43.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247821 -lt 98784247820 2020-11-16T20:33:43.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:33:43.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084299 2020-11-16T20:33:43.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:33:43.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T20:33:43.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:33:43.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084299 2020-11-16T20:33:43.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084299 2020-11-16T20:33:43.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084299' 2020-11-16T20:33:43.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T20:33:43.879 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 120259084299 2020-11-16T20:33:44.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084299 -lt 120259084299 2020-11-16T20:33:44.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2020-11-16T20:33:44.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2020-11-16T20:33:44.618 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:33:44.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2020-11-16T20:33:44.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2020-11-16T20:33:44.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:33:44.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:33:44.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:33:44.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:33:44.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:33:44.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:33:44.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2020-11-16T20:33:44.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-11-16T20:33:44.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:33:44.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-11-16T20:33:44.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T20:33:44.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=4 2020-11-16T20:33:44.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T20:33:44.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-11-16T20:33:44.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-11-16T20:33:45.732 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:33:45.733 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_log_t": { 2020-11-16T20:33:45.733 INFO:tasks.workunit.client.0.smithi189.stdout: "head": "32'40", 2020-11-16T20:33:45.733 INFO:tasks.workunit.client.0.smithi189.stdout: "tail": "0'0", 2020-11-16T20:33:45.733 INFO:tasks.workunit.client.0.smithi189.stdout: "log": [ 2020-11-16T20:33:45.733 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.733 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.734 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:c00128d9:::obj-1:head", 2020-11-16T20:33:45.734 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "31'1", 2020-11-16T20:33:45.734 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.734 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4189.0:1", 2020-11-16T20:33:45.734 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.734 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:11.330378+0000", 2020-11-16T20:33:45.735 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.735 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.735 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.735 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.735 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.735 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.735 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.735 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.736 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.736 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.736 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.736 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.736 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.736 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.736 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.737 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.737 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.737 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.737 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:d7942fc0:::obj-2:head", 2020-11-16T20:33:45.737 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'2", 2020-11-16T20:33:45.737 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.737 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4192.0:1", 2020-11-16T20:33:45.738 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.738 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:12.001717+0000", 2020-11-16T20:33:45.738 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.738 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.738 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.738 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.738 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.738 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.739 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.739 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.739 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.739 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.739 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.739 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.739 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.740 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.740 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.740 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.740 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.740 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.740 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1dda57ab:::obj-3:head", 2020-11-16T20:33:45.740 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'3", 2020-11-16T20:33:45.740 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.741 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4195.0:1", 2020-11-16T20:33:45.741 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.741 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:12.338369+0000", 2020-11-16T20:33:45.741 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.741 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.741 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.741 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.742 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.742 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.742 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.742 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.742 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.742 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.742 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.743 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.743 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.743 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.743 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.743 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.743 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.743 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.743 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:ba1802d0:::obj-4:head", 2020-11-16T20:33:45.744 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'4", 2020-11-16T20:33:45.744 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.744 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4198.0:1", 2020-11-16T20:33:45.744 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.744 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:12.513954+0000", 2020-11-16T20:33:45.744 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.744 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.745 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.745 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.745 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.745 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.745 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.745 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.745 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.745 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.746 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.746 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.746 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.746 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.746 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.746 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.746 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.747 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.747 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:d5d4a582:::obj-5:head", 2020-11-16T20:33:45.747 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'5", 2020-11-16T20:33:45.747 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.747 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4201.0:1", 2020-11-16T20:33:45.747 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.747 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:12.715057+0000", 2020-11-16T20:33:45.747 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.748 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.748 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.748 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.748 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.748 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.748 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.748 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.749 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.749 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.749 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.749 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.749 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.749 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.749 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.749 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.750 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.750 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.750 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:25cac09a:::obj-6:head", 2020-11-16T20:33:45.750 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'6", 2020-11-16T20:33:45.750 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.750 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4204.0:1", 2020-11-16T20:33:45.750 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.751 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:12.890078+0000", 2020-11-16T20:33:45.751 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.751 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.751 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.751 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.751 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.751 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.752 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.752 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.752 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.752 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.752 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.752 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.752 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.752 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.753 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.753 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.753 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.753 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.753 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:b11b7838:::obj-7:head", 2020-11-16T20:33:45.753 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'7", 2020-11-16T20:33:45.753 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.754 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4207.0:1", 2020-11-16T20:33:45.754 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.754 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:13.042795+0000", 2020-11-16T20:33:45.754 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.754 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.754 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.754 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.754 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.755 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.755 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.755 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.755 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.755 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.755 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.755 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.756 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.756 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.756 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.756 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.756 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.756 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.756 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:a0f702de:::obj-8:head", 2020-11-16T20:33:45.756 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'8", 2020-11-16T20:33:45.757 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.757 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4210.0:1", 2020-11-16T20:33:45.757 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.757 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:13.200682+0000", 2020-11-16T20:33:45.757 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.757 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.757 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.757 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.758 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.758 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.758 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.758 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.758 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.758 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.758 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.759 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.759 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.759 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.759 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.759 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.759 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.759 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.759 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1e5c3677:::obj-9:head", 2020-11-16T20:33:45.760 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'9", 2020-11-16T20:33:45.760 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.760 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4213.0:1", 2020-11-16T20:33:45.760 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.760 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:13.417469+0000", 2020-11-16T20:33:45.760 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.760 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.761 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.761 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.761 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.761 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.761 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.761 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.761 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.761 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.762 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.762 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.762 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.762 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.762 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.762 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.762 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.763 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.763 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:86dc045c:::obj-10:head", 2020-11-16T20:33:45.763 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'10", 2020-11-16T20:33:45.763 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.763 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4216.0:1", 2020-11-16T20:33:45.763 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.763 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:13.588257+0000", 2020-11-16T20:33:45.763 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.764 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.764 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.764 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.764 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.764 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.764 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.764 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.765 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.765 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.765 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.765 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.765 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.765 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.765 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.765 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.766 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.766 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.766 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1e6eca61:::obj-11:head", 2020-11-16T20:33:45.766 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'11", 2020-11-16T20:33:45.766 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.766 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4219.0:1", 2020-11-16T20:33:45.766 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.767 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:13.743382+0000", 2020-11-16T20:33:45.767 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.767 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.767 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.767 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.767 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.767 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.767 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.768 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.768 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.768 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.768 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.768 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.768 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.768 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.769 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.769 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.769 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.769 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.769 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:18dee445:::obj-12:head", 2020-11-16T20:33:45.769 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'12", 2020-11-16T20:33:45.769 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.769 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4222.0:1", 2020-11-16T20:33:45.770 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.770 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:13.968237+0000", 2020-11-16T20:33:45.770 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.770 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.770 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.770 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.770 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.770 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.771 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.771 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.771 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.771 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.771 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.771 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.771 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.772 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.772 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.772 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.772 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.772 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.772 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:283c5c90:::obj-13:head", 2020-11-16T20:33:45.772 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'13", 2020-11-16T20:33:45.772 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.773 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4225.0:1", 2020-11-16T20:33:45.773 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.773 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:14.127337+0000", 2020-11-16T20:33:45.773 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.773 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.773 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.773 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.774 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.774 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.774 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.774 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.774 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.774 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.774 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.774 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.775 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.775 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.775 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.775 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.775 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.775 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.775 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:cc896bf0:::obj-14:head", 2020-11-16T20:33:45.776 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'14", 2020-11-16T20:33:45.776 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.776 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4228.0:1", 2020-11-16T20:33:45.776 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.776 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:14.288692+0000", 2020-11-16T20:33:45.776 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.776 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.776 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.777 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.777 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.777 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.777 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.777 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.777 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.777 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.777 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.778 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.778 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.778 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.778 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.778 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.778 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.778 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.779 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:a581f807:::obj-15:head", 2020-11-16T20:33:45.779 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'15", 2020-11-16T20:33:45.779 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.779 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4231.0:1", 2020-11-16T20:33:45.779 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.779 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:14.474297+0000", 2020-11-16T20:33:45.779 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.779 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.780 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.780 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.780 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.780 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.780 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.780 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.780 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.781 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.781 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.781 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.781 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.781 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.781 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.781 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.781 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.782 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.782 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:c902ca7c:::obj-16:head", 2020-11-16T20:33:45.782 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'16", 2020-11-16T20:33:45.782 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.782 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4234.0:1", 2020-11-16T20:33:45.782 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.782 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:14.649904+0000", 2020-11-16T20:33:45.782 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.783 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.783 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.783 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.783 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.783 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.783 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.783 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.784 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.784 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.784 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.784 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.784 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.784 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.784 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.784 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.785 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.785 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.785 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:655283c8:::obj-17:head", 2020-11-16T20:33:45.785 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'17", 2020-11-16T20:33:45.785 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.785 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4237.0:1", 2020-11-16T20:33:45.785 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.786 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:14.826210+0000", 2020-11-16T20:33:45.786 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.786 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.786 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.786 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.786 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.786 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.787 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.787 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.787 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.787 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.787 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.787 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.787 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.787 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.788 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.788 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.788 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.788 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.788 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:d759d762:::obj-18:head", 2020-11-16T20:33:45.788 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'18", 2020-11-16T20:33:45.788 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.789 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4240.0:1", 2020-11-16T20:33:45.789 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.789 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:14.983781+0000", 2020-11-16T20:33:45.789 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.789 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.789 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.789 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.789 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.790 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.790 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.790 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.790 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.790 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.790 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.790 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.791 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.791 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.791 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.791 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.791 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.791 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.791 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1b934a2d:::obj-19:head", 2020-11-16T20:33:45.791 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'19", 2020-11-16T20:33:45.792 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.792 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4243.0:1", 2020-11-16T20:33:45.792 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.792 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:15.177405+0000", 2020-11-16T20:33:45.792 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.792 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.792 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.793 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.793 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.793 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.793 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.793 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.793 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.793 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.793 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.794 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.794 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.794 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.794 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.794 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.794 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.794 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.794 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1d0ea2ca:::obj-20:head", 2020-11-16T20:33:45.795 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'20", 2020-11-16T20:33:45.795 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.795 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4246.0:1", 2020-11-16T20:33:45.795 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.795 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:15.330934+0000", 2020-11-16T20:33:45.795 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.795 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.796 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.796 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.796 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.796 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.796 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.796 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.796 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.796 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.797 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.797 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.797 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.797 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.797 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.797 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.797 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.798 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.798 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:ab396690:::obj-21:head", 2020-11-16T20:33:45.798 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'21", 2020-11-16T20:33:45.798 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.798 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4249.0:1", 2020-11-16T20:33:45.798 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.798 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:15.506302+0000", 2020-11-16T20:33:45.798 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.799 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.799 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.799 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.799 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.799 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.799 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.799 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.799 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.800 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.800 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.800 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.800 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.800 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.800 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.800 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.801 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.801 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.801 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:254983db:::obj-22:head", 2020-11-16T20:33:45.801 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'22", 2020-11-16T20:33:45.801 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.801 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4252.0:1", 2020-11-16T20:33:45.801 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.801 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:15.745216+0000", 2020-11-16T20:33:45.802 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.802 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.802 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.802 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.802 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.802 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.802 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.803 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.803 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.803 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.803 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.803 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.803 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.803 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.803 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.804 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.804 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.804 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.804 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:e656299f:::obj-23:head", 2020-11-16T20:33:45.804 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'23", 2020-11-16T20:33:45.804 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.804 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4255.0:1", 2020-11-16T20:33:45.805 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.805 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:15.932845+0000", 2020-11-16T20:33:45.805 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.805 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.805 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.805 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.805 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.805 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.806 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.806 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.806 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.806 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.806 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.806 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.806 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.806 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.807 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.807 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.807 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.807 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.807 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:5cdf412b:::obj-24:head", 2020-11-16T20:33:45.807 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'24", 2020-11-16T20:33:45.807 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.808 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4258.0:1", 2020-11-16T20:33:45.808 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.808 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:16.090370+0000", 2020-11-16T20:33:45.808 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.808 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.808 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.808 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.808 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.809 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.809 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.809 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.809 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.809 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.809 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.809 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.810 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.810 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.810 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.810 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.810 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.810 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.810 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:cacae143:::obj-25:head", 2020-11-16T20:33:45.810 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'25", 2020-11-16T20:33:45.811 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.811 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4261.0:1", 2020-11-16T20:33:45.811 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.811 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:16.305677+0000", 2020-11-16T20:33:45.811 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.811 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.811 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.812 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.812 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.812 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.812 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.812 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.812 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.812 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.812 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.813 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.813 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.813 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.813 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.813 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.813 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.813 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.813 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:34b5bff6:::obj-26:head", 2020-11-16T20:33:45.814 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'26", 2020-11-16T20:33:45.814 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.814 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4264.0:1", 2020-11-16T20:33:45.814 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.814 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:16.454791+0000", 2020-11-16T20:33:45.814 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.814 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.815 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.815 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.815 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.815 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.815 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.815 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.815 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.815 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.816 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.816 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.816 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.816 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.816 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.816 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.816 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.817 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.817 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:a354f8f6:::obj-27:head", 2020-11-16T20:33:45.817 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'27", 2020-11-16T20:33:45.817 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.817 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4267.0:1", 2020-11-16T20:33:45.817 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.817 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:16.598262+0000", 2020-11-16T20:33:45.818 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.818 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.818 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.818 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.818 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.818 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.818 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.818 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.819 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.819 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.819 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.819 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.819 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.819 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.819 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.820 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.820 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.820 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.820 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:7421db49:::obj-28:head", 2020-11-16T20:33:45.820 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'28", 2020-11-16T20:33:45.820 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.820 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4270.0:1", 2020-11-16T20:33:45.820 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.821 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:16.751941+0000", 2020-11-16T20:33:45.821 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.821 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.821 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.821 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.821 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.821 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.822 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.822 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.822 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.822 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.822 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.822 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.822 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.822 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.823 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.823 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.823 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.823 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.823 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:54bc79ba:::obj-29:head", 2020-11-16T20:33:45.823 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'29", 2020-11-16T20:33:45.823 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.823 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4273.0:1", 2020-11-16T20:33:45.824 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.824 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:16.926620+0000", 2020-11-16T20:33:45.824 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.824 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.824 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.824 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.824 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.825 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.825 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.825 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.825 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.825 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.825 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.825 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.825 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.826 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.826 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.826 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.826 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.826 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.826 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:01b15595:::obj-30:head", 2020-11-16T20:33:45.826 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'30", 2020-11-16T20:33:45.827 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.827 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4276.0:1", 2020-11-16T20:33:45.827 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.827 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:17.084832+0000", 2020-11-16T20:33:45.827 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.827 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.827 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.827 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.828 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.828 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.828 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.828 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.828 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.828 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.828 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.828 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.829 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.829 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.829 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.829 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.829 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.829 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.829 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:f9a0a8a2:::obj-31:head", 2020-11-16T20:33:45.830 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'31", 2020-11-16T20:33:45.830 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.830 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4279.0:1", 2020-11-16T20:33:45.830 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.830 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:17.244749+0000", 2020-11-16T20:33:45.830 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.830 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.831 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.831 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.831 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.831 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.831 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.831 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.831 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.831 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.832 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.832 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.832 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.832 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.832 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.832 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.832 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.832 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.833 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1bf2562f:::obj-32:head", 2020-11-16T20:33:45.833 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'32", 2020-11-16T20:33:45.833 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.833 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4282.0:1", 2020-11-16T20:33:45.833 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.833 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:17.436508+0000", 2020-11-16T20:33:45.833 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.834 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.834 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.834 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.834 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.834 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.834 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.834 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.834 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.835 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.835 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.835 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.835 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.835 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.835 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.835 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.836 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.836 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.836 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:b982ae22:::obj-33:head", 2020-11-16T20:33:45.836 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'33", 2020-11-16T20:33:45.836 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.836 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4285.0:1", 2020-11-16T20:33:45.836 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.836 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:17.640352+0000", 2020-11-16T20:33:45.837 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.837 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.837 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.837 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.837 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.837 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.837 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.837 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.838 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.838 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.838 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.838 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.838 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.838 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.838 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.839 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.839 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.839 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.839 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1ad9f98e:::obj-34:head", 2020-11-16T20:33:45.839 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'34", 2020-11-16T20:33:45.839 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.839 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4288.0:1", 2020-11-16T20:33:45.839 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.840 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:17.802587+0000", 2020-11-16T20:33:45.840 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.840 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.840 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.840 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.840 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.840 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.841 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.841 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.841 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.841 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.841 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.841 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.841 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.841 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.842 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.842 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.842 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.842 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.842 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:941f43ff:::obj-35:head", 2020-11-16T20:33:45.842 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'35", 2020-11-16T20:33:45.842 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.843 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4291.0:1", 2020-11-16T20:33:45.843 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.843 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:18.012636+0000", 2020-11-16T20:33:45.843 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.843 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.843 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.843 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.843 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.844 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.844 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.844 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.844 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.844 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.844 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.844 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.844 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.845 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.845 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.845 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.845 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.845 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.845 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:24ebabe9:::obj-36:head", 2020-11-16T20:33:45.845 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'36", 2020-11-16T20:33:45.846 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:45.846 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4294.0:1", 2020-11-16T20:33:45.846 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:45.846 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:18.175146+0000", 2020-11-16T20:33:45.846 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:45.846 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:45.846 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:45.846 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:45.847 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:45.847 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:45.847 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.847 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.847 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:45.847 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:45.847 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:45.848 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:45.848 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:45.848 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.848 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:45.848 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:45.848 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:45.848 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:45.848 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:5f3a5398:::obj-37:head", 2020-11-16T20:33:45.849 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'37", 2020-11-16T20:33:45.849 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:46.261 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4297.0:1", 2020-11-16T20:33:46.262 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:46.262 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:18.331363+0000", 2020-11-16T20:33:46.262 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:46.262 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:46.262 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:46.263 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:46.263 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:46.263 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:46.263 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.263 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:46.263 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:46.263 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:46.264 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:46.264 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:46.264 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:46.264 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.264 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.264 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:46.264 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:46.264 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:46.265 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:c96cb37e:::obj-38:head", 2020-11-16T20:33:46.265 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'38", 2020-11-16T20:33:46.265 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:46.265 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4300.0:1", 2020-11-16T20:33:46.265 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:46.265 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:18.483661+0000", 2020-11-16T20:33:46.265 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:46.266 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:46.266 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:46.266 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:46.266 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:46.266 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:46.266 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.266 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:46.267 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:46.267 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:46.267 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:46.267 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:46.267 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:46.267 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.267 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.268 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:46.268 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:46.268 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:46.268 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:69e36996:::obj-39:head", 2020-11-16T20:33:46.268 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'39", 2020-11-16T20:33:46.268 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:46.268 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4303.0:1", 2020-11-16T20:33:46.268 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:46.269 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:18.674408+0000", 2020-11-16T20:33:46.269 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:46.269 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:46.269 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:46.269 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:46.269 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:46.269 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:46.270 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.270 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:46.270 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:46.270 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:46.270 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:46.270 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:46.270 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:46.270 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.271 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.271 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:46.271 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:33:46.271 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:33:46.271 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:9f75773e:::obj-40:head", 2020-11-16T20:33:46.271 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'40", 2020-11-16T20:33:46.271 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:33:46.272 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4306.0:1", 2020-11-16T20:33:46.272 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:33:46.272 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:33:18.836137+0000", 2020-11-16T20:33:46.272 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:33:46.272 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:33:46.272 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:33:46.272 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:33:46.272 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:33:46.273 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:33:46.273 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.273 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:46.273 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:33:46.273 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:33:46.273 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:33:46.273 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:33:46.273 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:33:46.274 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.274 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.274 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.274 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:33:46.274 INFO:tasks.workunit.client.0.smithi189.stdout: "dups": [] 2020-11-16T20:33:46.274 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:33:46.274 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_missing_t": { 2020-11-16T20:33:46.275 INFO:tasks.workunit.client.0.smithi189.stdout: "missing": [], 2020-11-16T20:33:46.275 INFO:tasks.workunit.client.0.smithi189.stdout: "may_include_deletes": true 2020-11-16T20:33:46.275 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:33:46.275 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:33:46.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-11-16T20:33:46.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=40 2020-11-16T20:33:46.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:85: _common_test: '[' 40 '!=' 40 ']' 2020-11-16T20:33:46.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-11-16T20:33:46.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=0 2020-11-16T20:33:46.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:90: _common_test: '[' 0 '!=' 0 ']' 2020-11-16T20:33:46.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-11-16T20:33:46.283 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after v 0'0 2020-11-16T20:33:46.283 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'40 2020-11-16T20:33:46.283 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'39 2020-11-16T20:33:46.283 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'38 2020-11-16T20:33:46.283 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'37 2020-11-16T20:33:46.283 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'36 2020-11-16T20:33:46.284 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'35 2020-11-16T20:33:46.284 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'34 2020-11-16T20:33:46.284 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'33 2020-11-16T20:33:46.284 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'32 2020-11-16T20:33:46.284 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'31 2020-11-16T20:33:46.284 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'30 2020-11-16T20:33:46.284 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'29 2020-11-16T20:33:46.284 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'28 2020-11-16T20:33:46.285 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'27 2020-11-16T20:33:46.285 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'26 2020-11-16T20:33:46.285 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'25 2020-11-16T20:33:46.285 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'24 2020-11-16T20:33:46.285 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'23 2020-11-16T20:33:46.285 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'22 2020-11-16T20:33:46.285 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'21 2020-11-16T20:33:46.286 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'20 2020-11-16T20:33:46.286 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'19 2020-11-16T20:33:46.286 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'18 2020-11-16T20:33:46.286 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'17 2020-11-16T20:33:46.286 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'16 2020-11-16T20:33:46.286 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'15 2020-11-16T20:33:46.286 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'14 2020-11-16T20:33:46.287 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'13 2020-11-16T20:33:46.287 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'12 2020-11-16T20:33:46.287 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'11 2020-11-16T20:33:46.287 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'10 2020-11-16T20:33:46.287 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'9 2020-11-16T20:33:46.287 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'8 2020-11-16T20:33:46.290 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'7 2020-11-16T20:33:46.291 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'6 2020-11-16T20:33:46.291 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'5 2020-11-16T20:33:46.291 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'4 2020-11-16T20:33:46.291 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'3 2020-11-16T20:33:46.291 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 32'2 2020-11-16T20:33:46.291 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_after copy log version 31'1 2020-11-16T20:33:46.291 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:33:20.190+0000 7f85ea13e700 20 copy_up_to/copy_after earliest_dup_version 0 2020-11-16T20:33:46.292 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after v 0'0 2020-11-16T20:33:46.292 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'40 2020-11-16T20:33:46.292 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'39 2020-11-16T20:33:46.292 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'38 2020-11-16T20:33:46.292 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'37 2020-11-16T20:33:46.292 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'36 2020-11-16T20:33:46.292 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'35 2020-11-16T20:33:46.292 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'34 2020-11-16T20:33:46.293 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'33 2020-11-16T20:33:46.293 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'32 2020-11-16T20:33:46.293 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'31 2020-11-16T20:33:46.293 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'30 2020-11-16T20:33:46.293 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'29 2020-11-16T20:33:46.293 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'28 2020-11-16T20:33:46.293 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'27 2020-11-16T20:33:46.294 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'26 2020-11-16T20:33:46.294 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'25 2020-11-16T20:33:46.294 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'24 2020-11-16T20:33:46.294 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'23 2020-11-16T20:33:46.294 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'22 2020-11-16T20:33:46.294 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'21 2020-11-16T20:33:46.294 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'20 2020-11-16T20:33:46.295 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'19 2020-11-16T20:33:46.295 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'18 2020-11-16T20:33:46.295 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'17 2020-11-16T20:33:46.295 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'16 2020-11-16T20:33:46.295 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'15 2020-11-16T20:33:46.295 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'14 2020-11-16T20:33:46.295 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'13 2020-11-16T20:33:46.295 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'12 2020-11-16T20:33:46.296 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'11 2020-11-16T20:33:46.296 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.438+0000 7f29550be700 20 copy_after copy log version 32'10 2020-11-16T20:33:46.296 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'9 2020-11-16T20:33:46.296 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'8 2020-11-16T20:33:46.296 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'7 2020-11-16T20:33:46.296 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'6 2020-11-16T20:33:46.296 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'5 2020-11-16T20:33:46.297 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'4 2020-11-16T20:33:46.297 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'3 2020-11-16T20:33:46.297 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'2 2020-11-16T20:33:46.297 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 31'1 2020-11-16T20:33:46.297 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_up_to/copy_after earliest_dup_version 0 2020-11-16T20:33:46.297 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after v 0'0 2020-11-16T20:33:46.297 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'40 2020-11-16T20:33:46.297 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'39 2020-11-16T20:33:46.298 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'38 2020-11-16T20:33:46.298 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'37 2020-11-16T20:33:46.298 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'36 2020-11-16T20:33:46.298 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'35 2020-11-16T20:33:46.298 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'34 2020-11-16T20:33:46.298 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'33 2020-11-16T20:33:46.298 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'32 2020-11-16T20:33:46.299 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'31 2020-11-16T20:33:46.299 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'30 2020-11-16T20:33:46.299 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'29 2020-11-16T20:33:46.299 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'28 2020-11-16T20:33:46.299 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'27 2020-11-16T20:33:46.299 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'26 2020-11-16T20:33:46.299 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'25 2020-11-16T20:33:46.299 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'24 2020-11-16T20:33:46.300 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'23 2020-11-16T20:33:46.300 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'22 2020-11-16T20:33:46.301 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'21 2020-11-16T20:33:46.301 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'20 2020-11-16T20:33:46.301 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'19 2020-11-16T20:33:46.301 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'18 2020-11-16T20:33:46.301 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'17 2020-11-16T20:33:46.302 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'16 2020-11-16T20:33:46.302 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'15 2020-11-16T20:33:46.302 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'14 2020-11-16T20:33:46.302 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'13 2020-11-16T20:33:46.302 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'12 2020-11-16T20:33:46.302 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'11 2020-11-16T20:33:46.302 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'10 2020-11-16T20:33:46.303 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'9 2020-11-16T20:33:46.303 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'8 2020-11-16T20:33:46.303 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'7 2020-11-16T20:33:46.303 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'6 2020-11-16T20:33:46.303 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'5 2020-11-16T20:33:46.303 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'4 2020-11-16T20:33:46.303 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'3 2020-11-16T20:33:46.303 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 32'2 2020-11-16T20:33:46.304 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_after copy log version 31'1 2020-11-16T20:33:46.304 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-11-16T20:33:20.442+0000 7f29550be700 20 copy_up_to/copy_after earliest_dup_version 0 2020-11-16T20:33:46.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-11-16T20:33:46.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2020-11-16T20:33:46.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-11-16T20:33:46.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:33:46.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:33:46.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-11-16T20:33:46.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:33:46.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:33:46.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:33:46.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:33:46.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:33:46.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:33:46.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:33:46.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:33:46.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:33:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:33:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:33:46.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:33:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:33:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:33:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:33:46.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:33:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:33:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:33:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-11-16T20:33:46.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:33:46.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:33:46.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:33:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90694 2020-11-16T20:33:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:33:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:33:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-11-16T20:33:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-11-16T20:33:46.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-11-16T20:33:46.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-11-16T20:33:46.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:33:46.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:33:46.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-11-16T20:33:46.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:33:46.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:33:46.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:33:46.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:33:46.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:33:46.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:33:46.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:33:46.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:33:46.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:33:46.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:33:46.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:33:46.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:33:46.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:33:46.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:33:46.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:33:46.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:33:46.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:33:46.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:33:46.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-11-16T20:33:46.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:33:46.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:33:46.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:33:46.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90694 2020-11-16T20:33:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:33:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:33:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-11-16T20:33:46.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:33:46.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:33:46.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:33:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.90694 2020-11-16T20:33:46.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:33:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T20:33:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:33:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-11-16T20:33:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_recovery_2 td/osd-backfill-recovery-log 2020-11-16T20:33:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:129: TEST_recovery_2: local dir=td/osd-backfill-recovery-log 2020-11-16T20:33:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:131: TEST_recovery_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 151 10 141 20 2020-11-16T20:33:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-11-16T20:33:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-11-16T20:33:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=151 2020-11-16T20:33:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=10 2020-11-16T20:33:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=141 2020-11-16T20:33:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=20 2020-11-16T20:33:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-11-16T20:33:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-11-16T20:33:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-11-16T20:33:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:33:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:33:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:33:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-11-16T20:33:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-11-16T20:33:46.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:33:46.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:33:46.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:33:46.686 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:33:46.686 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:33:46.686 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:33:46.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:33:46.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:33:46.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:33:46.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:33:46.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:33:46.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:33:46.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:33:46.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:33:46.914 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:33:46.914 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:33:46.914 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:33:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:33:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:33:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:33:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90694/ceph-mon.a.asok 2020-11-16T20:33:46.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:33:46.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.90694/ceph-mon.a.asok config get fsid 2020-11-16T20:33:47.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:33:47.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:33:47.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:33:47.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:33:47.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:33:47.027 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:33:47.027 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:33:47.028 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:33:47.028 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:33:47.028 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:33:47.029 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:33:47.029 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90694/ceph-mon.a.asok 2020-11-16T20:33:47.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:33:47.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.90694/ceph-mon.a.asok config get mon_host 2020-11-16T20:33:47.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-11-16T20:33:47.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-recovery-log 2020-11-16T20:33:47.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:33:47.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:33:47.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:33:47.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-11-16T20:33:47.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:33:48.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:33:48.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:33:48.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:33:48.082 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:33:48.082 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:33:48.082 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:33:48.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:33:48.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:33:48.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:33:48.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-11-16T20:33:48.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-11-16T20:33:48.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-11-16T20:33:48.115 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-11-16T20:33:48.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-11-16T20:33:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:33:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-11-16T20:33:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:33:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:33:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:33:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:33:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-11-16T20:33:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:33:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:33:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:33:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:33:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-11-16T20:33:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-11-16T20:33:48.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:33:48.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-11-16T20:33:48.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:33:48.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:33:48.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:33:48.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:33:48.122 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:33:48.122 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:33:48.123 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:33:48.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:33:48.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:33:48.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:33:48.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:33:48.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:33:48.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:33:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:33:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:33:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:33:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:33:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:33:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:33:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-11-16T20:33:48.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:33:48.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8c9bf8b9-31eb-40ee-b604-88f04af3d3fe 2020-11-16T20:33:48.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 8c9bf8b9-31eb-40ee-b604-88f04af3d3fe' 2020-11-16T20:33:48.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:33:48.126 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 8c9bf8b9-31eb-40ee-b604-88f04af3d3fe 2020-11-16T20:33:48.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAs4rJfvEdkCBAAMKtL2rC0fvSdAFJmoPJsDw== 2020-11-16T20:33:48.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAs4rJfvEdkCBAAMKtL2rC0fvSdAFJmoPJsDw=="}' 2020-11-16T20:33:48.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8c9bf8b9-31eb-40ee-b604-88f04af3d3fe -i td/osd-backfill-recovery-log/0/new.json 2020-11-16T20:33:48.475 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:33:48.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-11-16T20:33:48.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAs4rJfvEdkCBAAMKtL2rC0fvSdAFJmoPJsDw== --osd-uuid 8c9bf8b9-31eb-40ee-b604-88f04af3d3fe 2020-11-16T20:33:48.523 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:48.522+0000 7f4351d6ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:48.525 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:48.522+0000 7f4351d6ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:48.526 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:48.522+0000 7f4351d6ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:48.527 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:48.526+0000 7f4351d6ed80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:33:48.527 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:48.526+0000 7f4351d6ed80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:33:48.527 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:48.526+0000 7f4351d6ed80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-11-16T20:33:48.581 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:48.578+0000 7f4351d6ed80 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-11-16T20:33:53.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-11-16T20:33:53.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:33:53.206 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:33:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:33:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:33:53.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:33:53.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:33:53.679 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:33:53.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:33:53.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:33:53.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:33:53.701 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:53.698+0000 7fe2f7a53d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:53.704 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:53.702+0000 7fe2f7a53d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:53.705 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:53.702+0000 7fe2f7a53d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:54.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:33:54.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:33:54.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:33:54.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:33:54.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:33:54.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:33:54.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:33:54.060 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:33:54.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:33:54.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:33:54.237 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:54.234+0000 7fe2f7a53d80 -1 Falling back to public interface 2020-11-16T20:33:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:33:55.088 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:55.086+0000 7fe2f7a53d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:33:55.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:33:55.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:33:55.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:33:55.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:33:55.450 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:33:55.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:33:55.835 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/125896,v1:127.0.0.1:6803/125896] [v2:127.0.0.1:6804/125896,v1:127.0.0.1:6805/125896] exists,up 8c9bf8b9-31eb-40ee-b604-88f04af3d3fe 2020-11-16T20:33:55.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:33:55.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:33:55.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:33:55.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:33:55.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-11-16T20:33:55.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:33:55.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:33:55.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:33:55.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:33:55.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-11-16T20:33:55.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:33:55.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:33:55.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:33:55.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:33:55.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-11-16T20:33:55.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-11-16T20:33:55.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:33:55.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-11-16T20:33:55.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:33:55.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:33:55.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:33:55.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:33:55.840 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:33:55.840 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:33:55.840 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:33:55.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:33:55.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:33:55.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:33:55.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:33:55.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:33:55.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:33:55.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:33:55.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:33:55.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:33:55.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:33:55.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:33:55.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:33:55.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-11-16T20:33:55.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:33:55.843 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 1dbf38dc-738d-4a5f-aac5-13931eefd0d5 2020-11-16T20:33:55.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1dbf38dc-738d-4a5f-aac5-13931eefd0d5 2020-11-16T20:33:55.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 1dbf38dc-738d-4a5f-aac5-13931eefd0d5' 2020-11-16T20:33:55.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:33:55.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAz4rJf6v0KMxAAcz8igCofxJr6tBu45ZBAGQ== 2020-11-16T20:33:55.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAz4rJf6v0KMxAAcz8igCofxJr6tBu45ZBAGQ=="}' 2020-11-16T20:33:55.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1dbf38dc-738d-4a5f-aac5-13931eefd0d5 -i td/osd-backfill-recovery-log/1/new.json 2020-11-16T20:33:56.322 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:33:56.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-11-16T20:33:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAz4rJf6v0KMxAAcz8igCofxJr6tBu45ZBAGQ== --osd-uuid 1dbf38dc-738d-4a5f-aac5-13931eefd0d5 2020-11-16T20:33:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:56.362+0000 7f5819788d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:56.366+0000 7f5819788d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:56.366+0000 7f5819788d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:33:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:56.366+0000 7f5819788d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:33:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:56.366+0000 7f5819788d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:33:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:56.366+0000 7f5819788d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-11-16T20:33:56.465 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:33:56.466+0000 7f5819788d80 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-11-16T20:34:01.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-11-16T20:34:01.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:34:01.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:34:01.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:34:01.146 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:34:01.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:34:01.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:34:01.629 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:34:01.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:34:01.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:34:01.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:34:01.651 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:01.649+0000 7fe89bdabd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:01.654 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:01.649+0000 7fe89bdabd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:01.655 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:01.653+0000 7fe89bdabd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:01.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:34:01.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:34:01.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:34:01.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:34:01.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:34:01.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:34:01.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:34:01.998 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:34:01.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:34:01.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:34:02.193 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:02.189+0000 7fe89bdabd80 -1 Falling back to public interface 2020-11-16T20:34:02.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:34:03.055 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:03.049+0000 7fe89bdabd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:34:03.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:34:03.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:34:03.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:34:03.419 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:34:03.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:34:03.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:34:03.825 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/126890,v1:127.0.0.1:6811/126890] [v2:127.0.0.1:6812/126890,v1:127.0.0.1:6813/126890] exists,up 1dbf38dc-738d-4a5f-aac5-13931eefd0d5 2020-11-16T20:34:03.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:34:03.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:34:03.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:34:03.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:34:03.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-11-16T20:34:03.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:34:03.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:34:03.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:34:03.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:34:03.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-11-16T20:34:03.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:34:03.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:34:03.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:34:03.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:34:03.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-11-16T20:34:03.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-11-16T20:34:03.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:34:03.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-11-16T20:34:03.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:34:03.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:34:03.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:34:03.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:34:03.829 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:34:03.830 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:34:03.830 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:34:03.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:34:03.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:34:03.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:34:03.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:34:03.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:34:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:34:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:34:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:34:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:34:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:34:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:34:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:34:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-11-16T20:34:03.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:34:03.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c55ed986-523d-4129-9366-ccbd3e29ee89 2020-11-16T20:34:03.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 c55ed986-523d-4129-9366-ccbd3e29ee89' 2020-11-16T20:34:03.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:34:03.833 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 c55ed986-523d-4129-9366-ccbd3e29ee89 2020-11-16T20:34:03.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA74rJfphxhMhAAmKXyTPkp5Cz1tDIxk5/PnA== 2020-11-16T20:34:03.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA74rJfphxhMhAAmKXyTPkp5Cz1tDIxk5/PnA=="}' 2020-11-16T20:34:03.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c55ed986-523d-4129-9366-ccbd3e29ee89 -i td/osd-backfill-recovery-log/2/new.json 2020-11-16T20:34:04.500 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:34:04.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-11-16T20:34:04.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA74rJfphxhMhAAmKXyTPkp5Cz1tDIxk5/PnA== --osd-uuid c55ed986-523d-4129-9366-ccbd3e29ee89 2020-11-16T20:34:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:04.541+0000 7fb018eafd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:04.598 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:04.593+0000 7fb018eafd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:04.599 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:04.597+0000 7fb018eafd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:04.600 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:04.597+0000 7fb018eafd80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:34:04.600 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:04.597+0000 7fb018eafd80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:34:04.601 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:04.597+0000 7fb018eafd80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-11-16T20:34:04.656 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:04.653+0000 7fb018eafd80 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-11-16T20:34:09.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-11-16T20:34:09.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:34:09.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:34:09.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:34:09.102 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:34:09.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:34:09.577 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:34:09.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:34:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:34:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:34:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:34:09.608 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:09.605+0000 7f2fc896cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:09.611 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:09.609+0000 7f2fc896cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:09.612 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:09.609+0000 7f2fc896cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:09.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:34:09.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:34:09.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:34:09.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:34:09.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:34:09.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:34:09.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:34:09.979 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:34:09.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:34:09.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:34:10.152 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:10.149+0000 7f2fc896cd80 -1 Falling back to public interface 2020-11-16T20:34:10.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:34:10.962 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:10.957+0000 7f2fc896cd80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:34:11.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:34:11.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:34:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:34:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:34:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:34:11.387 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:34:11.756 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/127884,v1:127.0.0.1:6819/127884] [v2:127.0.0.1:6820/127884,v1:127.0.0.1:6821/127884] exists,up c55ed986-523d-4129-9366-ccbd3e29ee89 2020-11-16T20:34:11.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:34:11.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:34:11.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:34:11.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:34:11.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-11-16T20:34:11.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:34:11.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:34:11.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T20:34:11.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:34:11.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-11-16T20:34:11.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:34:11.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:34:11.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:34:11.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:34:11.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-11-16T20:34:11.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-11-16T20:34:11.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:34:11.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-11-16T20:34:11.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:34:11.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:34:11.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:34:11.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:34:11.761 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:34:11.761 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:34:11.761 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:34:11.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:34:11.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:34:11.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:34:11.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:34:11.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:34:11.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:34:11.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:34:11.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:34:11.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:34:11.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:34:11.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:34:11.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:34:11.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-11-16T20:34:11.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:34:11.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a19a7ff8-9f42-4d3e-9410-b6b1daa60e7d 2020-11-16T20:34:11.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 a19a7ff8-9f42-4d3e-9410-b6b1daa60e7d' 2020-11-16T20:34:11.764 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 a19a7ff8-9f42-4d3e-9410-b6b1daa60e7d 2020-11-16T20:34:11.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:34:11.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBD4rJf7oWZLhAARtYkJ0ttPKHbM0I5lyVl8Q== 2020-11-16T20:34:11.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBD4rJf7oWZLhAARtYkJ0ttPKHbM0I5lyVl8Q=="}' 2020-11-16T20:34:11.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a19a7ff8-9f42-4d3e-9410-b6b1daa60e7d -i td/osd-backfill-recovery-log/3/new.json 2020-11-16T20:34:12.394 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T20:34:12.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-11-16T20:34:12.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBD4rJf7oWZLhAARtYkJ0ttPKHbM0I5lyVl8Q== --osd-uuid a19a7ff8-9f42-4d3e-9410-b6b1daa60e7d 2020-11-16T20:34:12.439 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:12.437+0000 7f0b5b402d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:12.441 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:12.437+0000 7f0b5b402d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:12.443 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:12.441+0000 7f0b5b402d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:12.444 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:12.441+0000 7f0b5b402d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:34:12.444 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:12.441+0000 7f0b5b402d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:34:12.444 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:12.441+0000 7f0b5b402d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-11-16T20:34:12.486 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:12.481+0000 7f0b5b402d80 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-11-16T20:34:17.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-11-16T20:34:17.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:34:17.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T20:34:17.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:34:17.156 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T20:34:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T20:34:17.662 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T20:34:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:34:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:34:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:34:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:34:17.690 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:17.685+0000 7fe8559c9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:17.693 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:17.689+0000 7fe8559c9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:17.694 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:17.689+0000 7fe8559c9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:18.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T20:34:18.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:34:18.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T20:34:18.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:34:18.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:34:18.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:34:18.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:34:18.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:34:18.062 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:34:18.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:34:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:18.241+0000 7fe8559c9d80 -1 Falling back to public interface 2020-11-16T20:34:18.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:34:19.086 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:19.081+0000 7fe8559c9d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T20:34:19.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:34:19.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:34:19.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:34:19.453 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:34:19.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T20:34:19.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:34:19.847 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/128878,v1:127.0.0.1:6827/128878] [v2:127.0.0.1:6828/128878,v1:127.0.0.1:6829/128878] exists,up a19a7ff8-9f42-4d3e-9410-b6b1daa60e7d 2020-11-16T20:34:19.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:34:19.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:34:19.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:34:19.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:34:19.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-11-16T20:34:19.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:34:19.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:34:19.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T20:34:19.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:34:19.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-11-16T20:34:19.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:34:19.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:34:19.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:34:19.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:34:19.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-11-16T20:34:19.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-11-16T20:34:19.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:34:19.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-11-16T20:34:19.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:34:19.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:34:19.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:34:19.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:34:19.852 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:34:19.853 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:34:19.853 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:34:19.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:34:19.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:34:19.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:34:19.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:34:19.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:34:19.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:34:19.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:34:19.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:34:19.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:34:19.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:34:19.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:34:19.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:34:19.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-11-16T20:34:19.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:34:19.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0c270f15-0b58-4040-a00b-4cfd861b8912 2020-11-16T20:34:19.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 0c270f15-0b58-4040-a00b-4cfd861b8912' 2020-11-16T20:34:19.856 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 0c270f15-0b58-4040-a00b-4cfd861b8912 2020-11-16T20:34:19.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:34:19.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBL4rJfJkYuNBAATT8YGDqvWQrgQ1EfUT4s5w== 2020-11-16T20:34:19.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBL4rJfJkYuNBAATT8YGDqvWQrgQ1EfUT4s5w=="}' 2020-11-16T20:34:19.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0c270f15-0b58-4040-a00b-4cfd861b8912 -i td/osd-backfill-recovery-log/4/new.json 2020-11-16T20:34:20.550 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T20:34:20.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-11-16T20:34:20.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBL4rJfJkYuNBAATT8YGDqvWQrgQ1EfUT4s5w== --osd-uuid 0c270f15-0b58-4040-a00b-4cfd861b8912 2020-11-16T20:34:20.595 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:20.593+0000 7f61f0ff0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:20.611 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:20.605+0000 7f61f0ff0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:20.613 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:20.609+0000 7f61f0ff0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:20.613 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:20.609+0000 7f61f0ff0d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:34:20.614 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:20.609+0000 7f61f0ff0d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:34:20.614 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:20.609+0000 7f61f0ff0d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-11-16T20:34:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:20.673+0000 7f61f0ff0d80 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-11-16T20:34:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-11-16T20:34:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:34:25.150 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T20:34:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T20:34:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:34:25.661 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T20:34:25.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T20:34:25.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:34:25.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:34:25.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:34:25.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:34:25.688 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:25.685+0000 7fea3eb76d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:25.691 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:25.689+0000 7fea3eb76d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:25.692 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:25.689+0000 7fea3eb76d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:26.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T20:34:26.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:34:26.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T20:34:26.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:34:26.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:34:26.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:34:26.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:34:26.051 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:34:26.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:34:26.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:34:26.232 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:26.229+0000 7fea3eb76d80 -1 Falling back to public interface 2020-11-16T20:34:26.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:34:26.922 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:26.913+0000 7fea3eb76d80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T20:34:27.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:34:27.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:34:27.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:34:27.478 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:34:27.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:34:27.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T20:34:27.856 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/129872,v1:127.0.0.1:6835/129872] [v2:127.0.0.1:6836/129872,v1:127.0.0.1:6837/129872] exists,up 0c270f15-0b58-4040-a00b-4cfd861b8912 2020-11-16T20:34:27.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:34:27.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:34:27.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:34:27.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:34:27.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-11-16T20:34:27.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2020-11-16T20:34:27.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:34:27.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T20:34:27.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:34:27.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-11-16T20:34:27.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-11-16T20:34:27.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:34:27.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:34:27.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:34:27.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-11-16T20:34:27.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-11-16T20:34:27.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:34:27.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-11-16T20:34:27.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-11-16T20:34:27.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:34:27.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:34:27.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:34:27.861 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:34:27.861 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:34:27.862 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:34:27.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:34:27.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' 2020-11-16T20:34:27.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:34:27.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:34:27.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:34:27.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-11-16T20:34:27.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-11-16T20:34:27.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:34:27.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:34:27.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:34:27.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:34:27.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:34:27.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-11-16T20:34:27.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:34:27.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=adf3f28e-f144-4198-9ccb-cfac44210648 2020-11-16T20:34:27.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 adf3f28e-f144-4198-9ccb-cfac44210648' 2020-11-16T20:34:27.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:34:27.864 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 adf3f28e-f144-4198-9ccb-cfac44210648 2020-11-16T20:34:27.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBT4rJfUp53NBAA5lYwCAvBiCbOiMcQ1hPwUA== 2020-11-16T20:34:27.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBT4rJfUp53NBAA5lYwCAvBiCbOiMcQ1hPwUA=="}' 2020-11-16T20:34:27.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new adf3f28e-f144-4198-9ccb-cfac44210648 -i td/osd-backfill-recovery-log/5/new.json 2020-11-16T20:34:28.404 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T20:34:28.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-11-16T20:34:28.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBT4rJfUp53NBAA5lYwCAvBiCbOiMcQ1hPwUA== --osd-uuid adf3f28e-f144-4198-9ccb-cfac44210648 2020-11-16T20:34:28.451 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:28.449+0000 7fb358929d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:28.478 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:28.473+0000 7fb358929d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:28.479 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:28.477+0000 7fb358929d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:28.480 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:28.477+0000 7fb358929d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:34:28.480 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:28.477+0000 7fb358929d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:34:28.480 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:28.477+0000 7fb358929d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-11-16T20:34:28.536 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:28.533+0000 7fb358929d80 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-11-16T20:34:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-11-16T20:34:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:34:33.363 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T20:34:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T20:34:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:34:33.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T20:34:33.856 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T20:34:33.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=0fd08cb7-7161-499b-8d80-993b68b700ad --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.90694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:34:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:34:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:34:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:34:33.885 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:33.881+0000 7f3c54427d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:33.888 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:33.885+0000 7f3c54427d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:33.889 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:33.885+0000 7f3c54427d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:34:34.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T20:34:34.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:34:34.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T20:34:34.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:34:34.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:34:34.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:34:34.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:34:34.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:34:34.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T20:34:34.239 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:34:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:34.413+0000 7f3c54427d80 -1 Falling back to public interface 2020-11-16T20:34:34.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:34:35.325 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:34:35.321+0000 7f3c54427d80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T20:34:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:34:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:34:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:34:35.622 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:34:35.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:34:35.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T20:34:36.131 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/130866,v1:127.0.0.1:6843/130866] [v2:127.0.0.1:6844/130866,v1:127.0.0.1:6845/130866] exists,up adf3f28e-f144-4198-9ccb-cfac44210648 2020-11-16T20:34:36.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:34:36.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:34:36.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:34:36.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-11-16T20:34:36.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T20:34:36.631 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T20:34:36.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:34:37.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 141 2020-11-16T20:34:37.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:37.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-11-16T20:34:37.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:37.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-11-16T20:34:38.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:38.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-11-16T20:34:38.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:38.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-11-16T20:34:38.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:38.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-11-16T20:34:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-11-16T20:34:39.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:39.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-11-16T20:34:39.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:39.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-11-16T20:34:39.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:39.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-11-16T20:34:39.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:39.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-11-16T20:34:39.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:39.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-11-16T20:34:39.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:39.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-11-16T20:34:40.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:40.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-11-16T20:34:40.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:40.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-11-16T20:34:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-11-16T20:34:40.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:40.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-11-16T20:34:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:40.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-11-16T20:34:41.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:41.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-11-16T20:34:41.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:41.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-11-16T20:34:41.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:41.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-11-16T20:34:41.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:41.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-11-16T20:34:41.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:41.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-11-16T20:34:41.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:41.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-11-16T20:34:42.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:42.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-11-16T20:34:42.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:42.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-11-16T20:34:42.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:42.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-11-16T20:34:42.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:42.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-11-16T20:34:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:42.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-11-16T20:34:43.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:43.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-11-16T20:34:43.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:43.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-11-16T20:34:43.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:43.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-11-16T20:34:43.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:43.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-11-16T20:34:43.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:43.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-11-16T20:34:43.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:43.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-11-16T20:34:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-11-16T20:34:44.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:44.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-11-16T20:34:44.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:44.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-11-16T20:34:44.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:44.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-11-16T20:34:44.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:44.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-11-16T20:34:44.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:44.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-11-16T20:34:45.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:45.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-11-16T20:34:45.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:45.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-11-16T20:34:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-11-16T20:34:45.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-11-16T20:34:45.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:45.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-11-16T20:34:46.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:46.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-11-16T20:34:46.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:46.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-11-16T20:34:46.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:46.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-11-16T20:34:46.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:46.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-11-16T20:34:46.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:46.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-11-16T20:34:46.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:46.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-11-16T20:34:47.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:47.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-11-16T20:34:47.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:47.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-11-16T20:34:47.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:47.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-11-16T20:34:47.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:47.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-11-16T20:34:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-11-16T20:34:47.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:47.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-11-16T20:34:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-11-16T20:34:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-11-16T20:34:48.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:48.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-11-16T20:34:48.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:48.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-11-16T20:34:48.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:48.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-11-16T20:34:48.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:48.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-11-16T20:34:49.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:49.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-11-16T20:34:49.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:49.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-11-16T20:34:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-11-16T20:34:49.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:49.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-11-16T20:34:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-11-16T20:34:50.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:50.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-11-16T20:34:50.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:50.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-11-16T20:34:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-11-16T20:34:50.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:50.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-11-16T20:34:50.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:50.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-11-16T20:34:50.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:50.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-11-16T20:34:51.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:51.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-11-16T20:34:51.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:51.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-11-16T20:34:51.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:51.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-11-16T20:34:51.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:51.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-11-16T20:34:51.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:51.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-11-16T20:34:51.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:51.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-11-16T20:34:52.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:52.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-11-16T20:34:52.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:52.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-11-16T20:34:52.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:52.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-11-16T20:34:52.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:52.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-11-16T20:34:52.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:52.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-11-16T20:34:53.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-11-16T20:34:53.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:53.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-11-16T20:34:53.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:53.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-11-16T20:34:53.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:53.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-11-16T20:34:53.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:53.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-11-16T20:34:53.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:53.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-11-16T20:34:53.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:53.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-11-16T20:34:54.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:54.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-11-16T20:34:54.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:54.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-11-16T20:34:54.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:54.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-11-16T20:34:54.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:54.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-11-16T20:34:54.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:54.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-11-16T20:34:55.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:55.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-11-16T20:34:55.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:55.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-11-16T20:34:55.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:55.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-11-16T20:34:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-11-16T20:34:55.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:55.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-11-16T20:34:55.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:55.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-11-16T20:34:56.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:56.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-11-16T20:34:56.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:56.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-11-16T20:34:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-11-16T20:34:56.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:56.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-11-16T20:34:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:56.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-11-16T20:34:56.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:56.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-11-16T20:34:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-11-16T20:34:57.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:57.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-11-16T20:34:57.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:57.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-11-16T20:34:57.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:57.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-11-16T20:34:57.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:57.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-11-16T20:34:57.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:57.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-11-16T20:34:58.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:58.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-11-16T20:34:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:58.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-11-16T20:34:58.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:58.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-11-16T20:34:58.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:58.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-11-16T20:34:58.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:58.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-11-16T20:34:59.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:59.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-11-16T20:34:59.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:59.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-11-16T20:34:59.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:59.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-11-16T20:34:59.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:59.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-11-16T20:34:59.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:59.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-11-16T20:34:59.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:34:59.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-11-16T20:35:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-11-16T20:35:00.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:00.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-11-16T20:35:00.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:00.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-11-16T20:35:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-11-16T20:35:00.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:00.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-11-16T20:35:00.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:00.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-11-16T20:35:01.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:01.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-11-16T20:35:01.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:01.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-11-16T20:35:01.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:01.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-11-16T20:35:01.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:01.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-11-16T20:35:01.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:01.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-11-16T20:35:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-11-16T20:35:02.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:02.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-11-16T20:35:02.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:02.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-11-16T20:35:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-11-16T20:35:02.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-11-16T20:35:02.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-11-16T20:35:02.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-11-16T20:35:03.079 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:35:03.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-11-16T20:35:03.638 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-11-16T20:35:03.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 20 '!=' 0 ']' 2020-11-16T20:35:03.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: seq 1 20 2020-11-16T20:35:03.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:03.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-1 /etc/passwd 2020-11-16T20:35:05.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:05.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-2 /etc/passwd 2020-11-16T20:35:06.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:06.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-3 /etc/passwd 2020-11-16T20:35:06.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:06.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-4 /etc/passwd 2020-11-16T20:35:06.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:06.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-5 /etc/passwd 2020-11-16T20:35:06.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:06.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-6 /etc/passwd 2020-11-16T20:35:07.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:07.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-7 /etc/passwd 2020-11-16T20:35:07.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:07.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-8 /etc/passwd 2020-11-16T20:35:07.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:07.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-9 /etc/passwd 2020-11-16T20:35:07.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:07.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-10 /etc/passwd 2020-11-16T20:35:08.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:08.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-11 /etc/passwd 2020-11-16T20:35:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-12 /etc/passwd 2020-11-16T20:35:08.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:08.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-13 /etc/passwd 2020-11-16T20:35:09.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:09.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-14 /etc/passwd 2020-11-16T20:35:09.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:09.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-15 /etc/passwd 2020-11-16T20:35:09.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:09.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-16 /etc/passwd 2020-11-16T20:35:09.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:09.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-17 /etc/passwd 2020-11-16T20:35:10.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:10.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-18 /etc/passwd 2020-11-16T20:35:10.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:10.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-19 /etc/passwd 2020-11-16T20:35:10.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-11-16T20:35:10.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-20 /etc/passwd 2020-11-16T20:35:10.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-11-16T20:35:11.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-11-16T20:35:11.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:35:11.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:35:11.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:35:11.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:35:11.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:35:11.948 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:35:11.948 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:35:11.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:35:11.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:35:11.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:35:12.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:35:12.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:35:12.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:35:12.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:35:12.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:35:12.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:35:12.544 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:35:12.544 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:35:12.545 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:35:12.545 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T20:35:12.545 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T20:35:12.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:35:12.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:12.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:35:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836498 2020-11-16T20:35:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836498 2020-11-16T20:35:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498' 2020-11-16T20:35:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:12.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:35:12.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705681 2020-11-16T20:35:12.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705681 2020-11-16T20:35:12.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-38654705681' 2020-11-16T20:35:12.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:12.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:35:13.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542159 2020-11-16T20:35:13.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542159 2020-11-16T20:35:13.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-38654705681 2-60129542159' 2020-11-16T20:35:13.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:13.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:35:13.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378637 2020-11-16T20:35:13.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378637 2020-11-16T20:35:13.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-38654705681 2-60129542159 3-81604378637' 2020-11-16T20:35:13.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:13.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:35:13.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215116 2020-11-16T20:35:13.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215116 2020-11-16T20:35:13.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-38654705681 2-60129542159 3-81604378637 4-103079215116' 2020-11-16T20:35:13.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:13.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T20:35:13.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051595 2020-11-16T20:35:13.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051595 2020-11-16T20:35:13.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-38654705681 2-60129542159 3-81604378637 4-103079215116 5-124554051595' 2020-11-16T20:35:13.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:13.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836498 2020-11-16T20:35:13.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:13.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:35:13.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836498 2020-11-16T20:35:13.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:13.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836498 2020-11-16T20:35:13.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2020-11-16T20:35:13.540 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836498 2020-11-16T20:35:13.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:35:13.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836498 2020-11-16T20:35:13.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:35:14.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:35:14.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:35:15.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836498 -lt 21474836498 2020-11-16T20:35:15.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:15.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705681 2020-11-16T20:35:15.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:15.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:35:15.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705681 2020-11-16T20:35:15.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:15.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705681 2020-11-16T20:35:15.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705681' 2020-11-16T20:35:15.344 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705681 2020-11-16T20:35:15.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:35:15.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705681 -lt 38654705681 2020-11-16T20:35:15.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:15.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542159 2020-11-16T20:35:15.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:15.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:35:15.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542159 2020-11-16T20:35:15.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:15.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542159 2020-11-16T20:35:15.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542159' 2020-11-16T20:35:15.738 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542159 2020-11-16T20:35:15.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:35:16.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542159 -lt 60129542159 2020-11-16T20:35:16.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:16.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378637 2020-11-16T20:35:16.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:16.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:35:16.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:16.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378637 2020-11-16T20:35:16.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378637 2020-11-16T20:35:16.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378637' 2020-11-16T20:35:16.168 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378637 2020-11-16T20:35:16.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:35:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378638 -lt 81604378637 2020-11-16T20:35:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:16.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215116 2020-11-16T20:35:16.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:35:16.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215116 2020-11-16T20:35:16.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215116 2020-11-16T20:35:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215116' 2020-11-16T20:35:16.558 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215116 2020-11-16T20:35:16.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:35:16.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215116 -lt 103079215116 2020-11-16T20:35:16.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:16.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051595 2020-11-16T20:35:16.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:16.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T20:35:16.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051595 2020-11-16T20:35:16.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:16.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051595 2020-11-16T20:35:16.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051595' 2020-11-16T20:35:16.954 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051595 2020-11-16T20:35:16.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T20:35:17.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051595 -lt 124554051595 2020-11-16T20:35:17.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:35:17.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:17.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:17.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:35:17.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:17.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:17.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:17.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:17.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:17.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:17.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:18.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:18.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:18.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:18.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:35:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:35:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:35:18.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:18.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:18.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:18.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:18.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:18.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:18.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:18.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:18.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:18.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:19.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:19.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:19.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:19.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:19.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:19.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:19.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:19.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:19.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:19.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:20.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27036 2020-11-16T20:35:20.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27036 '!=' null 2020-11-16T20:35:20.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:20.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:20.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:20.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:20.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:20.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:20.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:20.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:20.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:20.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:20.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:20.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:20.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:20.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:21.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:21.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:21.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:21.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:21.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:21.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:21.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:21.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:21.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:21.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27036 2020-11-16T20:35:21.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27036 '!=' null 2020-11-16T20:35:21.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:21.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:21.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:21.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:21.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:21.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:21.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:21.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:21.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:21.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:22.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:22.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:22.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:22.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:22.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:22.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:22.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:22.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:22.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:22.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:22.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:22.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:22.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40885 2020-11-16T20:35:22.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40885 '!=' null 2020-11-16T20:35:22.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:22.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:22.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:23.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:23.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:23.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:23.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:23.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:23.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:23.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:23.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:23.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:23.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:23.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:23.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:24.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:24.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:24.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:24.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:24.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:24.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:24.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:24.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:24.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:24.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28355 2020-11-16T20:35:24.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28355 '!=' null 2020-11-16T20:35:24.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:24.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:24.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:24.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:24.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:24.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:24.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:24.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:24.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:24.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:24.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:24.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:24.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:24.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:24.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:25.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:25.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:25.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:25.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:25.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:25.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:25.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:25.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:25.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:25.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28355 2020-11-16T20:35:25.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28355 '!=' null 2020-11-16T20:35:25.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:25.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:25.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:26.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:26.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:26.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:26.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:26.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:26.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:26.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:26.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:26.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:26.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:26.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:26.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:26.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:26.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:26.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:26.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:26.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:26.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:26.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:26.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:26.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:27.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24399 2020-11-16T20:35:27.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24399 '!=' null 2020-11-16T20:35:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:27.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:27.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:27.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:27.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:27.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:27.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:27.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:27.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:27.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:27.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:27.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:27.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:28.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:28.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:28.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:28.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:28.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:28.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:28.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:28.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:28.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:28.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=37588 2020-11-16T20:35:28.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 37588 '!=' null 2020-11-16T20:35:28.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:28.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:28.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:28.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:28.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:28.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:28.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:28.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:28.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:28.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:28.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:29.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:29.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:29.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:29.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:29.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:29.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:29.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:29.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:29.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:29.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:29.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:29.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:29.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:30.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27036 2020-11-16T20:35:30.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27036 '!=' null 2020-11-16T20:35:30.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:30.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:30.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:30.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:30.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:30.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:30.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:30.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:30.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:30.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:30.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:30.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:30.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:31.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:31.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:31.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:31.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:31.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:31.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:31.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:31.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:31.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:31.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27036 2020-11-16T20:35:31.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27036 '!=' null 2020-11-16T20:35:31.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:31.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:31.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:31.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:31.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:31.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:31.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:31.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:31.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:31.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:31.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:32.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:32.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:32.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:32.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:32.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:32.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:32.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:32.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:32.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:32.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:32.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:32.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:32.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:33.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40883 2020-11-16T20:35:33.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40883 '!=' null 2020-11-16T20:35:33.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:33.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:33.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:33.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:33.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:33.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:33.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:33.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:33.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:33.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:33.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:33.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:33.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:33.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:34.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:34.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:34.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:34.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:34.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:34.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:34.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:34.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:34.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:34.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27035 2020-11-16T20:35:34.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27035 '!=' null 2020-11-16T20:35:34.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:34.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:34.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:34.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:34.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:34.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:34.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:34.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:34.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:34.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:34.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:35.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:35.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:35.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:35.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:35.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:35.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:35.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:35.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:35.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:35.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:35.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:35.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:36.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27035 2020-11-16T20:35:36.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27035 '!=' null 2020-11-16T20:35:36.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:36.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:36.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:36.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:36.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:36.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:36.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:36.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:36.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:36.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:36.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:36.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:36.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:36.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:36.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:37.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:37.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:37.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:37.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:37.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:37.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:37.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:37.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:37.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:37.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27035 2020-11-16T20:35:37.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27035 '!=' null 2020-11-16T20:35:37.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:37.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:37.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:37.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:37.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:37.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:37.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:37.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:37.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:37.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:37.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:38.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:38.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:38.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:38.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:38.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:38.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:38.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:39.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40882 2020-11-16T20:35:39.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40882 '!=' null 2020-11-16T20:35:39.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:39.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:39.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:39.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:39.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:39.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:39.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:39.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:39.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:39.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:39.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:39.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:39.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:39.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:39.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:40.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:40.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:40.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:40.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:40.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:40.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:40.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:40.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:40.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:40.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27694 2020-11-16T20:35:40.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27694 '!=' null 2020-11-16T20:35:40.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:40.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:40.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:40.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:40.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:40.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:40.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:40.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:40.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:40.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:40.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:41.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:41.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:41.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:41.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:41.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:41.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:41.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:41.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:41.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:41.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:41.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:41.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:41.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:42.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20771 2020-11-16T20:35:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20771 '!=' null 2020-11-16T20:35:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:42.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:42.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:42.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:42.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:42.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:42.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:42.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:42.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:42.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:42.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:35:42.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:42.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:42.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:43.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:35:43.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:35:43.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:35:43.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:35:43.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:35:43.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:35:43.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:35:43.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:35:43.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:35:43.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20771 2020-11-16T20:35:43.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20771 '!=' null 2020-11-16T20:35:43.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:35:43.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:35:43.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:35:43.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:35:43.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:35:43.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:35:43.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:35:43.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:35:43.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:35:43.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:35:43.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:35:43.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:35:43.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:35:43.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:35:43.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:35:44.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:35:44.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:35:44.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:35:44.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2020-11-16T20:35:44.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:35:44.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:35:44.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:35:44.817 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:35:44.817 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T20:35:44.817 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T20:35:44.818 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T20:35:44.818 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T20:35:44.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:35:44.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:44.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:35:44.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836506 2020-11-16T20:35:44.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836506 2020-11-16T20:35:44.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506' 2020-11-16T20:35:44.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:45.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:35:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705690 2020-11-16T20:35:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705690 2020-11-16T20:35:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-38654705690' 2020-11-16T20:35:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:45.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:35:45.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542167 2020-11-16T20:35:45.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542167 2020-11-16T20:35:45.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-38654705690 2-60129542167' 2020-11-16T20:35:45.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:45.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T20:35:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378645 2020-11-16T20:35:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378645 2020-11-16T20:35:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-38654705690 2-60129542167 3-81604378645' 2020-11-16T20:35:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:45.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T20:35:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215124 2020-11-16T20:35:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215124 2020-11-16T20:35:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-38654705690 2-60129542167 3-81604378645 4-103079215124' 2020-11-16T20:35:45.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:35:45.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T20:35:45.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051603 2020-11-16T20:35:45.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051603 2020-11-16T20:35:45.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-38654705690 2-60129542167 3-81604378645 4-103079215124 5-124554051603' 2020-11-16T20:35:45.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:45.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836506 2020-11-16T20:35:45.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:45.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:35:45.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836506 2020-11-16T20:35:45.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:45.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836506 2020-11-16T20:35:45.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2020-11-16T20:35:45.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:35:45.816 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836506 2020-11-16T20:35:46.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836506 -lt 21474836506 2020-11-16T20:35:46.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:46.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705690 2020-11-16T20:35:46.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:46.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:35:46.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705690 2020-11-16T20:35:46.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:46.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705690 2020-11-16T20:35:46.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705690' 2020-11-16T20:35:46.203 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705690 2020-11-16T20:35:46.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:35:46.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705690 -lt 38654705690 2020-11-16T20:35:46.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:46.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542167 2020-11-16T20:35:46.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:46.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:35:46.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542167 2020-11-16T20:35:46.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:46.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542167 2020-11-16T20:35:46.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542167' 2020-11-16T20:35:46.597 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542167 2020-11-16T20:35:46.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:35:46.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542167 -lt 60129542167 2020-11-16T20:35:46.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:46.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378645 2020-11-16T20:35:46.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:46.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T20:35:46.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:46.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378645 2020-11-16T20:35:46.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378645 2020-11-16T20:35:46.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378645' 2020-11-16T20:35:46.986 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378645 2020-11-16T20:35:46.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T20:35:47.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378646 -lt 81604378645 2020-11-16T20:35:47.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:47.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215124 2020-11-16T20:35:47.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:47.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T20:35:47.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215124 2020-11-16T20:35:47.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:47.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215124 2020-11-16T20:35:47.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215124' 2020-11-16T20:35:47.417 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215124 2020-11-16T20:35:47.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T20:35:47.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215124 -lt 103079215124 2020-11-16T20:35:47.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:35:47.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:35:47.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051603 2020-11-16T20:35:47.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T20:35:47.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051603 2020-11-16T20:35:47.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:35:47.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051603 2020-11-16T20:35:47.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051603' 2020-11-16T20:35:47.831 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051603 2020-11-16T20:35:47.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T20:35:48.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051604 -lt 124554051603 2020-11-16T20:35:48.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2020-11-16T20:35:48.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2020-11-16T20:35:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T20:35:48.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2020-11-16T20:35:48.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2020-11-16T20:35:48.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:35:48.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:35:48.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:35:48.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:35:48.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:35:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:35:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2020-11-16T20:35:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-11-16T20:35:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:35:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-11-16T20:35:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T20:35:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=4 2020-11-16T20:35:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T20:35:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-11-16T20:35:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-11-16T20:35:49.931 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T20:35:49.931 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_log_t": { 2020-11-16T20:35:49.931 INFO:tasks.workunit.client.0.smithi189.stdout: "head": "36'161", 2020-11-16T20:35:49.932 INFO:tasks.workunit.client.0.smithi189.stdout: "tail": "32'10", 2020-11-16T20:35:49.932 INFO:tasks.workunit.client.0.smithi189.stdout: "log": [ 2020-11-16T20:35:49.932 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.932 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.932 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1e6eca61:::obj-11:head", 2020-11-16T20:35:49.932 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'11", 2020-11-16T20:35:49.932 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.933 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4217.0:1", 2020-11-16T20:35:49.933 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.933 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:39.821465+0000", 2020-11-16T20:35:49.933 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.933 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.933 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.933 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.934 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.934 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.934 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.934 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.934 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.934 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.934 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.935 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.935 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.935 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.935 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.935 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.935 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.935 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.935 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:18dee445:::obj-12:head", 2020-11-16T20:35:49.936 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'12", 2020-11-16T20:35:49.936 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.936 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4220.0:1", 2020-11-16T20:35:49.936 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.936 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:40.051145+0000", 2020-11-16T20:35:49.936 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.936 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.937 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.937 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.937 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.937 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.937 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.937 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.937 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.938 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.938 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.938 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.938 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.938 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.938 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.938 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.938 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.939 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.939 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:283c5c90:::obj-13:head", 2020-11-16T20:35:49.939 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'13", 2020-11-16T20:35:49.939 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.939 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4223.0:1", 2020-11-16T20:35:49.939 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.939 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:40.257264+0000", 2020-11-16T20:35:49.940 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.940 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.940 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.940 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.940 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.940 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.940 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.940 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.941 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.941 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.941 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.941 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.941 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.941 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.941 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.942 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.942 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.942 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.942 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:cc896bf0:::obj-14:head", 2020-11-16T20:35:49.942 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'14", 2020-11-16T20:35:49.942 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.942 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4226.0:1", 2020-11-16T20:35:49.943 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.943 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:40.406536+0000", 2020-11-16T20:35:49.943 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.943 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.943 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.943 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.943 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.943 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.944 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.944 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.944 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.944 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.944 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.944 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.944 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.945 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.945 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.945 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.945 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.945 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.945 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:a581f807:::obj-15:head", 2020-11-16T20:35:49.945 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'15", 2020-11-16T20:35:49.946 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.946 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4229.0:1", 2020-11-16T20:35:49.946 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.946 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:40.564532+0000", 2020-11-16T20:35:49.946 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.946 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.946 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.946 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.947 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.947 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.947 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.947 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.947 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.947 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.947 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.948 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.948 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.948 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.948 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.948 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.948 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.948 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.948 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:c902ca7c:::obj-16:head", 2020-11-16T20:35:49.949 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'16", 2020-11-16T20:35:49.949 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.949 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4232.0:1", 2020-11-16T20:35:49.949 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.949 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:40.764928+0000", 2020-11-16T20:35:49.949 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.949 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.950 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.950 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.950 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.950 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.950 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.950 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.950 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.951 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.951 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.951 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.951 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.951 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.951 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.951 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.952 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.952 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.952 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:655283c8:::obj-17:head", 2020-11-16T20:35:49.952 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'17", 2020-11-16T20:35:49.952 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.952 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4235.0:1", 2020-11-16T20:35:49.952 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.953 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:40.925465+0000", 2020-11-16T20:35:49.953 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.953 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.953 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.953 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.953 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.953 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.953 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.954 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.954 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.954 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.954 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.954 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.954 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.954 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.955 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.955 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.955 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.955 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.955 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:d759d762:::obj-18:head", 2020-11-16T20:35:49.955 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'18", 2020-11-16T20:35:49.955 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.956 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4238.0:1", 2020-11-16T20:35:49.956 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.956 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:41.100975+0000", 2020-11-16T20:35:49.956 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.956 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.956 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.956 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.956 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.957 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.957 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.957 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.957 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.957 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.957 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.957 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.958 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.958 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.958 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.958 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.958 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.958 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.958 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1b934a2d:::obj-19:head", 2020-11-16T20:35:49.959 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'19", 2020-11-16T20:35:49.959 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.959 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4241.0:1", 2020-11-16T20:35:49.959 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.959 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:41.298705+0000", 2020-11-16T20:35:49.959 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.959 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.959 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.960 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.960 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.960 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.960 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.960 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.960 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.960 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.961 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.961 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.961 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.961 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.961 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.961 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.961 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.961 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.962 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1d0ea2ca:::obj-20:head", 2020-11-16T20:35:49.962 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'20", 2020-11-16T20:35:49.962 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.962 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4244.0:1", 2020-11-16T20:35:49.962 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.962 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:41.476676+0000", 2020-11-16T20:35:49.962 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.963 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.963 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.963 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.963 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.963 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.963 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.963 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.964 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.964 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.964 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.964 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.964 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.964 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.964 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.964 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.965 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.965 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.965 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:ab396690:::obj-21:head", 2020-11-16T20:35:49.965 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'21", 2020-11-16T20:35:49.965 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.965 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4247.0:1", 2020-11-16T20:35:49.965 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.966 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:41.636391+0000", 2020-11-16T20:35:49.966 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.966 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.966 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.966 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.966 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.966 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.966 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.967 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.967 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.967 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.967 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.967 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.967 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.967 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.968 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.968 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.968 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.968 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.968 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:254983db:::obj-22:head", 2020-11-16T20:35:49.968 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'22", 2020-11-16T20:35:49.968 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.968 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4250.0:1", 2020-11-16T20:35:49.969 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.969 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:41.793278+0000", 2020-11-16T20:35:49.969 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.969 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.969 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.969 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.969 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.970 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.970 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.970 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.970 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.970 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.970 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.970 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.971 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.971 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.971 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.971 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.971 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.971 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.971 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:e656299f:::obj-23:head", 2020-11-16T20:35:49.971 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'23", 2020-11-16T20:35:49.972 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.972 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4253.0:1", 2020-11-16T20:35:49.972 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.972 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:42.016141+0000", 2020-11-16T20:35:49.972 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.972 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.972 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.973 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.973 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.973 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.973 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.973 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.973 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.973 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.973 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.974 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.974 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.974 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.974 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.974 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.974 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.974 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.975 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:5cdf412b:::obj-24:head", 2020-11-16T20:35:49.975 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'24", 2020-11-16T20:35:49.975 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.975 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4256.0:1", 2020-11-16T20:35:49.975 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.975 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:42.172690+0000", 2020-11-16T20:35:49.975 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.976 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.976 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.976 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.976 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.976 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.976 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.976 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.976 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.977 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.977 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.977 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.977 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.977 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.977 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.977 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.978 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.978 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.978 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:cacae143:::obj-25:head", 2020-11-16T20:35:49.978 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'25", 2020-11-16T20:35:49.978 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.978 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4259.0:1", 2020-11-16T20:35:49.978 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.978 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:42.341388+0000", 2020-11-16T20:35:49.979 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.979 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.979 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.979 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.979 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.979 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.979 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.980 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.980 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.980 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.980 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.980 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.980 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.980 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.980 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.981 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.981 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.981 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.981 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:34b5bff6:::obj-26:head", 2020-11-16T20:35:49.981 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'26", 2020-11-16T20:35:49.981 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.981 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4262.0:1", 2020-11-16T20:35:49.982 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.982 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:42.522492+0000", 2020-11-16T20:35:49.982 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.982 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.982 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.982 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.982 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.982 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.983 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.983 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.983 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.983 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.983 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.983 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.983 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.984 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.984 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.984 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.984 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.984 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.984 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:a354f8f6:::obj-27:head", 2020-11-16T20:35:49.984 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'27", 2020-11-16T20:35:49.984 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.985 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4265.0:1", 2020-11-16T20:35:49.985 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.985 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:42.708150+0000", 2020-11-16T20:35:49.985 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.985 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.985 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.985 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.986 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.986 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.986 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.986 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.986 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.986 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.986 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.986 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.987 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.987 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.987 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.987 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.987 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.987 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.987 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:7421db49:::obj-28:head", 2020-11-16T20:35:49.987 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'28", 2020-11-16T20:35:49.988 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.988 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4268.0:1", 2020-11-16T20:35:49.988 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.988 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:42.875686+0000", 2020-11-16T20:35:49.988 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.988 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.988 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.989 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.989 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.989 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.989 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.989 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.989 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.989 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.989 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.990 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.990 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.990 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.990 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.990 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.990 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.990 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.991 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:54bc79ba:::obj-29:head", 2020-11-16T20:35:49.991 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'29", 2020-11-16T20:35:49.991 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.991 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4271.0:1", 2020-11-16T20:35:49.991 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.991 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:43.077373+0000", 2020-11-16T20:35:49.991 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.991 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.992 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.992 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.992 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.992 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.992 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.992 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.992 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.993 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.993 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.993 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.993 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.993 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.993 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.993 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.993 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.994 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.994 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:01b15595:::obj-30:head", 2020-11-16T20:35:49.994 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'30", 2020-11-16T20:35:49.994 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.994 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4274.0:1", 2020-11-16T20:35:49.994 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.994 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:43.235100+0000", 2020-11-16T20:35:49.995 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.995 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.995 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.995 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.995 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.995 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.995 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.995 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.996 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.996 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.996 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.996 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.996 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.996 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.996 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.997 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.997 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:49.997 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:49.997 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:f9a0a8a2:::obj-31:head", 2020-11-16T20:35:49.997 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'31", 2020-11-16T20:35:49.997 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:49.997 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4277.0:1", 2020-11-16T20:35:49.997 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:49.998 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:43.392412+0000", 2020-11-16T20:35:49.998 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:49.998 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:49.998 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:49.998 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:49.998 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:49.998 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:49.999 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:49.999 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:49.999 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:49.999 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:49.999 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:49.999 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:49.999 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:49.999 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.000 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.000 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.000 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.000 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.000 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1bf2562f:::obj-32:head", 2020-11-16T20:35:50.000 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'32", 2020-11-16T20:35:50.000 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.001 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4280.0:1", 2020-11-16T20:35:50.001 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.001 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:43.544767+0000", 2020-11-16T20:35:50.001 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.001 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.001 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.001 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.001 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.002 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.002 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.002 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.002 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.002 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.002 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.002 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.003 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.003 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.003 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.003 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.003 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.003 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.003 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:b982ae22:::obj-33:head", 2020-11-16T20:35:50.003 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'33", 2020-11-16T20:35:50.004 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.004 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4283.0:1", 2020-11-16T20:35:50.004 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.004 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:43.747240+0000", 2020-11-16T20:35:50.004 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.004 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.004 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.005 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.005 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.005 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.005 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.005 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.005 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.005 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.005 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.006 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.006 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.006 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.006 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.006 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.006 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.006 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.007 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1ad9f98e:::obj-34:head", 2020-11-16T20:35:50.007 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'34", 2020-11-16T20:35:50.007 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.007 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4286.0:1", 2020-11-16T20:35:50.007 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.007 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:43.915540+0000", 2020-11-16T20:35:50.007 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.007 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.008 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.008 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.008 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.008 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.008 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.008 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.008 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.009 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.009 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.009 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.009 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.009 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.009 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.009 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.009 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.010 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.010 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:941f43ff:::obj-35:head", 2020-11-16T20:35:50.010 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'35", 2020-11-16T20:35:50.010 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.010 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4289.0:1", 2020-11-16T20:35:50.010 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.010 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:44.081107+0000", 2020-11-16T20:35:50.011 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.011 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.011 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.011 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.011 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.011 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.011 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.012 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.012 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.012 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.012 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.012 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.012 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.012 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.012 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.013 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.013 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.013 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.013 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:24ebabe9:::obj-36:head", 2020-11-16T20:35:50.013 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'36", 2020-11-16T20:35:50.013 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.013 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4292.0:1", 2020-11-16T20:35:50.014 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.014 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:44.298114+0000", 2020-11-16T20:35:50.014 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.014 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.014 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.014 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.014 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.015 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.015 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.015 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.015 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.015 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.015 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.015 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.015 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.016 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.016 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.016 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.016 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.016 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.016 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:5f3a5398:::obj-37:head", 2020-11-16T20:35:50.016 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'37", 2020-11-16T20:35:50.016 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.017 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4295.0:1", 2020-11-16T20:35:50.017 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.017 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:44.457797+0000", 2020-11-16T20:35:50.017 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.017 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.017 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.017 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.018 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.018 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.018 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.018 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.018 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.018 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.018 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.018 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.019 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.019 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.019 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.019 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.019 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.019 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.019 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:c96cb37e:::obj-38:head", 2020-11-16T20:35:50.020 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'38", 2020-11-16T20:35:50.020 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.020 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4298.0:1", 2020-11-16T20:35:50.020 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.020 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:44.617262+0000", 2020-11-16T20:35:50.020 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.020 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.021 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.021 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.021 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.021 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.021 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.021 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.021 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.021 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.022 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.022 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.022 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.022 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.022 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.022 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.022 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.023 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.023 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:69e36996:::obj-39:head", 2020-11-16T20:35:50.023 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'39", 2020-11-16T20:35:50.023 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.023 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4301.0:1", 2020-11-16T20:35:50.023 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.023 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:44.833018+0000", 2020-11-16T20:35:50.023 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.024 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.024 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.024 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.024 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.024 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.024 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.024 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.025 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.025 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.025 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.025 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.025 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.025 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.025 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.025 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.026 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.026 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.026 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:9f75773e:::obj-40:head", 2020-11-16T20:35:50.026 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'40", 2020-11-16T20:35:50.026 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.026 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4304.0:1", 2020-11-16T20:35:50.026 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.027 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:44.990494+0000", 2020-11-16T20:35:50.027 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.027 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.027 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.027 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.027 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.027 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.027 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.028 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.028 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.028 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.028 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.028 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.028 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.028 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.029 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.029 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.029 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.029 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.029 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:2c01cd48:::obj-41:head", 2020-11-16T20:35:50.029 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'41", 2020-11-16T20:35:50.029 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.029 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4307.0:1", 2020-11-16T20:35:50.030 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.030 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:45.142294+0000", 2020-11-16T20:35:50.030 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.030 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.030 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.030 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.030 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.030 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.031 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.031 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.031 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.031 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.031 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.031 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.031 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.032 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.032 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.032 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.032 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.032 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.032 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:5e676e3d:::obj-42:head", 2020-11-16T20:35:50.032 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'42", 2020-11-16T20:35:50.032 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.033 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4310.0:1", 2020-11-16T20:35:50.033 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.033 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:45.302689+0000", 2020-11-16T20:35:50.033 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.033 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.033 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.033 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.034 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.034 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.034 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.034 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.034 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.034 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.034 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.034 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.035 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.035 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.035 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.035 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.035 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.035 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.035 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1215f697:::obj-43:head", 2020-11-16T20:35:50.035 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'43", 2020-11-16T20:35:50.036 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.036 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4313.0:1", 2020-11-16T20:35:50.036 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.036 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:45.502654+0000", 2020-11-16T20:35:50.036 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.036 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.036 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.037 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.037 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.037 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.037 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.037 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.037 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.037 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.037 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.038 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.038 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.038 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.038 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.038 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.038 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.038 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.039 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:3a0aa357:::obj-44:head", 2020-11-16T20:35:50.039 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'44", 2020-11-16T20:35:50.039 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.039 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4316.0:1", 2020-11-16T20:35:50.039 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.039 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:45.671054+0000", 2020-11-16T20:35:50.039 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.040 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.040 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.040 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.040 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.040 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.040 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.040 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.040 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.041 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.041 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.041 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.041 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.041 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.041 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.041 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.041 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.042 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.042 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:6366950a:::obj-45:head", 2020-11-16T20:35:50.042 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'45", 2020-11-16T20:35:50.042 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.042 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4319.0:1", 2020-11-16T20:35:50.042 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.042 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:45.828202+0000", 2020-11-16T20:35:50.043 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.043 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.043 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.043 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.043 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.043 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.043 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.044 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.044 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.044 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.044 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.044 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.044 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.044 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.045 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.045 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.045 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.045 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.045 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:621cb2b3:::obj-46:head", 2020-11-16T20:35:50.045 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'46", 2020-11-16T20:35:50.045 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.045 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4322.0:1", 2020-11-16T20:35:50.046 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.046 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:46.055043+0000", 2020-11-16T20:35:50.046 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.046 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.046 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.046 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.046 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.046 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.047 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.047 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.047 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.047 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.047 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.047 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.047 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.048 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.048 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.048 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.048 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.048 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.048 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:d83ab3cd:::obj-47:head", 2020-11-16T20:35:50.048 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'47", 2020-11-16T20:35:50.049 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.049 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4325.0:1", 2020-11-16T20:35:50.049 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.049 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:46.223628+0000", 2020-11-16T20:35:50.049 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.049 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.049 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.049 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.050 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.050 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.050 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.050 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.050 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.050 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.050 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.051 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.051 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.051 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.051 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.051 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.051 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.051 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.051 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:3d8e6d4a:::obj-48:head", 2020-11-16T20:35:50.052 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'48", 2020-11-16T20:35:50.052 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.052 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4328.0:1", 2020-11-16T20:35:50.052 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.052 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:46.380346+0000", 2020-11-16T20:35:50.052 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.052 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.053 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.053 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.053 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.053 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.053 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.053 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.053 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.053 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.054 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.054 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.054 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.054 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.054 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.054 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.054 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.055 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.055 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:50b1e2f2:::obj-49:head", 2020-11-16T20:35:50.055 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'49", 2020-11-16T20:35:50.055 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.055 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4331.0:1", 2020-11-16T20:35:50.055 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.055 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:46.565666+0000", 2020-11-16T20:35:50.055 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.056 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.056 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.056 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.056 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.056 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.056 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.056 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.057 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.057 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.057 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.057 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.057 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.057 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.057 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.057 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.058 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.058 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.058 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:c75430e6:::obj-50:head", 2020-11-16T20:35:50.058 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'50", 2020-11-16T20:35:50.058 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.058 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4334.0:1", 2020-11-16T20:35:50.058 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.059 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:46.725264+0000", 2020-11-16T20:35:50.059 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.059 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.059 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.059 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.059 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.059 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.059 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.060 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.060 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.060 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.060 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.060 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.060 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.060 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.060 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.061 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.061 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.061 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.061 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:eaf3ca0a:::obj-51:head", 2020-11-16T20:35:50.061 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'51", 2020-11-16T20:35:50.061 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.061 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4337.0:1", 2020-11-16T20:35:50.062 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.062 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:46.884894+0000", 2020-11-16T20:35:50.062 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.062 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.062 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.062 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.062 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.062 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.063 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.063 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.063 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.063 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.063 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.063 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.063 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.064 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.064 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.064 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.064 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.064 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.064 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:86a15bea:::obj-52:head", 2020-11-16T20:35:50.064 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'52", 2020-11-16T20:35:50.064 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.065 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4340.0:1", 2020-11-16T20:35:50.065 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.065 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:47.040317+0000", 2020-11-16T20:35:50.065 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.065 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.065 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.065 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.065 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.066 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.066 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.066 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.066 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.066 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.066 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.066 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.067 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.067 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.067 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.067 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.067 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.067 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.067 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1df15dae:::obj-53:head", 2020-11-16T20:35:50.068 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'53", 2020-11-16T20:35:50.068 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.068 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4343.0:1", 2020-11-16T20:35:50.068 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.068 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:47.241674+0000", 2020-11-16T20:35:50.068 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.068 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.068 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.069 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.069 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.069 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.069 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.069 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.069 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.069 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.070 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.070 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.070 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.070 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.070 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.070 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.070 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.070 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.071 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:29304dc8:::obj-54:head", 2020-11-16T20:35:50.071 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'54", 2020-11-16T20:35:50.071 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.071 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4346.0:1", 2020-11-16T20:35:50.071 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.071 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:47.403183+0000", 2020-11-16T20:35:50.071 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.072 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.072 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.072 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.072 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.072 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.072 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.072 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.072 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.073 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.073 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.073 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.073 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.073 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.073 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.073 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.074 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.074 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.074 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:de02c26d:::obj-55:head", 2020-11-16T20:35:50.074 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'55", 2020-11-16T20:35:50.074 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.074 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4349.0:1", 2020-11-16T20:35:50.074 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.074 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:47.560735+0000", 2020-11-16T20:35:50.075 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.075 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.075 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.075 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.075 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.075 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.075 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.076 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.076 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.076 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.076 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.076 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.076 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.077 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.077 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.077 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.077 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.077 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.077 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:0d36fc08:::obj-56:head", 2020-11-16T20:35:50.077 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'56", 2020-11-16T20:35:50.078 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.078 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4352.0:1", 2020-11-16T20:35:50.078 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.078 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:47.763681+0000", 2020-11-16T20:35:50.078 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.078 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.078 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.079 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.079 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.079 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.079 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.079 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.079 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.079 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.080 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.080 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.080 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.080 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.080 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.080 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.080 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.081 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.081 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:2b2cbcba:::obj-57:head", 2020-11-16T20:35:50.081 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'57", 2020-11-16T20:35:50.081 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.081 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4355.0:1", 2020-11-16T20:35:50.081 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.081 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:47.998378+0000", 2020-11-16T20:35:50.081 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.082 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.082 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.082 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.082 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.082 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.082 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.082 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.083 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.083 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.083 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.083 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.083 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.083 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.083 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.084 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.084 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.084 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.084 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:838be76e:::obj-58:head", 2020-11-16T20:35:50.084 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'58", 2020-11-16T20:35:50.084 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.084 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4358.0:1", 2020-11-16T20:35:50.085 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.085 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:48.154007+0000", 2020-11-16T20:35:50.085 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.085 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.085 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.085 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.085 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.085 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.086 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.086 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.086 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.086 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.086 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.086 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.086 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.087 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.087 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.087 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.087 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.087 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.087 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:2c66fd6f:::obj-59:head", 2020-11-16T20:35:50.087 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'59", 2020-11-16T20:35:50.088 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.088 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4361.0:1", 2020-11-16T20:35:50.088 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.088 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:48.371924+0000", 2020-11-16T20:35:50.088 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.088 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.088 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.089 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.089 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.089 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.089 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.089 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.089 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.089 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.089 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.090 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.090 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.090 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.090 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.090 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.090 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.090 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.091 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:2b18273b:::obj-60:head", 2020-11-16T20:35:50.091 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'60", 2020-11-16T20:35:50.091 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.091 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4364.0:1", 2020-11-16T20:35:50.091 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.091 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:48.523177+0000", 2020-11-16T20:35:50.091 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.091 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.092 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.092 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.092 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.092 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.092 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.092 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.092 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.093 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.093 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.093 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.093 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.093 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.093 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.093 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.094 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.094 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.094 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:fcfa3cd1:::obj-61:head", 2020-11-16T20:35:50.094 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'61", 2020-11-16T20:35:50.094 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.094 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4367.0:1", 2020-11-16T20:35:50.094 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.095 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:48.681894+0000", 2020-11-16T20:35:50.095 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.095 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.095 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.095 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.095 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.095 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.096 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.096 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.096 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.096 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.096 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.096 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.096 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.096 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.097 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.097 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.097 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.097 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.097 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:16e51a68:::obj-62:head", 2020-11-16T20:35:50.097 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'62", 2020-11-16T20:35:50.097 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.098 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4370.0:1", 2020-11-16T20:35:50.098 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.098 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:48.837243+0000", 2020-11-16T20:35:50.098 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.098 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.098 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.098 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.098 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.099 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.099 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.099 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.099 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.099 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.099 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.099 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.100 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.100 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.100 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.100 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.100 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.100 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.100 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:c15678fc:::obj-63:head", 2020-11-16T20:35:50.101 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'63", 2020-11-16T20:35:50.101 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.101 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4373.0:1", 2020-11-16T20:35:50.101 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.101 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:49.026639+0000", 2020-11-16T20:35:50.101 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.101 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.102 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.102 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.102 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.102 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.102 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.102 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.102 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.103 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.103 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.103 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.103 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.103 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.103 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.103 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.103 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.104 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.104 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:e066252d:::obj-64:head", 2020-11-16T20:35:50.104 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'64", 2020-11-16T20:35:50.104 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.104 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4376.0:1", 2020-11-16T20:35:50.104 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.104 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:49.185220+0000", 2020-11-16T20:35:50.105 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.105 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.105 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.105 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.105 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.105 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.105 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.105 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.106 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.106 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.106 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.106 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.106 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.106 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.106 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.107 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.107 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.107 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.107 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:95a4ceaf:::obj-65:head", 2020-11-16T20:35:50.107 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'65", 2020-11-16T20:35:50.107 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.108 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4379.0:1", 2020-11-16T20:35:50.108 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.108 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:49.359636+0000", 2020-11-16T20:35:50.108 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.108 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.108 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.109 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.109 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.109 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.109 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.109 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.109 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.109 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.110 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.110 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.110 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.110 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.110 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.110 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.110 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.110 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.111 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:6e97b31a:::obj-66:head", 2020-11-16T20:35:50.111 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'66", 2020-11-16T20:35:50.111 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.111 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4382.0:1", 2020-11-16T20:35:50.111 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.111 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:49.552838+0000", 2020-11-16T20:35:50.111 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.112 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.112 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.112 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.112 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.112 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.112 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.112 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.112 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.113 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.113 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.113 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.113 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.113 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.113 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.113 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.114 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.114 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.114 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:ceed8b4e:::obj-67:head", 2020-11-16T20:35:50.114 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'67", 2020-11-16T20:35:50.114 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.114 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4385.0:1", 2020-11-16T20:35:50.114 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.115 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:49.713099+0000", 2020-11-16T20:35:50.115 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.115 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.115 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.115 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.115 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.115 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.115 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.116 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.116 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.116 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.116 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.116 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.116 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.116 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.117 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.117 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.117 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.117 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.117 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:05e2e722:::obj-68:head", 2020-11-16T20:35:50.117 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'68", 2020-11-16T20:35:50.117 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.117 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4388.0:1", 2020-11-16T20:35:50.118 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.118 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:49.871839+0000", 2020-11-16T20:35:50.118 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.118 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.118 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.118 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.118 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.119 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.119 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.119 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.119 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.119 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.119 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.119 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.119 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.120 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.120 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.120 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.120 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.120 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.120 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:c40f7de9:::obj-69:head", 2020-11-16T20:35:50.120 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'69", 2020-11-16T20:35:50.121 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.121 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4391.0:1", 2020-11-16T20:35:50.121 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.121 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:50.046775+0000", 2020-11-16T20:35:50.121 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.121 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.121 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.122 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.122 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.122 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.122 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.122 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.122 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.122 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.123 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.123 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.123 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.123 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.123 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.123 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.123 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.123 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.124 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:7bab3581:::obj-70:head", 2020-11-16T20:35:50.124 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'70", 2020-11-16T20:35:50.124 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.124 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4394.0:1", 2020-11-16T20:35:50.124 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.124 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:50.222420+0000", 2020-11-16T20:35:50.124 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.125 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.125 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.125 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.125 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.125 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.125 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.125 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.125 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.126 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.126 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.126 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.126 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.126 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.126 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.126 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.127 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.127 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.127 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:cb9edc4f:::obj-71:head", 2020-11-16T20:35:50.127 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'71", 2020-11-16T20:35:50.127 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.127 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4397.0:1", 2020-11-16T20:35:50.127 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.128 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:50.374205+0000", 2020-11-16T20:35:50.128 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.128 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.128 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.128 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.128 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.128 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.128 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.129 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.129 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.129 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.129 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.129 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.129 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.129 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.130 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.130 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.130 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.130 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.130 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:88ae729f:::obj-72:head", 2020-11-16T20:35:50.130 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'72", 2020-11-16T20:35:50.130 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.130 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4400.0:1", 2020-11-16T20:35:50.131 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.131 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:50.532456+0000", 2020-11-16T20:35:50.131 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.131 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.131 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.131 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.131 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.131 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.132 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.132 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.132 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.132 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.132 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.132 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.132 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.133 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.133 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.133 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.133 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.133 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.133 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:887b5de4:::obj-73:head", 2020-11-16T20:35:50.133 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'73", 2020-11-16T20:35:50.133 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.134 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4403.0:1", 2020-11-16T20:35:50.134 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.134 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:50.732907+0000", 2020-11-16T20:35:50.134 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.134 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.134 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.134 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.135 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.135 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.135 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.135 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.135 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.135 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.135 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.135 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.136 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.136 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.136 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.136 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.136 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.136 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.136 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:996efeaa:::obj-74:head", 2020-11-16T20:35:50.136 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'74", 2020-11-16T20:35:50.137 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.137 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4406.0:1", 2020-11-16T20:35:50.137 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.137 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:50.893959+0000", 2020-11-16T20:35:50.137 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.137 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.137 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.138 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.138 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.138 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.138 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.138 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.138 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.138 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.139 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.139 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.139 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.139 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.139 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.139 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.139 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.140 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.140 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:942222a0:::obj-75:head", 2020-11-16T20:35:50.140 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'75", 2020-11-16T20:35:50.140 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.140 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4409.0:1", 2020-11-16T20:35:50.140 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.140 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:51.053699+0000", 2020-11-16T20:35:50.141 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.141 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.141 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.141 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.141 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.141 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.141 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.142 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.142 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.142 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.142 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.142 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.142 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.142 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.143 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.143 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.143 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.143 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.143 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:35f0499f:::obj-76:head", 2020-11-16T20:35:50.143 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'76", 2020-11-16T20:35:50.143 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.143 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4412.0:1", 2020-11-16T20:35:50.144 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.144 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:51.218778+0000", 2020-11-16T20:35:50.144 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.144 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.144 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.144 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.144 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.145 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.145 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.145 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.145 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.145 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.145 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.145 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.145 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.146 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.146 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.146 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.146 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.146 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.146 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:15cd38e3:::obj-77:head", 2020-11-16T20:35:50.146 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'77", 2020-11-16T20:35:50.147 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.147 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4415.0:1", 2020-11-16T20:35:50.147 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.147 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:51.403482+0000", 2020-11-16T20:35:50.147 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.147 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.147 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.147 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.148 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.148 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.148 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.148 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.148 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.148 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.148 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.149 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.149 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.149 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.149 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.149 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.149 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.149 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.149 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:7adeb77b:::obj-78:head", 2020-11-16T20:35:50.150 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'78", 2020-11-16T20:35:50.150 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.150 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4418.0:1", 2020-11-16T20:35:50.150 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.150 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:51.553549+0000", 2020-11-16T20:35:50.150 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.150 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.151 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.151 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.151 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.151 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.151 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.151 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.151 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.152 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.152 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.152 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.152 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.152 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.152 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.152 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.152 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.153 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.153 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:c92176ea:::obj-79:head", 2020-11-16T20:35:50.153 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'79", 2020-11-16T20:35:50.153 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.153 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4421.0:1", 2020-11-16T20:35:50.153 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.153 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:51.711874+0000", 2020-11-16T20:35:50.154 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.154 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.154 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.154 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.154 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.154 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.154 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.154 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.155 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.155 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.155 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.155 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.155 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.155 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.155 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.156 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.156 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.156 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.156 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:14abac33:::obj-80:head", 2020-11-16T20:35:50.156 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'80", 2020-11-16T20:35:50.156 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.156 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4424.0:1", 2020-11-16T20:35:50.156 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.157 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:51.919756+0000", 2020-11-16T20:35:50.157 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.157 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.157 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.157 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.157 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.157 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.157 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.158 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.158 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.158 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.158 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.158 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.158 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.158 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.159 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.159 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.159 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.159 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.159 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:ea1005df:::obj-81:head", 2020-11-16T20:35:50.159 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'81", 2020-11-16T20:35:50.159 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.159 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4427.0:1", 2020-11-16T20:35:50.160 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.160 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:52.089025+0000", 2020-11-16T20:35:50.160 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.160 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.160 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.160 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.160 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.161 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.161 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.161 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.161 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.161 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.161 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.161 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.162 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.162 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.162 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.162 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.162 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.162 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.162 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:df18954d:::obj-82:head", 2020-11-16T20:35:50.162 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'82", 2020-11-16T20:35:50.163 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.163 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4430.0:1", 2020-11-16T20:35:50.163 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.163 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:52.254184+0000", 2020-11-16T20:35:50.163 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.163 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.163 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.164 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.164 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.164 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.164 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.164 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.164 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.164 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.165 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.165 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.165 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.165 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.165 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.165 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.165 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.165 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.166 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:cdd3834b:::obj-83:head", 2020-11-16T20:35:50.166 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'83", 2020-11-16T20:35:50.166 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.166 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4433.0:1", 2020-11-16T20:35:50.166 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.166 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:52.466061+0000", 2020-11-16T20:35:50.166 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.167 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.167 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.167 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.167 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.167 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.167 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.167 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.167 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.168 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.168 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.168 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.168 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.168 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.168 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.168 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.169 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.169 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.169 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:dfbf520a:::obj-84:head", 2020-11-16T20:35:50.169 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'84", 2020-11-16T20:35:50.169 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.169 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4436.0:1", 2020-11-16T20:35:50.169 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.169 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:52.622614+0000", 2020-11-16T20:35:50.170 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.170 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.170 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.170 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.170 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.170 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.170 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.171 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.171 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.171 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.171 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.171 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.171 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.171 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.171 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.172 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.172 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.172 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.172 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1dd3df2f:::obj-85:head", 2020-11-16T20:35:50.172 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'85", 2020-11-16T20:35:50.172 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.172 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4439.0:1", 2020-11-16T20:35:50.173 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.173 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:52.776663+0000", 2020-11-16T20:35:50.173 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.173 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.173 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.173 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.173 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.173 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.174 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.174 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.174 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.174 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.174 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.174 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.174 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.175 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.175 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.175 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.175 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.175 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.175 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:54554b77:::obj-86:head", 2020-11-16T20:35:50.175 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'86", 2020-11-16T20:35:50.176 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.176 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4442.0:1", 2020-11-16T20:35:50.176 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.176 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:53.037433+0000", 2020-11-16T20:35:50.176 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.176 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.176 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.176 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.177 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.177 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.177 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.177 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.177 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.177 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.177 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.177 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.178 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.178 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.178 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.178 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.178 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.178 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.178 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:43a48f98:::obj-87:head", 2020-11-16T20:35:50.179 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'87", 2020-11-16T20:35:50.179 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.179 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4445.0:1", 2020-11-16T20:35:50.179 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.179 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:53.201938+0000", 2020-11-16T20:35:50.179 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.179 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.180 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.180 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.180 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.180 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.180 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.180 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.180 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.181 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.181 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.181 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.181 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.181 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.181 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.181 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.181 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.182 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.182 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:3f7b3ecc:::obj-88:head", 2020-11-16T20:35:50.182 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'88", 2020-11-16T20:35:50.182 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.182 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4448.0:1", 2020-11-16T20:35:50.182 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.182 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:53.362852+0000", 2020-11-16T20:35:50.183 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.183 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.183 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.183 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.183 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.183 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.183 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.184 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.184 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.184 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.184 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.184 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.184 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.184 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.184 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.185 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.185 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.185 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.185 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:d25ef7d2:::obj-89:head", 2020-11-16T20:35:50.185 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'89", 2020-11-16T20:35:50.185 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.185 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4451.0:1", 2020-11-16T20:35:50.186 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.186 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:53.518678+0000", 2020-11-16T20:35:50.186 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.186 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.186 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.186 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.186 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.186 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.187 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.187 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.187 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.187 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.187 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.187 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.187 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.188 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.188 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.188 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.188 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.188 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.188 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:ea27d688:::obj-90:head", 2020-11-16T20:35:50.188 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'90", 2020-11-16T20:35:50.188 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.189 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4454.0:1", 2020-11-16T20:35:50.189 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.189 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:53.711316+0000", 2020-11-16T20:35:50.189 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.189 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.189 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.189 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.190 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.190 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.190 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.190 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.190 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.190 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.190 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.190 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.191 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.191 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.191 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.191 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.191 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.191 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.191 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:088f9d12:::obj-91:head", 2020-11-16T20:35:50.192 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'91", 2020-11-16T20:35:50.192 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.192 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4457.0:1", 2020-11-16T20:35:50.192 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.192 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:53.861321+0000", 2020-11-16T20:35:50.192 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.192 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.192 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.193 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.193 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.193 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.193 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.193 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.193 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.193 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.194 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.194 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.194 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.194 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.194 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.194 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.194 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.194 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.195 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:206f25a5:::obj-92:head", 2020-11-16T20:35:50.195 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'92", 2020-11-16T20:35:50.195 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.195 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4460.0:1", 2020-11-16T20:35:50.195 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.195 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:54.033749+0000", 2020-11-16T20:35:50.195 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.196 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.196 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.196 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.196 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.196 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.196 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.196 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.197 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.197 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.197 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.197 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.197 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.197 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.197 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.197 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.198 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.198 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.198 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:501cca61:::obj-93:head", 2020-11-16T20:35:50.198 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'93", 2020-11-16T20:35:50.198 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.198 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4463.0:1", 2020-11-16T20:35:50.198 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.199 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:54.234563+0000", 2020-11-16T20:35:50.199 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.199 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.199 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.199 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.199 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.199 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.199 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.200 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.200 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.200 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.200 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.200 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.200 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.200 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.200 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.201 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.201 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.201 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.201 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1ed468c0:::obj-94:head", 2020-11-16T20:35:50.201 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'94", 2020-11-16T20:35:50.201 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.201 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4466.0:1", 2020-11-16T20:35:50.202 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.202 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:54.389973+0000", 2020-11-16T20:35:50.202 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.202 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.202 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.202 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.202 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.203 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.203 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.203 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.203 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.203 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.203 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.203 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.203 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.204 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.204 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.204 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.204 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.204 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.204 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:e8cba918:::obj-95:head", 2020-11-16T20:35:50.204 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'95", 2020-11-16T20:35:50.204 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.205 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4469.0:1", 2020-11-16T20:35:50.205 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.205 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:54.548702+0000", 2020-11-16T20:35:50.205 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.205 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.205 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.205 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.206 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.206 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.206 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.206 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.206 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.206 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.206 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.206 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.207 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.207 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.207 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.207 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.207 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.207 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.207 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:403c3dfc:::obj-96:head", 2020-11-16T20:35:50.208 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'96", 2020-11-16T20:35:50.208 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.208 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4472.0:1", 2020-11-16T20:35:50.208 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.208 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:54.700618+0000", 2020-11-16T20:35:50.208 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.208 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.208 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.209 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.209 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.209 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.209 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.209 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.209 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.209 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.210 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.210 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.210 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.210 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.210 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.210 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.210 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.210 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.211 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:7c2d0cb1:::obj-97:head", 2020-11-16T20:35:50.211 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'97", 2020-11-16T20:35:50.211 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.211 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4475.0:1", 2020-11-16T20:35:50.211 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.211 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:54.901896+0000", 2020-11-16T20:35:50.211 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.212 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.212 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.212 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.212 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.212 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.212 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.212 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.212 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.213 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.213 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.213 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.213 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.213 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.213 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.213 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.213 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.214 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.214 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:6f3e26a7:::obj-98:head", 2020-11-16T20:35:50.214 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'98", 2020-11-16T20:35:50.214 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.214 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4478.0:1", 2020-11-16T20:35:50.214 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.214 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:55.061787+0000", 2020-11-16T20:35:50.215 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.215 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.215 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.215 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.215 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.215 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.215 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.216 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.216 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.216 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.216 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.216 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.216 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.216 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.216 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.217 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.217 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.217 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.217 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:a4b22c6a:::obj-99:head", 2020-11-16T20:35:50.217 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'99", 2020-11-16T20:35:50.217 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.217 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4481.0:1", 2020-11-16T20:35:50.217 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.218 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:55.211878+0000", 2020-11-16T20:35:50.218 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.218 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.218 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.218 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.218 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.218 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.219 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.219 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.219 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.219 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.219 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.219 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.219 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.219 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.220 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.220 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.220 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.220 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.220 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:25c09ea8:::obj-100:head", 2020-11-16T20:35:50.220 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'100", 2020-11-16T20:35:50.220 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.221 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4484.0:1", 2020-11-16T20:35:50.221 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.221 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:55.419943+0000", 2020-11-16T20:35:50.221 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.221 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.221 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.221 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.221 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.222 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.222 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.222 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.222 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.222 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.222 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.222 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.223 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.223 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.223 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.223 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.223 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.223 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.223 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:be8b0193:::obj-101:head", 2020-11-16T20:35:50.223 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'101", 2020-11-16T20:35:50.224 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.224 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4487.0:1", 2020-11-16T20:35:50.224 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.224 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:55.581241+0000", 2020-11-16T20:35:50.224 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.224 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.224 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.224 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.225 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.225 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.225 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.225 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.225 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.225 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.225 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.226 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.226 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.226 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.226 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.226 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.226 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.226 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.226 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:6aa5e219:::obj-102:head", 2020-11-16T20:35:50.227 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'102", 2020-11-16T20:35:50.227 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.227 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4490.0:1", 2020-11-16T20:35:50.227 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.227 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:55.739140+0000", 2020-11-16T20:35:50.227 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.227 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.228 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.228 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.228 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.228 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.228 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.228 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.228 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.228 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.229 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.229 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.229 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.229 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.229 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.229 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.229 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.229 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.230 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:ff8cd32f:::obj-103:head", 2020-11-16T20:35:50.230 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'103", 2020-11-16T20:35:50.230 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.230 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4493.0:1", 2020-11-16T20:35:50.230 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.230 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:55.957011+0000", 2020-11-16T20:35:50.230 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.231 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.231 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.231 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.231 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.231 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.231 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.231 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.232 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.232 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.232 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.232 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.232 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.232 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.232 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.232 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.233 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.233 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.233 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:fcaf5c1c:::obj-104:head", 2020-11-16T20:35:50.233 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'104", 2020-11-16T20:35:50.233 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.233 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4496.0:1", 2020-11-16T20:35:50.233 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.234 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:56.133357+0000", 2020-11-16T20:35:50.234 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.234 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.234 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.234 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.234 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.234 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.235 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.235 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.235 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.235 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.235 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.235 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.235 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.236 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.236 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.236 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.236 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.236 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.236 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:2111aa60:::obj-105:head", 2020-11-16T20:35:50.236 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'105", 2020-11-16T20:35:50.236 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.237 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4499.0:1", 2020-11-16T20:35:50.237 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.237 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:56.308873+0000", 2020-11-16T20:35:50.237 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.237 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.237 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.237 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.237 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.238 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.238 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.238 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.238 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.238 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.238 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.238 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.239 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.239 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.239 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.239 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.239 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.239 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.239 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:9fed4eb7:::obj-106:head", 2020-11-16T20:35:50.240 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'106", 2020-11-16T20:35:50.240 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.240 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4502.0:1", 2020-11-16T20:35:50.240 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.240 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:56.465816+0000", 2020-11-16T20:35:50.240 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.240 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.240 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.241 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.241 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.241 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.241 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.241 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.241 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.241 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.242 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.242 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.242 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.242 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.242 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.242 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.242 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.243 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.243 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:0b0c0bce:::obj-107:head", 2020-11-16T20:35:50.243 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'107", 2020-11-16T20:35:50.243 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.243 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4505.0:1", 2020-11-16T20:35:50.243 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.243 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:56.659445+0000", 2020-11-16T20:35:50.243 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.244 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.244 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.244 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.244 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.244 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.244 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.244 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.245 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.245 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.245 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.245 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.245 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.245 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.245 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.245 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.246 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.246 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.246 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:b7ce613a:::obj-108:head", 2020-11-16T20:35:50.246 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'108", 2020-11-16T20:35:50.246 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.246 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4508.0:1", 2020-11-16T20:35:50.246 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.247 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:56.816055+0000", 2020-11-16T20:35:50.247 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.247 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.247 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.247 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.247 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.247 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.248 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.248 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.248 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.248 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.248 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.248 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.248 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.248 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.249 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.249 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.249 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.249 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.249 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:4f32cd61:::obj-109:head", 2020-11-16T20:35:50.249 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'109", 2020-11-16T20:35:50.250 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.250 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4511.0:1", 2020-11-16T20:35:50.250 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.250 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:56.970520+0000", 2020-11-16T20:35:50.250 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.250 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.250 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.251 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.251 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.251 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.251 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.251 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.251 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.251 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.252 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.252 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.252 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.252 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.252 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.252 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.252 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.253 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.253 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:40f97f88:::obj-110:head", 2020-11-16T20:35:50.253 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'110", 2020-11-16T20:35:50.253 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.253 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4514.0:1", 2020-11-16T20:35:50.253 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.254 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:57.158569+0000", 2020-11-16T20:35:50.254 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.254 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.254 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.254 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.254 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.254 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.254 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.255 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.255 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.255 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.255 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.255 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.255 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.255 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.256 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.256 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.256 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.256 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.256 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:1d8e7a62:::obj-111:head", 2020-11-16T20:35:50.256 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'111", 2020-11-16T20:35:50.256 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.256 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4517.0:1", 2020-11-16T20:35:50.257 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.257 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:57.323192+0000", 2020-11-16T20:35:50.257 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.257 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.257 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.257 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.257 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.258 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.258 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.258 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.258 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.258 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.258 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.258 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.259 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.259 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.259 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.259 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.259 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.259 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.259 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:3a283967:::obj-112:head", 2020-11-16T20:35:50.259 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'112", 2020-11-16T20:35:50.260 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.260 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4520.0:1", 2020-11-16T20:35:50.260 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.260 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:57.487425+0000", 2020-11-16T20:35:50.260 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.260 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.260 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.260 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.261 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.261 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.261 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.261 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.261 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.261 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.261 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.262 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.262 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.262 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.262 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.262 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.262 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.262 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.263 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:2acdc9e7:::obj-113:head", 2020-11-16T20:35:50.263 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'113", 2020-11-16T20:35:50.263 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.263 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4523.0:1", 2020-11-16T20:35:50.263 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.263 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:57.639373+0000", 2020-11-16T20:35:50.263 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.263 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.264 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.264 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.264 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.264 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.264 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.264 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.264 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.264 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.265 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.265 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.265 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.265 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.265 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.265 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.265 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.266 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.266 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:7f18a9ad:::obj-114:head", 2020-11-16T20:35:50.266 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'114", 2020-11-16T20:35:50.266 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.266 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4526.0:1", 2020-11-16T20:35:50.266 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.266 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:57.839244+0000", 2020-11-16T20:35:50.266 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.267 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.267 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.267 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.267 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.267 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.267 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.267 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.268 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.268 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.268 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.268 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.268 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.268 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.268 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.268 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.269 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.269 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.269 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:852b8a34:::obj-115:head", 2020-11-16T20:35:50.269 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'115", 2020-11-16T20:35:50.269 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.269 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4529.0:1", 2020-11-16T20:35:50.269 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.270 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:58.016469+0000", 2020-11-16T20:35:50.270 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.270 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.270 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.270 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.270 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.270 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.270 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.271 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.271 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.271 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.271 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.271 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.271 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.271 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.271 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.272 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.272 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.272 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.272 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:589de4f7:::obj-116:head", 2020-11-16T20:35:50.272 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'116", 2020-11-16T20:35:50.272 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.272 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4532.0:1", 2020-11-16T20:35:50.273 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.273 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:58.211598+0000", 2020-11-16T20:35:50.273 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.273 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.273 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.273 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.273 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.274 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.274 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.274 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.274 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.274 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.274 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.274 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.275 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.275 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.275 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.275 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.275 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.275 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.275 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:14f2571c:::obj-117:head", 2020-11-16T20:35:50.275 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'117", 2020-11-16T20:35:50.276 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.276 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4535.0:1", 2020-11-16T20:35:50.276 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.276 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:58.426165+0000", 2020-11-16T20:35:50.276 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.276 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.276 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.276 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.277 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.277 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.277 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.277 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.277 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.277 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.277 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.278 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.278 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.278 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.278 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.278 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.278 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.278 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.278 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:dd36f0e5:::obj-118:head", 2020-11-16T20:35:50.279 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'118", 2020-11-16T20:35:50.279 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.279 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4538.0:1", 2020-11-16T20:35:50.279 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.279 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:58.582985+0000", 2020-11-16T20:35:50.279 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.279 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.280 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.280 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.280 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.280 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.280 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.280 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.280 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.280 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.281 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.281 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.281 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.281 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.281 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.281 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.281 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.282 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.282 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:0d7fa432:::obj-119:head", 2020-11-16T20:35:50.282 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'119", 2020-11-16T20:35:50.282 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.282 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4541.0:1", 2020-11-16T20:35:50.282 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.282 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:58.734814+0000", 2020-11-16T20:35:50.282 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.283 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.283 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.283 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.283 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.283 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.283 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.283 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.284 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.284 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.284 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.284 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.284 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.284 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.284 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.285 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.285 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.285 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.285 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:295ec31a:::obj-120:head", 2020-11-16T20:35:50.285 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'120", 2020-11-16T20:35:50.285 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.285 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4544.0:1", 2020-11-16T20:35:50.286 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.286 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:58.894415+0000", 2020-11-16T20:35:50.286 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.286 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.286 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.286 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.286 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.286 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.287 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.287 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.287 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.287 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.287 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.287 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.287 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.287 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.288 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.288 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.288 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.288 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.288 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:c2863561:::obj-121:head", 2020-11-16T20:35:50.288 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'121", 2020-11-16T20:35:50.288 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.289 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4547.0:1", 2020-11-16T20:35:50.289 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.289 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:59.078599+0000", 2020-11-16T20:35:50.289 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.289 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.289 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.289 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.289 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.290 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.290 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.290 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.290 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.290 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.290 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.290 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.291 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.291 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.291 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.291 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.291 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.291 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.291 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:0b943493:::obj-122:head", 2020-11-16T20:35:50.291 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'122", 2020-11-16T20:35:50.292 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.292 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4550.0:1", 2020-11-16T20:35:50.292 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.292 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:59.237981+0000", 2020-11-16T20:35:50.292 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.292 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.292 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.292 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.293 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.293 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.293 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.293 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.293 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.293 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.293 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.294 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.294 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.294 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.294 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.294 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.294 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.294 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.294 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:4cfecbf8:::obj-123:head", 2020-11-16T20:35:50.295 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'123", 2020-11-16T20:35:50.295 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.295 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4553.0:1", 2020-11-16T20:35:50.295 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.295 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:59.395957+0000", 2020-11-16T20:35:50.295 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.295 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.296 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.296 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.296 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.296 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.296 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.296 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.296 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.296 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.297 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.297 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.297 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.297 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.297 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.297 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.297 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.298 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.298 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:fd18e511:::obj-124:head", 2020-11-16T20:35:50.298 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'124", 2020-11-16T20:35:50.298 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.298 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4556.0:1", 2020-11-16T20:35:50.298 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.298 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:59.597734+0000", 2020-11-16T20:35:50.298 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.299 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.299 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.299 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.299 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.299 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.299 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.299 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.299 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.300 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.300 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.300 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.300 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.300 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.300 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.300 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.301 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.301 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.301 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:4f299322:::obj-125:head", 2020-11-16T20:35:50.301 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'125", 2020-11-16T20:35:50.301 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.301 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4559.0:1", 2020-11-16T20:35:50.301 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.302 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:59.756268+0000", 2020-11-16T20:35:50.302 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.302 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.302 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.302 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.302 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.302 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.302 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.303 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.303 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.303 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.303 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.303 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.303 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.303 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.304 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.304 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.304 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.304 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.304 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:302ecc7e:::obj-126:head", 2020-11-16T20:35:50.304 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'126", 2020-11-16T20:35:50.304 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.304 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4562.0:1", 2020-11-16T20:35:50.305 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.305 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:34:59.917616+0000", 2020-11-16T20:35:50.305 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.305 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.305 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.305 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.305 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.306 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.306 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.306 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.306 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.306 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.306 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.306 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.306 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.307 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.307 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.307 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.307 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.307 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.307 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:42feb657:::obj-127:head", 2020-11-16T20:35:50.307 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'127", 2020-11-16T20:35:50.308 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.308 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4565.0:1", 2020-11-16T20:35:50.308 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.308 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:00.111067+0000", 2020-11-16T20:35:50.308 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.308 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.308 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.309 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.309 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.309 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.309 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.309 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.309 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.309 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.309 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.310 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.310 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.310 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.310 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.310 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.310 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.310 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.311 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:5e6b9a81:::obj-128:head", 2020-11-16T20:35:50.311 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'128", 2020-11-16T20:35:50.311 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.311 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4568.0:1", 2020-11-16T20:35:50.311 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.311 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:00.305120+0000", 2020-11-16T20:35:50.311 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.311 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.312 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.312 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.312 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.312 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.312 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.312 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.312 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.312 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.313 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.313 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.313 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.313 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.313 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.313 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.313 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.314 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.314 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:0d242242:::obj-129:head", 2020-11-16T20:35:50.314 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'129", 2020-11-16T20:35:50.314 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.314 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4571.0:1", 2020-11-16T20:35:50.314 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.314 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:00.466559+0000", 2020-11-16T20:35:50.315 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.315 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.315 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.315 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.315 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.315 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.315 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.316 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.316 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.316 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.316 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.316 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.316 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.316 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.316 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.317 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.317 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.317 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.317 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:ed409294:::obj-130:head", 2020-11-16T20:35:50.317 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'130", 2020-11-16T20:35:50.317 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.317 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4574.0:1", 2020-11-16T20:35:50.317 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.318 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:00.624760+0000", 2020-11-16T20:35:50.318 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.318 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.318 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.318 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.318 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.318 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.319 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.319 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.319 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.319 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.319 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.319 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.319 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.319 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.320 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.320 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.320 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.320 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.320 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:720761ff:::obj-131:head", 2020-11-16T20:35:50.320 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'131", 2020-11-16T20:35:50.321 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.321 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4577.0:1", 2020-11-16T20:35:50.321 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.321 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:00.801869+0000", 2020-11-16T20:35:50.321 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.321 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.321 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.321 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.322 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.322 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.322 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.322 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.322 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.322 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.322 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.323 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.323 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.323 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.323 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.323 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.323 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.323 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.323 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:7715ee72:::obj-132:head", 2020-11-16T20:35:50.324 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'132", 2020-11-16T20:35:50.324 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.324 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4580.0:1", 2020-11-16T20:35:50.324 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.324 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:00.960844+0000", 2020-11-16T20:35:50.324 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.324 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.325 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.325 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.325 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.325 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.325 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.325 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.325 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.325 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.326 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.326 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.326 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.326 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.326 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.326 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.326 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.327 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.327 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:580938e3:::obj-133:head", 2020-11-16T20:35:50.327 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'133", 2020-11-16T20:35:50.327 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.327 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4583.0:1", 2020-11-16T20:35:50.327 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.327 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:01.121342+0000", 2020-11-16T20:35:50.327 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.328 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.328 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.328 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.328 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.328 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.328 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.328 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.329 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.329 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.329 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.329 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.329 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.329 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.329 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.329 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.330 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.330 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.330 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:05dc98b5:::obj-134:head", 2020-11-16T20:35:50.330 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'134", 2020-11-16T20:35:50.330 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.330 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4586.0:1", 2020-11-16T20:35:50.330 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.331 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:01.329766+0000", 2020-11-16T20:35:50.331 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.331 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.331 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.331 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.331 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.331 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.331 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.332 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.332 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.332 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.332 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.332 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.332 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.332 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.333 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.333 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.333 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.333 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.333 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:73221e51:::obj-135:head", 2020-11-16T20:35:50.333 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'135", 2020-11-16T20:35:50.333 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.333 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4589.0:1", 2020-11-16T20:35:50.334 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.334 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:01.491734+0000", 2020-11-16T20:35:50.334 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.334 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.334 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.334 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.334 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.335 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.335 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.335 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.335 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.335 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.335 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.335 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.335 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.336 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.336 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.336 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.336 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.336 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.336 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:3a97ede2:::obj-136:head", 2020-11-16T20:35:50.336 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'136", 2020-11-16T20:35:50.336 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.337 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4592.0:1", 2020-11-16T20:35:50.337 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.337 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:01.650821+0000", 2020-11-16T20:35:50.337 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.337 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.337 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.337 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.338 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.338 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.338 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.338 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.338 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.338 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.338 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.338 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.339 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.339 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.339 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.339 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.339 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.339 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.339 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:9e95a205:::obj-137:head", 2020-11-16T20:35:50.340 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'137", 2020-11-16T20:35:50.340 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.340 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4595.0:1", 2020-11-16T20:35:50.340 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.340 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:01.805293+0000", 2020-11-16T20:35:50.340 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.340 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.340 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.341 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.341 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.341 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.341 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.341 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.341 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.341 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.342 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.342 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.342 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.343 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.343 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.343 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.343 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.343 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.343 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:27f846bd:::obj-138:head", 2020-11-16T20:35:50.343 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'138", 2020-11-16T20:35:50.344 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.344 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4598.0:1", 2020-11-16T20:35:50.344 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.344 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:02.008119+0000", 2020-11-16T20:35:50.344 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.344 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.344 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.345 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.345 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.345 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.345 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.345 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.345 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.345 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.345 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.346 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.346 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.346 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.346 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.346 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.346 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.346 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.347 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:bb2f7605:::obj-139:head", 2020-11-16T20:35:50.347 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'139", 2020-11-16T20:35:50.347 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.347 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4601.0:1", 2020-11-16T20:35:50.347 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.347 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:02.213646+0000", 2020-11-16T20:35:50.347 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.347 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.348 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.348 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.348 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.348 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.348 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.348 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.348 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.349 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.349 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.349 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.349 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.349 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.349 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.349 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.350 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.350 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.350 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:27668d5f:::obj-140:head", 2020-11-16T20:35:50.350 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'140", 2020-11-16T20:35:50.350 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.350 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4604.0:1", 2020-11-16T20:35:50.350 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.350 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:02.367665+0000", 2020-11-16T20:35:50.351 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.351 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.351 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.351 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.351 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.351 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.351 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.352 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.352 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.352 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.352 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.352 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.352 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.352 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.352 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.353 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.353 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.353 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.353 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:0d70cb86:::obj-141:head", 2020-11-16T20:35:50.353 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'141", 2020-11-16T20:35:50.353 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.353 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4607.0:1", 2020-11-16T20:35:50.354 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.354 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:02.578069+0000", 2020-11-16T20:35:50.354 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.354 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.354 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.354 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.354 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.354 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.355 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.355 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.355 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.355 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.355 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.355 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.355 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.356 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.356 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.356 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.356 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.356 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.356 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:347b39b5:::obj-more-1:head", 2020-11-16T20:35:50.356 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'142", 2020-11-16T20:35:50.356 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.357 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4614.0:1", 2020-11-16T20:35:50.357 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.357 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:03.685544+0000", 2020-11-16T20:35:50.357 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.357 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.357 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.357 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.358 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.358 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.358 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.358 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.358 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.358 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.358 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.358 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.359 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.359 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.359 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.359 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.359 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.359 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.359 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:0c63cc63:::obj-more-2:head", 2020-11-16T20:35:50.360 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'143", 2020-11-16T20:35:50.360 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.360 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4617.0:1", 2020-11-16T20:35:50.360 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.360 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:05.832811+0000", 2020-11-16T20:35:50.360 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.360 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.360 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.361 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.361 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.361 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.361 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.361 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.361 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.361 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.362 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.362 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.362 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.362 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.362 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.362 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.362 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.362 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.363 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:3ce69b57:::obj-more-3:head", 2020-11-16T20:35:50.363 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'144", 2020-11-16T20:35:50.363 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.363 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4620.0:1", 2020-11-16T20:35:50.363 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.363 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:06.165018+0000", 2020-11-16T20:35:50.363 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.363 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.364 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.364 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.364 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.364 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.364 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.364 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.364 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.365 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.365 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.365 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.365 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.365 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.365 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.365 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.365 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.366 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.366 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:50476200:::obj-more-4:head", 2020-11-16T20:35:50.366 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'145", 2020-11-16T20:35:50.366 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.366 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4623.0:1", 2020-11-16T20:35:50.366 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.366 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:06.403001+0000", 2020-11-16T20:35:50.367 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.367 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.367 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.367 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.367 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.367 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.367 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.368 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.368 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.368 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.368 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.368 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.368 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.368 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.369 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.369 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.369 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.369 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.369 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:7a39cfd8:::obj-more-5:head", 2020-11-16T20:35:50.369 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'146", 2020-11-16T20:35:50.369 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.370 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4626.0:1", 2020-11-16T20:35:50.370 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.370 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:06.661660+0000", 2020-11-16T20:35:50.370 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.370 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.370 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.370 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.371 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.371 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.371 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.371 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.371 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.371 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.371 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.372 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.372 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.372 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.372 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.372 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.372 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.372 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.373 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:a7159f95:::obj-more-6:head", 2020-11-16T20:35:50.373 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'147", 2020-11-16T20:35:50.373 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.373 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4629.0:1", 2020-11-16T20:35:50.373 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.373 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:06.878496+0000", 2020-11-16T20:35:50.373 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.373 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.374 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.374 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.374 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.374 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.374 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.374 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.374 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.375 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.375 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.375 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.375 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.375 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.375 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.375 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.375 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.376 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.376 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:37f29f5f:::obj-more-7:head", 2020-11-16T20:35:50.376 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'148", 2020-11-16T20:35:50.376 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.376 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4632.0:1", 2020-11-16T20:35:50.376 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.376 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:07.173050+0000", 2020-11-16T20:35:50.377 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.377 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.377 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.377 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.377 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.377 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.377 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.377 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.378 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.378 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.378 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.378 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.378 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.378 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.378 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.379 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.379 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.379 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.379 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:4e4836c1:::obj-more-8:head", 2020-11-16T20:35:50.379 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'149", 2020-11-16T20:35:50.379 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.379 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4635.0:1", 2020-11-16T20:35:50.379 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.380 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:07.413392+0000", 2020-11-16T20:35:50.380 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.380 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.380 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.380 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.380 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.381 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.381 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.381 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.381 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.381 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.381 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.381 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.381 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.382 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.382 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.382 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.382 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.382 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.382 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:d56cad57:::obj-more-9:head", 2020-11-16T20:35:50.382 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'150", 2020-11-16T20:35:50.383 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.383 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4638.0:1", 2020-11-16T20:35:50.383 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.383 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:07.737856+0000", 2020-11-16T20:35:50.383 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.383 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.383 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.383 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.384 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.384 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.384 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.384 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.384 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.384 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.384 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.385 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.385 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.385 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.385 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.385 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.385 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.385 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.385 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:96ff4b77:::obj-more-10:head", 2020-11-16T20:35:50.386 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'151", 2020-11-16T20:35:50.386 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.386 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4641.0:1", 2020-11-16T20:35:50.386 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.386 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:07.993215+0000", 2020-11-16T20:35:50.386 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.386 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.387 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.387 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.387 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.387 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.387 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.387 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.387 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.387 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.388 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.388 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.388 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.388 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.388 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.388 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.388 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.388 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.389 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:b8937ca5:::obj-more-11:head", 2020-11-16T20:35:50.389 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'152", 2020-11-16T20:35:50.389 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.389 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4644.0:1", 2020-11-16T20:35:50.389 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.389 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:08.244829+0000", 2020-11-16T20:35:50.389 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.390 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.390 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.390 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.390 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.390 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.390 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.390 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.391 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.391 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.391 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.391 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.391 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.391 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.391 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.391 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.392 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.392 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.392 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:c949902e:::obj-more-12:head", 2020-11-16T20:35:50.392 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'153", 2020-11-16T20:35:50.392 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.392 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4647.0:1", 2020-11-16T20:35:50.392 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.392 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:08.606089+0000", 2020-11-16T20:35:50.393 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.393 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.393 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.393 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.393 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.393 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.393 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.394 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.394 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.394 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.394 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.394 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.394 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.394 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.395 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.395 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.395 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.395 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.395 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:4909985f:::obj-more-13:head", 2020-11-16T20:35:50.395 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'154", 2020-11-16T20:35:50.395 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.396 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4650.0:1", 2020-11-16T20:35:50.396 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.396 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:08.856329+0000", 2020-11-16T20:35:50.396 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.396 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.396 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.396 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.397 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.397 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.397 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.397 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.397 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.397 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.397 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.397 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.398 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.398 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.398 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.398 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.398 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.398 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.398 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:acf906ce:::obj-more-14:head", 2020-11-16T20:35:50.399 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'155", 2020-11-16T20:35:50.399 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.399 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4653.0:1", 2020-11-16T20:35:50.399 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.399 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:09.133995+0000", 2020-11-16T20:35:50.399 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.399 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.400 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.400 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.400 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.400 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.400 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.400 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.400 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.400 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.401 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.401 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.401 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.401 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.401 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.401 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.401 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.402 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.402 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:44d31ae5:::obj-more-15:head", 2020-11-16T20:35:50.402 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'156", 2020-11-16T20:35:50.402 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.402 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4656.0:1", 2020-11-16T20:35:50.402 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.402 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:09.378341+0000", 2020-11-16T20:35:50.403 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.403 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.403 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.403 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.403 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.403 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.403 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.403 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.404 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.404 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.404 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.404 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.404 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.404 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.404 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.405 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.405 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.405 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.405 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:54dfd3e0:::obj-more-16:head", 2020-11-16T20:35:50.405 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'157", 2020-11-16T20:35:50.405 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.405 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4659.0:1", 2020-11-16T20:35:50.405 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.406 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:09.661220+0000", 2020-11-16T20:35:50.406 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.406 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.406 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.406 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.406 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.406 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.407 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.407 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.407 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.407 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.407 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.407 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.407 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.407 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.408 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.408 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.408 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.408 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.408 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:6a7dcd50:::obj-more-17:head", 2020-11-16T20:35:50.408 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'158", 2020-11-16T20:35:50.408 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.409 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4662.0:1", 2020-11-16T20:35:50.409 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.409 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:09.959427+0000", 2020-11-16T20:35:50.409 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.409 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.409 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.409 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.409 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.410 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.410 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.410 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.410 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.410 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.410 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.411 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.411 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.411 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.411 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.411 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.411 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.411 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.412 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:fb20876f:::obj-more-18:head", 2020-11-16T20:35:50.412 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'159", 2020-11-16T20:35:50.412 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.412 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4665.0:1", 2020-11-16T20:35:50.412 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.412 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:10.244357+0000", 2020-11-16T20:35:50.412 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.412 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.413 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.413 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.413 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.413 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.413 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.413 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.413 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.414 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.414 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.414 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.414 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.414 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.414 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.414 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.414 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.415 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.415 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:ed63ae36:::obj-more-19:head", 2020-11-16T20:35:50.415 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'160", 2020-11-16T20:35:50.415 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.415 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4668.0:1", 2020-11-16T20:35:50.415 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.415 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:10.441069+0000", 2020-11-16T20:35:50.461 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.461 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.461 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.461 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.462 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.462 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.462 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.462 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.462 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.462 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.462 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.463 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.463 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.463 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.463 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.463 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.463 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.463 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "modify", 2020-11-16T20:35:50.464 INFO:tasks.workunit.client.0.smithi189.stdout: "object": "1:6a8a392a:::obj-more-20:head", 2020-11-16T20:35:50.464 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "36'161", 2020-11-16T20:35:50.464 INFO:tasks.workunit.client.0.smithi189.stdout: "prior_version": "0'0", 2020-11-16T20:35:50.464 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4671.0:1", 2020-11-16T20:35:50.464 INFO:tasks.workunit.client.0.smithi189.stdout: "extra_reqids": [], 2020-11-16T20:35:50.464 INFO:tasks.workunit.client.0.smithi189.stdout: "mtime": "2020-11-16T20:35:10.735592+0000", 2020-11-16T20:35:50.464 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": 0, 2020-11-16T20:35:50.465 INFO:tasks.workunit.client.0.smithi189.stdout: "mod_desc": { 2020-11-16T20:35:50.465 INFO:tasks.workunit.client.0.smithi189.stdout: "object_mod_desc": { 2020-11-16T20:35:50.465 INFO:tasks.workunit.client.0.smithi189.stdout: "can_local_rollback": false, 2020-11-16T20:35:50.465 INFO:tasks.workunit.client.0.smithi189.stdout: "rollback_info_completed": false, 2020-11-16T20:35:50.465 INFO:tasks.workunit.client.0.smithi189.stdout: "ops": [] 2020-11-16T20:35:50.465 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.465 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.465 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_regions": { 2020-11-16T20:35:50.466 INFO:tasks.workunit.client.0.smithi189.stdout: "object_clean_regions": { 2020-11-16T20:35:50.466 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_offsets": "[7914~18446744073709543701]", 2020-11-16T20:35:50.466 INFO:tasks.workunit.client.0.smithi189.stdout: "clean_omap": true, 2020-11-16T20:35:50.466 INFO:tasks.workunit.client.0.smithi189.stdout: "new_object": false 2020-11-16T20:35:50.466 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.466 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.466 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.467 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T20:35:50.467 INFO:tasks.workunit.client.0.smithi189.stdout: "dups": [ 2020-11-16T20:35:50.467 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.467 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4187.0:1", 2020-11-16T20:35:50.467 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "31'1", 2020-11-16T20:35:50.467 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "1", 2020-11-16T20:35:50.467 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:35:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4190.0:1", 2020-11-16T20:35:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'2", 2020-11-16T20:35:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "2", 2020-11-16T20:35:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:35:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4193.0:1", 2020-11-16T20:35:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'3", 2020-11-16T20:35:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "3", 2020-11-16T20:35:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:35:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4196.0:1", 2020-11-16T20:35:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'4", 2020-11-16T20:35:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "4", 2020-11-16T20:35:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:35:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4199.0:1", 2020-11-16T20:35:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'5", 2020-11-16T20:35:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "5", 2020-11-16T20:35:50.471 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:35:50.471 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.471 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.471 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4202.0:1", 2020-11-16T20:35:50.471 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'6", 2020-11-16T20:35:50.471 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "6", 2020-11-16T20:35:50.471 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:35:50.471 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.472 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.472 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4205.0:1", 2020-11-16T20:35:50.472 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'7", 2020-11-16T20:35:50.472 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "7", 2020-11-16T20:35:50.472 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:35:50.472 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.472 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.473 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4208.0:1", 2020-11-16T20:35:50.473 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'8", 2020-11-16T20:35:50.473 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "8", 2020-11-16T20:35:50.473 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:35:50.473 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.473 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.473 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4211.0:1", 2020-11-16T20:35:50.473 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'9", 2020-11-16T20:35:50.474 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "9", 2020-11-16T20:35:50.474 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:35:50.474 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.474 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T20:35:50.474 INFO:tasks.workunit.client.0.smithi189.stdout: "reqid": "client.4214.0:1", 2020-11-16T20:35:50.474 INFO:tasks.workunit.client.0.smithi189.stdout: "version": "32'10", 2020-11-16T20:35:50.474 INFO:tasks.workunit.client.0.smithi189.stdout: "user_version": "10", 2020-11-16T20:35:50.475 INFO:tasks.workunit.client.0.smithi189.stdout: "return_code": "0" 2020-11-16T20:35:50.475 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.475 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T20:35:50.475 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T20:35:50.475 INFO:tasks.workunit.client.0.smithi189.stdout: "pg_missing_t": { 2020-11-16T20:35:50.475 INFO:tasks.workunit.client.0.smithi189.stdout: "missing": [], 2020-11-16T20:35:50.475 INFO:tasks.workunit.client.0.smithi189.stdout: "may_include_deletes": true 2020-11-16T20:35:50.475 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T20:35:50.476 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T20:35:50.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-11-16T20:35:50.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=151 2020-11-16T20:35:50.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:85: _common_test: '[' 151 '!=' 151 ']' 2020-11-16T20:35:50.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-11-16T20:35:50.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=10 2020-11-16T20:35:50.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:90: _common_test: '[' 10 '!=' 10 ']' 2020-11-16T20:35:50.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-11-16T20:35:50.492 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after v 0'0 2020-11-16T20:35:50.493 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'141 2020-11-16T20:35:50.493 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'140 2020-11-16T20:35:50.493 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'139 2020-11-16T20:35:50.493 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'138 2020-11-16T20:35:50.493 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'137 2020-11-16T20:35:50.493 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'136 2020-11-16T20:35:50.494 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'135 2020-11-16T20:35:50.494 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'134 2020-11-16T20:35:50.494 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'133 2020-11-16T20:35:50.494 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'132 2020-11-16T20:35:50.494 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'131 2020-11-16T20:35:50.494 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'130 2020-11-16T20:35:50.495 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'129 2020-11-16T20:35:50.495 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'128 2020-11-16T20:35:50.495 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'127 2020-11-16T20:35:50.495 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'126 2020-11-16T20:35:50.495 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'125 2020-11-16T20:35:50.495 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'124 2020-11-16T20:35:50.495 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'123 2020-11-16T20:35:50.495 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'122 2020-11-16T20:35:50.496 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'121 2020-11-16T20:35:50.496 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'120 2020-11-16T20:35:50.496 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'119 2020-11-16T20:35:50.496 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'118 2020-11-16T20:35:50.496 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'117 2020-11-16T20:35:50.496 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'116 2020-11-16T20:35:50.496 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'115 2020-11-16T20:35:50.497 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'114 2020-11-16T20:35:50.497 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'113 2020-11-16T20:35:50.497 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'112 2020-11-16T20:35:50.497 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'111 2020-11-16T20:35:50.497 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'110 2020-11-16T20:35:50.497 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'109 2020-11-16T20:35:50.498 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'108 2020-11-16T20:35:50.498 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'107 2020-11-16T20:35:50.498 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'106 2020-11-16T20:35:50.498 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'105 2020-11-16T20:35:50.498 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'104 2020-11-16T20:35:50.498 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'103 2020-11-16T20:35:50.498 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'102 2020-11-16T20:35:50.499 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'101 2020-11-16T20:35:50.499 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'100 2020-11-16T20:35:50.499 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'99 2020-11-16T20:35:50.499 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'98 2020-11-16T20:35:50.499 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'97 2020-11-16T20:35:50.499 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'96 2020-11-16T20:35:50.499 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'95 2020-11-16T20:35:50.499 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'94 2020-11-16T20:35:50.500 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'93 2020-11-16T20:35:50.500 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'92 2020-11-16T20:35:50.500 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'91 2020-11-16T20:35:50.500 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'90 2020-11-16T20:35:50.500 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'89 2020-11-16T20:35:50.500 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'88 2020-11-16T20:35:50.500 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'87 2020-11-16T20:35:50.501 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'86 2020-11-16T20:35:50.501 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'85 2020-11-16T20:35:50.501 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'84 2020-11-16T20:35:50.501 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'83 2020-11-16T20:35:50.501 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'82 2020-11-16T20:35:50.501 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'81 2020-11-16T20:35:50.501 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'80 2020-11-16T20:35:50.501 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'79 2020-11-16T20:35:50.502 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'78 2020-11-16T20:35:50.502 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'77 2020-11-16T20:35:50.502 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'76 2020-11-16T20:35:50.502 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'75 2020-11-16T20:35:50.502 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'74 2020-11-16T20:35:50.502 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'73 2020-11-16T20:35:50.502 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'72 2020-11-16T20:35:50.503 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'71 2020-11-16T20:35:50.503 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'70 2020-11-16T20:35:50.503 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'69 2020-11-16T20:35:50.503 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'68 2020-11-16T20:35:50.503 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'67 2020-11-16T20:35:50.503 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'66 2020-11-16T20:35:50.503 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'65 2020-11-16T20:35:50.504 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'64 2020-11-16T20:35:50.504 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'63 2020-11-16T20:35:50.504 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'62 2020-11-16T20:35:50.504 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'61 2020-11-16T20:35:50.504 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'60 2020-11-16T20:35:50.504 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'59 2020-11-16T20:35:50.504 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'58 2020-11-16T20:35:50.504 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'57 2020-11-16T20:35:50.505 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'56 2020-11-16T20:35:50.505 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'55 2020-11-16T20:35:50.505 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'54 2020-11-16T20:35:50.505 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'53 2020-11-16T20:35:50.505 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'52 2020-11-16T20:35:50.505 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'51 2020-11-16T20:35:50.505 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'50 2020-11-16T20:35:50.506 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'49 2020-11-16T20:35:50.506 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'48 2020-11-16T20:35:50.506 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'47 2020-11-16T20:35:50.506 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'46 2020-11-16T20:35:50.506 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'45 2020-11-16T20:35:50.506 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'44 2020-11-16T20:35:50.506 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'43 2020-11-16T20:35:50.506 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'42 2020-11-16T20:35:50.507 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'41 2020-11-16T20:35:50.507 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'40 2020-11-16T20:35:50.507 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'39 2020-11-16T20:35:50.507 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'38 2020-11-16T20:35:50.507 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'37 2020-11-16T20:35:50.507 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'36 2020-11-16T20:35:50.507 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'35 2020-11-16T20:35:50.508 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'34 2020-11-16T20:35:50.508 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'33 2020-11-16T20:35:50.508 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'32 2020-11-16T20:35:50.508 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'31 2020-11-16T20:35:50.508 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'30 2020-11-16T20:35:50.508 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'29 2020-11-16T20:35:50.508 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'28 2020-11-16T20:35:50.508 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'27 2020-11-16T20:35:50.509 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'26 2020-11-16T20:35:50.509 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'25 2020-11-16T20:35:50.509 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'24 2020-11-16T20:35:50.509 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'23 2020-11-16T20:35:50.509 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'22 2020-11-16T20:35:50.509 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'21 2020-11-16T20:35:50.509 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'20 2020-11-16T20:35:50.510 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'19 2020-11-16T20:35:50.510 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'18 2020-11-16T20:35:50.510 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'17 2020-11-16T20:35:50.510 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'16 2020-11-16T20:35:50.510 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'15 2020-11-16T20:35:50.510 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'14 2020-11-16T20:35:50.510 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'13 2020-11-16T20:35:50.511 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'12 2020-11-16T20:35:50.511 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'11 2020-11-16T20:35:50.511 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'10 2020-11-16T20:35:50.511 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'9 2020-11-16T20:35:50.511 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'8 2020-11-16T20:35:50.511 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'7 2020-11-16T20:35:50.511 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'6 2020-11-16T20:35:50.511 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'5 2020-11-16T20:35:50.512 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'4 2020-11-16T20:35:50.512 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'3 2020-11-16T20:35:50.512 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 32'2 2020-11-16T20:35:50.512 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_after copy log version 31'1 2020-11-16T20:35:50.512 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.796+0000 7fe2d5edf700 20 copy_up_to/copy_after earliest_dup_version 0 2020-11-16T20:35:50.512 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after v 0'0 2020-11-16T20:35:50.512 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'141 2020-11-16T20:35:50.513 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'140 2020-11-16T20:35:50.513 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'139 2020-11-16T20:35:50.513 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'138 2020-11-16T20:35:50.513 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'137 2020-11-16T20:35:50.513 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'136 2020-11-16T20:35:50.513 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'135 2020-11-16T20:35:50.513 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'134 2020-11-16T20:35:50.513 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'133 2020-11-16T20:35:50.514 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'132 2020-11-16T20:35:50.514 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'131 2020-11-16T20:35:50.514 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'130 2020-11-16T20:35:50.514 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'129 2020-11-16T20:35:50.514 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'128 2020-11-16T20:35:50.514 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'127 2020-11-16T20:35:50.515 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'126 2020-11-16T20:35:50.515 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'125 2020-11-16T20:35:50.515 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'124 2020-11-16T20:35:50.515 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'123 2020-11-16T20:35:50.515 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'122 2020-11-16T20:35:50.515 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'121 2020-11-16T20:35:50.515 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'120 2020-11-16T20:35:50.516 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'119 2020-11-16T20:35:50.516 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'118 2020-11-16T20:35:50.516 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'117 2020-11-16T20:35:50.516 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'116 2020-11-16T20:35:50.516 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'115 2020-11-16T20:35:50.516 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'114 2020-11-16T20:35:50.516 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'113 2020-11-16T20:35:50.517 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'112 2020-11-16T20:35:50.517 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'111 2020-11-16T20:35:50.517 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'110 2020-11-16T20:35:50.517 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'109 2020-11-16T20:35:50.517 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'108 2020-11-16T20:35:50.517 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'107 2020-11-16T20:35:50.517 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'106 2020-11-16T20:35:50.517 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'105 2020-11-16T20:35:50.518 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'104 2020-11-16T20:35:50.518 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'103 2020-11-16T20:35:50.518 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'102 2020-11-16T20:35:50.518 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'101 2020-11-16T20:35:50.518 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'100 2020-11-16T20:35:50.518 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'99 2020-11-16T20:35:50.518 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'98 2020-11-16T20:35:50.519 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'97 2020-11-16T20:35:50.519 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'96 2020-11-16T20:35:50.519 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'95 2020-11-16T20:35:50.519 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'94 2020-11-16T20:35:50.519 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'93 2020-11-16T20:35:50.519 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'92 2020-11-16T20:35:50.519 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'91 2020-11-16T20:35:50.519 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'90 2020-11-16T20:35:50.520 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'89 2020-11-16T20:35:50.520 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'88 2020-11-16T20:35:50.520 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'87 2020-11-16T20:35:50.520 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'86 2020-11-16T20:35:50.520 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'85 2020-11-16T20:35:50.520 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'84 2020-11-16T20:35:50.520 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'83 2020-11-16T20:35:50.521 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'82 2020-11-16T20:35:50.521 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'81 2020-11-16T20:35:50.521 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'80 2020-11-16T20:35:50.521 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'79 2020-11-16T20:35:50.521 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'78 2020-11-16T20:35:50.521 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'77 2020-11-16T20:35:50.521 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'76 2020-11-16T20:35:50.521 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'75 2020-11-16T20:35:50.522 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'74 2020-11-16T20:35:50.522 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'73 2020-11-16T20:35:50.522 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'72 2020-11-16T20:35:50.522 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'71 2020-11-16T20:35:50.522 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'70 2020-11-16T20:35:50.522 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'69 2020-11-16T20:35:50.522 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'68 2020-11-16T20:35:50.523 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'67 2020-11-16T20:35:50.523 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'66 2020-11-16T20:35:50.523 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'65 2020-11-16T20:35:50.523 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'64 2020-11-16T20:35:50.523 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'63 2020-11-16T20:35:50.523 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'62 2020-11-16T20:35:50.523 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'61 2020-11-16T20:35:50.523 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'60 2020-11-16T20:35:50.524 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'59 2020-11-16T20:35:50.524 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'58 2020-11-16T20:35:50.524 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'57 2020-11-16T20:35:50.524 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'56 2020-11-16T20:35:50.524 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'55 2020-11-16T20:35:50.524 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'54 2020-11-16T20:35:50.524 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'53 2020-11-16T20:35:50.525 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'52 2020-11-16T20:35:50.525 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'51 2020-11-16T20:35:50.525 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'50 2020-11-16T20:35:50.525 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'49 2020-11-16T20:35:50.525 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'48 2020-11-16T20:35:50.525 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'47 2020-11-16T20:35:50.525 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'46 2020-11-16T20:35:50.525 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'45 2020-11-16T20:35:50.526 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'44 2020-11-16T20:35:50.526 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'43 2020-11-16T20:35:50.526 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'42 2020-11-16T20:35:50.526 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'41 2020-11-16T20:35:50.526 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'40 2020-11-16T20:35:50.526 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'39 2020-11-16T20:35:50.526 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'38 2020-11-16T20:35:50.527 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'37 2020-11-16T20:35:50.527 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'36 2020-11-16T20:35:50.527 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'35 2020-11-16T20:35:50.527 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'34 2020-11-16T20:35:50.527 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'33 2020-11-16T20:35:50.527 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'32 2020-11-16T20:35:50.527 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'31 2020-11-16T20:35:50.527 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'30 2020-11-16T20:35:50.528 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'29 2020-11-16T20:35:50.528 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'28 2020-11-16T20:35:50.528 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'27 2020-11-16T20:35:50.528 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'26 2020-11-16T20:35:50.528 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'25 2020-11-16T20:35:50.528 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'24 2020-11-16T20:35:50.528 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'23 2020-11-16T20:35:50.529 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'22 2020-11-16T20:35:50.529 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'21 2020-11-16T20:35:50.529 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'20 2020-11-16T20:35:50.529 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'19 2020-11-16T20:35:50.529 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'18 2020-11-16T20:35:50.529 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'17 2020-11-16T20:35:50.529 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'16 2020-11-16T20:35:50.529 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'15 2020-11-16T20:35:50.530 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'14 2020-11-16T20:35:50.530 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'13 2020-11-16T20:35:50.530 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'12 2020-11-16T20:35:50.530 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'11 2020-11-16T20:35:50.530 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'10 2020-11-16T20:35:50.530 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'9 2020-11-16T20:35:50.530 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'8 2020-11-16T20:35:50.531 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'7 2020-11-16T20:35:50.531 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'6 2020-11-16T20:35:50.531 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'5 2020-11-16T20:35:50.531 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'4 2020-11-16T20:35:50.531 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'3 2020-11-16T20:35:50.531 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'2 2020-11-16T20:35:50.531 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 31'1 2020-11-16T20:35:50.532 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_up_to/copy_after earliest_dup_version 0 2020-11-16T20:35:50.532 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after v 0'0 2020-11-16T20:35:50.532 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'141 2020-11-16T20:35:50.532 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'140 2020-11-16T20:35:50.532 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'139 2020-11-16T20:35:50.532 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'138 2020-11-16T20:35:50.532 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'137 2020-11-16T20:35:50.533 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'136 2020-11-16T20:35:50.533 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'135 2020-11-16T20:35:50.533 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'134 2020-11-16T20:35:50.533 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'133 2020-11-16T20:35:50.533 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'132 2020-11-16T20:35:50.533 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'131 2020-11-16T20:35:50.533 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'130 2020-11-16T20:35:50.533 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'129 2020-11-16T20:35:50.534 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'128 2020-11-16T20:35:50.534 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'127 2020-11-16T20:35:50.534 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'126 2020-11-16T20:35:50.534 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'125 2020-11-16T20:35:50.534 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'124 2020-11-16T20:35:50.534 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'123 2020-11-16T20:35:50.534 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'122 2020-11-16T20:35:50.535 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'121 2020-11-16T20:35:50.535 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'120 2020-11-16T20:35:50.535 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'119 2020-11-16T20:35:50.535 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'118 2020-11-16T20:35:50.535 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'117 2020-11-16T20:35:50.535 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'116 2020-11-16T20:35:50.535 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'115 2020-11-16T20:35:50.535 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'114 2020-11-16T20:35:50.536 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'113 2020-11-16T20:35:50.536 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'112 2020-11-16T20:35:50.536 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'111 2020-11-16T20:35:50.536 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'110 2020-11-16T20:35:50.536 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'109 2020-11-16T20:35:50.536 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'108 2020-11-16T20:35:50.536 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'107 2020-11-16T20:35:50.537 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'106 2020-11-16T20:35:50.537 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'105 2020-11-16T20:35:50.537 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'104 2020-11-16T20:35:50.537 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'103 2020-11-16T20:35:50.537 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'102 2020-11-16T20:35:50.537 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'101 2020-11-16T20:35:50.537 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'100 2020-11-16T20:35:50.537 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'99 2020-11-16T20:35:50.538 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'98 2020-11-16T20:35:50.538 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'97 2020-11-16T20:35:50.538 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'96 2020-11-16T20:35:50.538 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'95 2020-11-16T20:35:50.538 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'94 2020-11-16T20:35:50.538 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'93 2020-11-16T20:35:50.538 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'92 2020-11-16T20:35:50.539 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'91 2020-11-16T20:35:50.539 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'90 2020-11-16T20:35:50.539 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'89 2020-11-16T20:35:50.539 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'88 2020-11-16T20:35:50.539 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'87 2020-11-16T20:35:50.539 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'86 2020-11-16T20:35:50.539 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'85 2020-11-16T20:35:50.539 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'84 2020-11-16T20:35:50.540 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'83 2020-11-16T20:35:50.540 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'82 2020-11-16T20:35:50.540 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'81 2020-11-16T20:35:50.540 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'80 2020-11-16T20:35:50.540 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'79 2020-11-16T20:35:50.540 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'78 2020-11-16T20:35:50.540 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'77 2020-11-16T20:35:50.541 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'76 2020-11-16T20:35:50.541 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'75 2020-11-16T20:35:50.541 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'74 2020-11-16T20:35:50.541 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'73 2020-11-16T20:35:50.541 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'72 2020-11-16T20:35:50.541 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'71 2020-11-16T20:35:50.541 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'70 2020-11-16T20:35:50.541 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'69 2020-11-16T20:35:50.542 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'68 2020-11-16T20:35:50.542 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'67 2020-11-16T20:35:50.542 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'66 2020-11-16T20:35:50.542 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'65 2020-11-16T20:35:50.542 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'64 2020-11-16T20:35:50.542 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'63 2020-11-16T20:35:50.542 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'62 2020-11-16T20:35:50.543 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'61 2020-11-16T20:35:50.543 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'60 2020-11-16T20:35:50.543 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'59 2020-11-16T20:35:50.543 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'58 2020-11-16T20:35:50.543 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'57 2020-11-16T20:35:50.543 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'56 2020-11-16T20:35:50.543 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'55 2020-11-16T20:35:50.543 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'54 2020-11-16T20:35:50.544 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'53 2020-11-16T20:35:50.544 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'52 2020-11-16T20:35:50.544 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'51 2020-11-16T20:35:50.544 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'50 2020-11-16T20:35:50.544 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'49 2020-11-16T20:35:50.544 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'48 2020-11-16T20:35:50.544 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'47 2020-11-16T20:35:50.545 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'46 2020-11-16T20:35:50.545 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'45 2020-11-16T20:35:50.545 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'44 2020-11-16T20:35:50.545 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'43 2020-11-16T20:35:50.545 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'42 2020-11-16T20:35:50.545 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'41 2020-11-16T20:35:50.545 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'40 2020-11-16T20:35:50.545 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'39 2020-11-16T20:35:50.546 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'38 2020-11-16T20:35:50.546 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'37 2020-11-16T20:35:50.546 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'36 2020-11-16T20:35:50.546 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'35 2020-11-16T20:35:50.546 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'34 2020-11-16T20:35:50.546 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'33 2020-11-16T20:35:50.546 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'32 2020-11-16T20:35:50.547 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'31 2020-11-16T20:35:50.547 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'30 2020-11-16T20:35:50.547 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'29 2020-11-16T20:35:50.547 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'28 2020-11-16T20:35:50.547 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'27 2020-11-16T20:35:50.547 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'26 2020-11-16T20:35:50.547 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'25 2020-11-16T20:35:50.547 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'24 2020-11-16T20:35:50.548 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'23 2020-11-16T20:35:50.548 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'22 2020-11-16T20:35:50.548 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'21 2020-11-16T20:35:50.548 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'20 2020-11-16T20:35:50.548 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'19 2020-11-16T20:35:50.548 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'18 2020-11-16T20:35:50.548 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'17 2020-11-16T20:35:50.548 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'16 2020-11-16T20:35:50.549 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'15 2020-11-16T20:35:50.549 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'14 2020-11-16T20:35:50.551 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'13 2020-11-16T20:35:50.551 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'12 2020-11-16T20:35:50.551 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'11 2020-11-16T20:35:50.551 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'10 2020-11-16T20:35:50.551 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'9 2020-11-16T20:35:50.552 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'8 2020-11-16T20:35:50.552 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'7 2020-11-16T20:35:50.552 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'6 2020-11-16T20:35:50.552 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'5 2020-11-16T20:35:50.552 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'4 2020-11-16T20:35:50.552 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'3 2020-11-16T20:35:50.552 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 32'2 2020-11-16T20:35:50.553 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_after copy log version 31'1 2020-11-16T20:35:50.553 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:04.800+0000 7fe2d5edf700 20 copy_up_to/copy_after earliest_dup_version 0 2020-11-16T20:35:50.553 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_after v 36'161 2020-11-16T20:35:50.553 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after earliest_dup_version 0 2020-11-16T20:35:50.553 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup version 31'1 2020-11-16T20:35:50.553 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup version 32'2 2020-11-16T20:35:50.553 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup version 32'3 2020-11-16T20:35:50.554 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup version 32'4 2020-11-16T20:35:50.554 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup version 32'5 2020-11-16T20:35:50.554 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup version 32'6 2020-11-16T20:35:50.554 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup version 32'7 2020-11-16T20:35:50.554 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup version 32'8 2020-11-16T20:35:50.554 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup version 32'9 2020-11-16T20:35:50.554 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup version 32'10 2020-11-16T20:35:50.555 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'11 2020-11-16T20:35:50.555 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'12 2020-11-16T20:35:50.555 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'13 2020-11-16T20:35:50.555 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'14 2020-11-16T20:35:50.555 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'15 2020-11-16T20:35:50.555 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'16 2020-11-16T20:35:50.555 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'17 2020-11-16T20:35:50.555 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'18 2020-11-16T20:35:50.556 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'19 2020-11-16T20:35:50.556 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'20 2020-11-16T20:35:50.556 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'21 2020-11-16T20:35:50.556 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'22 2020-11-16T20:35:50.556 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'23 2020-11-16T20:35:50.556 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'24 2020-11-16T20:35:50.556 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'25 2020-11-16T20:35:50.556 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'26 2020-11-16T20:35:50.557 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'27 2020-11-16T20:35:50.557 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'28 2020-11-16T20:35:50.557 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'29 2020-11-16T20:35:50.557 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'30 2020-11-16T20:35:50.557 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'31 2020-11-16T20:35:50.557 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'32 2020-11-16T20:35:50.557 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'33 2020-11-16T20:35:50.558 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'34 2020-11-16T20:35:50.558 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'35 2020-11-16T20:35:50.558 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'36 2020-11-16T20:35:50.558 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'37 2020-11-16T20:35:50.558 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'38 2020-11-16T20:35:50.558 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'39 2020-11-16T20:35:50.558 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'40 2020-11-16T20:35:50.559 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'41 2020-11-16T20:35:50.559 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'42 2020-11-16T20:35:50.559 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'43 2020-11-16T20:35:50.559 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'44 2020-11-16T20:35:50.559 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'45 2020-11-16T20:35:50.559 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'46 2020-11-16T20:35:50.559 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'47 2020-11-16T20:35:50.559 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'48 2020-11-16T20:35:50.560 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'49 2020-11-16T20:35:50.560 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'50 2020-11-16T20:35:50.560 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'51 2020-11-16T20:35:50.560 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'52 2020-11-16T20:35:50.560 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'53 2020-11-16T20:35:50.560 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'54 2020-11-16T20:35:50.560 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'55 2020-11-16T20:35:50.561 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'56 2020-11-16T20:35:50.561 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'57 2020-11-16T20:35:50.561 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'58 2020-11-16T20:35:50.561 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'59 2020-11-16T20:35:50.561 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'60 2020-11-16T20:35:50.561 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'61 2020-11-16T20:35:50.561 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'62 2020-11-16T20:35:50.561 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'63 2020-11-16T20:35:50.562 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'64 2020-11-16T20:35:50.562 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'65 2020-11-16T20:35:50.562 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'66 2020-11-16T20:35:50.562 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'67 2020-11-16T20:35:50.562 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'68 2020-11-16T20:35:50.562 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'69 2020-11-16T20:35:50.562 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'70 2020-11-16T20:35:50.563 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'71 2020-11-16T20:35:50.563 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'72 2020-11-16T20:35:50.563 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'73 2020-11-16T20:35:50.563 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'74 2020-11-16T20:35:50.563 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'75 2020-11-16T20:35:50.563 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'76 2020-11-16T20:35:50.563 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'77 2020-11-16T20:35:50.563 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'78 2020-11-16T20:35:50.564 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'79 2020-11-16T20:35:50.564 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'80 2020-11-16T20:35:50.564 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'81 2020-11-16T20:35:50.564 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'82 2020-11-16T20:35:50.564 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'83 2020-11-16T20:35:50.564 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'84 2020-11-16T20:35:50.564 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'85 2020-11-16T20:35:50.565 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'86 2020-11-16T20:35:50.565 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'87 2020-11-16T20:35:50.565 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'88 2020-11-16T20:35:50.565 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'89 2020-11-16T20:35:50.565 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'90 2020-11-16T20:35:50.565 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'91 2020-11-16T20:35:50.565 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'92 2020-11-16T20:35:50.565 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'93 2020-11-16T20:35:50.566 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'94 2020-11-16T20:35:50.566 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'95 2020-11-16T20:35:50.566 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'96 2020-11-16T20:35:50.566 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'97 2020-11-16T20:35:50.566 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'98 2020-11-16T20:35:50.566 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'99 2020-11-16T20:35:50.566 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'100 2020-11-16T20:35:50.567 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'101 2020-11-16T20:35:50.567 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'102 2020-11-16T20:35:50.567 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'103 2020-11-16T20:35:50.567 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'104 2020-11-16T20:35:50.567 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'105 2020-11-16T20:35:50.567 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'106 2020-11-16T20:35:50.567 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'107 2020-11-16T20:35:50.567 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'108 2020-11-16T20:35:50.568 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'109 2020-11-16T20:35:50.568 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'110 2020-11-16T20:35:50.568 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'111 2020-11-16T20:35:50.568 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'112 2020-11-16T20:35:50.568 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'113 2020-11-16T20:35:50.568 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'114 2020-11-16T20:35:50.568 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'115 2020-11-16T20:35:50.569 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'116 2020-11-16T20:35:50.569 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'117 2020-11-16T20:35:50.569 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'118 2020-11-16T20:35:50.569 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'119 2020-11-16T20:35:50.569 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'120 2020-11-16T20:35:50.569 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'121 2020-11-16T20:35:50.569 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'122 2020-11-16T20:35:50.569 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'123 2020-11-16T20:35:50.570 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'124 2020-11-16T20:35:50.570 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'125 2020-11-16T20:35:50.570 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'126 2020-11-16T20:35:50.570 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'127 2020-11-16T20:35:50.570 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'128 2020-11-16T20:35:50.570 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'129 2020-11-16T20:35:50.570 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'130 2020-11-16T20:35:50.571 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'131 2020-11-16T20:35:50.571 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'132 2020-11-16T20:35:50.571 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'133 2020-11-16T20:35:50.571 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'134 2020-11-16T20:35:50.571 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'135 2020-11-16T20:35:50.571 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'136 2020-11-16T20:35:50.571 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'137 2020-11-16T20:35:50.571 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'138 2020-11-16T20:35:50.572 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'139 2020-11-16T20:35:50.572 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'140 2020-11-16T20:35:50.572 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 32'141 2020-11-16T20:35:50.572 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'142 2020-11-16T20:35:50.572 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'143 2020-11-16T20:35:50.572 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'144 2020-11-16T20:35:50.572 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'145 2020-11-16T20:35:50.573 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'146 2020-11-16T20:35:50.573 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'147 2020-11-16T20:35:50.573 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'148 2020-11-16T20:35:50.573 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'149 2020-11-16T20:35:50.573 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'150 2020-11-16T20:35:50.573 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'151 2020-11-16T20:35:50.573 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'152 2020-11-16T20:35:50.573 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'153 2020-11-16T20:35:50.574 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'154 2020-11-16T20:35:50.574 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'155 2020-11-16T20:35:50.574 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'156 2020-11-16T20:35:50.574 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'157 2020-11-16T20:35:50.574 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'158 2020-11-16T20:35:50.574 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'159 2020-11-16T20:35:50.574 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'160 2020-11-16T20:35:50.574 INFO:tasks.workunit.client.0.smithi189.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-11-16T20:35:40.492+0000 7fe2d5edf700 20 copy_up_to/copy_after copy dup from log version 36'161 2020-11-16T20:35:50.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-11-16T20:35:50.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2020-11-16T20:35:50.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-11-16T20:35:50.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:35:50.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:35:50.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-11-16T20:35:50.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:35:50.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:35:50.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:35:50.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:35:50.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:35:50.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:35:50.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:35:50.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:35:50.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:35:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:35:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:35:50.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:35:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:35:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:35:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:35:50.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:35:50.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:35:50.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:35:50.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-11-16T20:35:50.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:35:50.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:35:50.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:35:50.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90694 2020-11-16T20:35:50.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:35:50.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:35:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T20:35:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-backfill-recovery-log 0 2020-11-16T20:35:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-11-16T20:35:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T20:35:50.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-11-16T20:35:50.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:35:50.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:35:50.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:35:50.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:35:50.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:35:50.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:35:50.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:35:50.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:35:50.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:35:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:35:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:35:50.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:35:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:35:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:35:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:35:50.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:35:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:35:50.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T20:35:50.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-11-16T20:35:50.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:35:50.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:35:50.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90694 2020-11-16T20:35:50.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90694 2020-11-16T20:35:50.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:35:50.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:35:50.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T20:35:50.586 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T20:35:50.587 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T20:35:50.600 INFO:tasks.workunit:Running workunit osd/osd-backfill-space.sh... 2020-11-16T20:35:50.600 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-backfill-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh 2020-11-16T20:35:50.660 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T20:35:50.685 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T20:35:50.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:35:50.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T20:35:50.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T20:35:50.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T20:35:50.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T20:35:50.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T20:35:50.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T20:35:50.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T20:35:50.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-backfill-space 2020-11-16T20:35:50.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:21: run: local dir=td/osd-backfill-space 2020-11-16T20:35:50.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:22: run: shift 2020-11-16T20:35:50.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:24: run: export CEPH_MON=127.0.0.1:7180 2020-11-16T20:35:50.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:24: run: CEPH_MON=127.0.0.1:7180 2020-11-16T20:35:50.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:25: run: export CEPH_ARGS 2020-11-16T20:35:50.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:26: run: uuidgen 2020-11-16T20:35:50.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:26: run: CEPH_ARGS+='--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none ' 2020-11-16T20:35:50.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7180 ' 2020-11-16T20:35:50.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:28: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T20:35:50.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:29: run: CEPH_ARGS+='--fake_statfs_for_testing=3686400 ' 2020-11-16T20:35:50.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:30: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2020-11-16T20:35:50.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:31: run: export objects=600 2020-11-16T20:35:50.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:31: run: objects=600 2020-11-16T20:35:50.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:32: run: export poolprefix=test 2020-11-16T20:35:50.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:32: run: poolprefix=test 2020-11-16T20:35:50.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:34: run: set 2020-11-16T20:35:50.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T20:35:50.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:34: run: local 'funcs=TEST_backfill_grow 2020-11-16T20:35:50.689 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_multi_partial 2020-11-16T20:35:50.689 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_test_multi 2020-11-16T20:35:50.689 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_test_sametarget 2020-11-16T20:35:50.689 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_test_simple 2020-11-16T20:35:50.689 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_ec_backfill_grow 2020-11-16T20:35:50.690 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_ec_backfill_multi 2020-11-16T20:35:50.690 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_ec_backfill_simple' 2020-11-16T20:35:50.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2020-11-16T20:35:50.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2020-11-16T20:35:50.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2020-11-16T20:35:50.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2020-11-16T20:35:50.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T20:35:50.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:35:50.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T20:35:50.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:35:50.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:35:50.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:35:50.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:35:50.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:35:50.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:35:50.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:35:50.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:35:50.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:35:50.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:35:50.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:35:50.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:35:50.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:35:50.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:35:50.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:35:50.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:35:50.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:35:50.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:35:50.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T20:35:50.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:35:50.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:35:50.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:35:50.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T20:35:50.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:35:50.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:35:50.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2020-11-16T20:35:50.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:35:50.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:35:50.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:35:50.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137323 2020-11-16T20:35:50.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:35:50.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T20:35:50.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T20:35:50.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:35:50.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2020-11-16T20:35:50.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_backfill_grow td/osd-backfill-space 2020-11-16T20:35:50.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:557: TEST_backfill_grow: local dir=td/osd-backfill-space 2020-11-16T20:35:50.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:558: TEST_backfill_grow: local poolname=test 2020-11-16T20:35:50.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:559: TEST_backfill_grow: local OSDS=3 2020-11-16T20:35:50.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:561: TEST_backfill_grow: run_mon td/osd-backfill-space a 2020-11-16T20:35:50.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2020-11-16T20:35:50.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:35:50.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:35:50.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:35:50.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2020-11-16T20:35:50.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2020-11-16T20:35:51.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:35:51.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:35:51.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:35:51.052 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:35:51.052 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:35:51.052 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:35:51.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:35:51.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:35:51.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:35:51.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:35:51.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:35:51.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:35:51.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:35:51.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:35:51.266 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:35:51.266 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:35:51.266 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:35:51.267 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:35:51.267 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:35:51.267 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:35:51.267 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T20:35:51.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:35:51.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get fsid 2020-11-16T20:35:51.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:35:51.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:35:51.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:35:51.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:35:51.376 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:35:51.376 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:35:51.376 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:35:51.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:35:51.377 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:35:51.377 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:35:51.377 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:35:51.377 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T20:35:51.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:35:51.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get mon_host 2020-11-16T20:35:51.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:562: TEST_backfill_grow: run_mgr td/osd-backfill-space x 2020-11-16T20:35:51.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2020-11-16T20:35:51.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:35:51.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:35:51.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:35:51.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2020-11-16T20:35:51.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:35:52.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:35:52.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:35:52.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:35:52.403 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:35:52.404 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:35:52.404 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:35:52.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:35:52.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:35:52.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:35:52.441 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:564: TEST_backfill_grow: expr 3 - 1 2020-11-16T20:35:52.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:564: TEST_backfill_grow: seq 0 2 2020-11-16T20:35:52.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:564: TEST_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:35:52.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:566: TEST_backfill_grow: run_osd td/osd-backfill-space 0 2020-11-16T20:35:52.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T20:35:52.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:35:52.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:35:52.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:35:52.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2020-11-16T20:35:52.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T20:35:52.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:35:52.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:35:52.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:35:52.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2020-11-16T20:35:52.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2020-11-16T20:35:52.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:35:52.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:35:52.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T20:35:52.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:35:52.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:35:52.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:35:52.448 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:35:52.448 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:35:52.448 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:35:52.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:35:52.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:35:52.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:35:52.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:35:52.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:35:52.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T20:35:52.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T20:35:52.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:35:52.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:35:52.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:35:52.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:35:52.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:35:52.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2020-11-16T20:35:52.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:35:52.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=63902c2d-77a1-4c5f-bcd4-ef7a61dc5208 2020-11-16T20:35:52.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 63902c2d-77a1-4c5f-bcd4-ef7a61dc5208' 2020-11-16T20:35:52.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:35:52.452 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 63902c2d-77a1-4c5f-bcd4-ef7a61dc5208 2020-11-16T20:35:52.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCo4rJfHoAEHBAAPo26RCpbKJ3Sa6wQ0WjsKQ== 2020-11-16T20:35:52.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCo4rJfHoAEHBAAPo26RCpbKJ3Sa6wQ0WjsKQ=="}' 2020-11-16T20:35:52.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 63902c2d-77a1-4c5f-bcd4-ef7a61dc5208 -i td/osd-backfill-space/0/new.json 2020-11-16T20:35:52.828 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:35:52.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2020-11-16T20:35:52.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCo4rJfHoAEHBAAPo26RCpbKJ3Sa6wQ0WjsKQ== --osd-uuid 63902c2d-77a1-4c5f-bcd4-ef7a61dc5208 2020-11-16T20:35:52.876 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:52.872+0000 7f819b8f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:35:52.879 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:52.876+0000 7f819b8f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:35:52.881 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:52.876+0000 7f819b8f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:35:52.882 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:52.876+0000 7f819b8f3d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T20:35:52.882 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:52.876+0000 7f819b8f3d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T20:35:52.882 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:52.876+0000 7f819b8f3d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T20:35:52.928 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:52.924+0000 7f819b8f3d80 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2020-11-16T20:35:57.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2020-11-16T20:35:57.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:35:57.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:35:57.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:35:57.379 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:35:57.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:35:57.879 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:35:57.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:35:57.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:35:57.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:35:57.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:35:57.909 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:57.904+0000 7f77ec4a4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:35:57.911 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:57.908+0000 7f77ec4a4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:35:57.913 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:57.908+0000 7f77ec4a4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:35:58.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:35:58.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:35:58.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:36:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:36:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:36:02.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:36:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:36:02.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:36:02.475 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:36:02.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:36:02.479 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:58.464+0000 7f77ec4a4d80 -1 Falling back to public interface 2020-11-16T20:36:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:36:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:35:59.308+0000 7f77ec4a4d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:36:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:36:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:36:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:36:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:36:02.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:36:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:36:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:36:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:36:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:564: TEST_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:36:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:566: TEST_backfill_grow: run_osd td/osd-backfill-space 1 2020-11-16T20:36:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T20:36:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:36:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:36:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:36:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2020-11-16T20:36:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T20:36:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:36:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:36:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:36:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2020-11-16T20:36:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2020-11-16T20:36:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:36:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:36:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T20:36:02.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:36:02.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:36:02.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:36:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:36:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:36:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:36:02.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:36:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:36:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:36:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:36:02.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:36:02.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T20:36:02.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T20:36:02.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:36:02.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:36:02.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:36:02.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:36:02.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:36:02.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2020-11-16T20:36:02.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:36:02.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6c9699c1-7545-47fb-bb4f-10ebf6c767cc 2020-11-16T20:36:02.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 6c9699c1-7545-47fb-bb4f-10ebf6c767cc' 2020-11-16T20:36:02.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:36:02.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCw4rJfeJOfBxAAHKzncRzXJeyMhVrsnpSVAA== 2020-11-16T20:36:02.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCw4rJfeJOfBxAAHKzncRzXJeyMhVrsnpSVAA=="}' 2020-11-16T20:36:02.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6c9699c1-7545-47fb-bb4f-10ebf6c767cc -i td/osd-backfill-space/1/new.json 2020-11-16T20:36:02.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2020-11-16T20:36:02.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCw4rJfeJOfBxAAHKzncRzXJeyMhVrsnpSVAA== --osd-uuid 6c9699c1-7545-47fb-bb4f-10ebf6c767cc 2020-11-16T20:36:02.487 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:00.748+0000 7f5313551d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:02.487 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:00.748+0000 7f5313551d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:02.487 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:00.752+0000 7f5313551d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:02.487 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:00.752+0000 7f5313551d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T20:36:02.488 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:00.752+0000 7f5313551d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T20:36:02.488 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:00.752+0000 7f5313551d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T20:36:02.488 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:00.868+0000 7f5313551d80 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2020-11-16T20:36:02.488 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:36:02.488 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/138179,v1:127.0.0.1:6803/138179] [v2:127.0.0.1:6804/138179,v1:127.0.0.1:6805/138179] exists,up 63902c2d-77a1-4c5f-bcd4-ef7a61dc5208 2020-11-16T20:36:02.488 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 6c9699c1-7545-47fb-bb4f-10ebf6c767cc 2020-11-16T20:36:02.489 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:36:05.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2020-11-16T20:36:05.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:36:05.369 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:36:05.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:36:05.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:36:05.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:36:05.870 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:36:05.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:36:05.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:36:05.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:36:05.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:36:05.903 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:05.900+0000 7f1f9f1c2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:05.906 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:05.904+0000 7f1f9f1c2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:05.907 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:05.904+0000 7f1f9f1c2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:06.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:36:06.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:36:06.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:36:06.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:36:06.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:36:06.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:36:06.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:36:06.265 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:36:06.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:36:06.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:36:06.453 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:06.448+0000 7f1f9f1c2d80 -1 Falling back to public interface 2020-11-16T20:36:06.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:36:07.378 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:07.368+0000 7f1f9f1c2d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:36:07.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:36:07.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:36:07.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:36:07.636 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:36:07.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:36:07.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:36:08.096 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/139173,v1:127.0.0.1:6811/139173] [v2:127.0.0.1:6812/139173,v1:127.0.0.1:6813/139173] exists,up 6c9699c1-7545-47fb-bb4f-10ebf6c767cc 2020-11-16T20:36:08.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:36:08.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:36:08.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:36:08.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:564: TEST_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:36:08.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:566: TEST_backfill_grow: run_osd td/osd-backfill-space 2 2020-11-16T20:36:08.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T20:36:08.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:36:08.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:36:08.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:36:08.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2020-11-16T20:36:08.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T20:36:08.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:36:08.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:36:08.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:36:08.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2020-11-16T20:36:08.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2020-11-16T20:36:08.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:36:08.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:36:08.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T20:36:08.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:36:08.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:36:08.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:36:08.101 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:36:08.102 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:36:08.102 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:36:08.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:36:08.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:36:08.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:36:08.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:36:08.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:36:08.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T20:36:08.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T20:36:08.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:36:08.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:36:08.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:36:08.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:36:08.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:36:08.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2020-11-16T20:36:08.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:36:08.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ce541578-978f-40f7-871b-0b94fbf520f4 2020-11-16T20:36:08.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 ce541578-978f-40f7-871b-0b94fbf520f4' 2020-11-16T20:36:08.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:36:08.107 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 ce541578-978f-40f7-871b-0b94fbf520f4 2020-11-16T20:36:08.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC44rJf3cBwBxAAqFqiNV10niwFdmIt+zxMSQ== 2020-11-16T20:36:08.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC44rJf3cBwBxAAqFqiNV10niwFdmIt+zxMSQ=="}' 2020-11-16T20:36:08.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ce541578-978f-40f7-871b-0b94fbf520f4 -i td/osd-backfill-space/2/new.json 2020-11-16T20:36:08.716 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:36:08.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2020-11-16T20:36:08.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC44rJf3cBwBxAAqFqiNV10niwFdmIt+zxMSQ== --osd-uuid ce541578-978f-40f7-871b-0b94fbf520f4 2020-11-16T20:36:08.756 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:08.752+0000 7f0c7377ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:08.758 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:08.756+0000 7f0c7377ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:08.760 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:08.756+0000 7f0c7377ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:08.761 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:08.756+0000 7f0c7377ed80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T20:36:08.761 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:08.756+0000 7f0c7377ed80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T20:36:08.761 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:08.756+0000 7f0c7377ed80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T20:36:08.815 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:08.812+0000 7f0c7377ed80 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2020-11-16T20:36:13.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2020-11-16T20:36:13.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:36:13.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:36:13.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:36:13.741 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:36:14.237 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:36:14.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:36:14.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:36:14.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:36:14.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:36:14.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:36:14.271 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:14.267+0000 7f45045e8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:14.273 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:14.267+0000 7f45045e8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:14.274 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:14.271+0000 7f45045e8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:36:14.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:36:14.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:36:14.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:36:14.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:36:14.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:36:14.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:36:14.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:36:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:36:14.635 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:36:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:36:14.824 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:14.819+0000 7f45045e8d80 -1 Falling back to public interface 2020-11-16T20:36:15.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:36:15.667 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:36:15.663+0000 7f45045e8d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:36:16.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:36:16.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:36:16.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:36:16.029 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:36:16.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:36:16.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:36:16.494 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/140167,v1:127.0.0.1:6819/140167] [v2:127.0.0.1:6820/140167,v1:127.0.0.1:6821/140167] exists,up ce541578-978f-40f7-871b-0b94fbf520f4 2020-11-16T20:36:16.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:36:16.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:36:16.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:36:16.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:569: TEST_backfill_grow: ceph osd set-backfillfull-ratio .85 2020-11-16T20:36:17.247 INFO:tasks.workunit.client.0.smithi189.stderr:osd set-backfillfull-ratio 0.85 2020-11-16T20:36:17.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:571: TEST_backfill_grow: create_pool test 1 1 2020-11-16T20:36:17.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T20:36:17.882 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T20:36:17.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:36:18.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:572: TEST_backfill_grow: ceph osd pool set test size 3 2020-11-16T20:36:19.547 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 3 2020-11-16T20:36:19.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:573: TEST_backfill_grow: sleep 5 2020-11-16T20:36:24.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:575: TEST_backfill_grow: wait_for_clean 2020-11-16T20:36:24.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:36:24.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:36:24.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:36:24.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:36:24.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:36:24.564 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:36:24.564 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:36:24.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:36:24.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:36:24.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:36:24.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:36:24.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:36:24.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:36:24.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:36:24.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:36:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:36:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:36:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:36:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:36:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:36:25.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:36:25.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2020-11-16T20:36:25.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2020-11-16T20:36:25.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2020-11-16T20:36:25.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:36:25.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:36:25.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672966 2020-11-16T20:36:25.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672966 2020-11-16T20:36:25.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2020-11-16T20:36:25.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:36:25.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:36:25.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2020-11-16T20:36:25.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2020-11-16T20:36:25.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-64424509444' 2020-11-16T20:36:25.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:36:25.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2020-11-16T20:36:25.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:36:25.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:36:25.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2020-11-16T20:36:25.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:36:25.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2020-11-16T20:36:25.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-11-16T20:36:25.823 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836487 2020-11-16T20:36:25.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:36:26.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836487 2020-11-16T20:36:26.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:36:26.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672966 2020-11-16T20:36:26.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:36:26.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:36:26.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2020-11-16T20:36:26.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:36:26.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672966 2020-11-16T20:36:26.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-11-16T20:36:26.234 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672966 2020-11-16T20:36:26.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:36:26.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672966 2020-11-16T20:36:26.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:36:26.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2020-11-16T20:36:26.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:36:26.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:36:26.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2020-11-16T20:36:26.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:36:26.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2020-11-16T20:36:26.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-11-16T20:36:26.618 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509444 2020-11-16T20:36:26.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:36:27.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2020-11-16T20:36:27.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:36:27.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:36:27.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:36:27.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:36:27.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:36:27.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:36:27.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:36:27.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:36:27.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:36:27.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:36:27.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:36:27.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:36:27.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:36:27.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:36:27.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:36:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:36:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:36:28.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:36:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:577: TEST_backfill_grow: dd if=/dev/urandom of=td/osd-backfill-space/4kdata bs=1k count=4 2020-11-16T20:36:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:4+0 records in 2020-11-16T20:36:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:4+0 records out 2020-11-16T20:36:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000227177 s, 18.0 MB/s 2020-11-16T20:36:28.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: seq 1 600 2020-11-16T20:36:28.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:28.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj1 td/osd-backfill-space/4kdata 2020-11-16T20:36:28.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:28.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj2 td/osd-backfill-space/4kdata 2020-11-16T20:36:28.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:28.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj3 td/osd-backfill-space/4kdata 2020-11-16T20:36:28.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:28.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj4 td/osd-backfill-space/4kdata 2020-11-16T20:36:29.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:29.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj5 td/osd-backfill-space/4kdata 2020-11-16T20:36:29.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:29.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj6 td/osd-backfill-space/4kdata 2020-11-16T20:36:29.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:29.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj7 td/osd-backfill-space/4kdata 2020-11-16T20:36:29.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:29.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj8 td/osd-backfill-space/4kdata 2020-11-16T20:36:29.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:29.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj9 td/osd-backfill-space/4kdata 2020-11-16T20:36:29.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:29.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj10 td/osd-backfill-space/4kdata 2020-11-16T20:36:30.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:30.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj11 td/osd-backfill-space/4kdata 2020-11-16T20:36:30.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:30.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj12 td/osd-backfill-space/4kdata 2020-11-16T20:36:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:30.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj13 td/osd-backfill-space/4kdata 2020-11-16T20:36:30.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:30.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj14 td/osd-backfill-space/4kdata 2020-11-16T20:36:30.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:30.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj15 td/osd-backfill-space/4kdata 2020-11-16T20:36:31.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:31.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj16 td/osd-backfill-space/4kdata 2020-11-16T20:36:31.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:31.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj17 td/osd-backfill-space/4kdata 2020-11-16T20:36:31.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:31.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj18 td/osd-backfill-space/4kdata 2020-11-16T20:36:31.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:31.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj19 td/osd-backfill-space/4kdata 2020-11-16T20:36:31.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:31.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj20 td/osd-backfill-space/4kdata 2020-11-16T20:36:31.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:31.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj21 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj22 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj23 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj24 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj25 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj26 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj27 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj28 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj29 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj30 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj31 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj32 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj33 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj34 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj35 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj36 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj37 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj38 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj39 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj40 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj41 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj42 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj43 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj44 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj45 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj46 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj47 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj48 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj49 td/osd-backfill-space/4kdata 2020-11-16T20:36:36.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:36.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj50 td/osd-backfill-space/4kdata 2020-11-16T20:36:37.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:37.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj51 td/osd-backfill-space/4kdata 2020-11-16T20:36:37.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:37.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj52 td/osd-backfill-space/4kdata 2020-11-16T20:36:37.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:37.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj53 td/osd-backfill-space/4kdata 2020-11-16T20:36:37.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:37.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj54 td/osd-backfill-space/4kdata 2020-11-16T20:36:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:37.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj55 td/osd-backfill-space/4kdata 2020-11-16T20:36:37.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:37.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj56 td/osd-backfill-space/4kdata 2020-11-16T20:36:38.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:38.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj57 td/osd-backfill-space/4kdata 2020-11-16T20:36:38.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:38.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj58 td/osd-backfill-space/4kdata 2020-11-16T20:36:38.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:38.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj59 td/osd-backfill-space/4kdata 2020-11-16T20:36:38.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:38.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj60 td/osd-backfill-space/4kdata 2020-11-16T20:36:38.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:38.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj61 td/osd-backfill-space/4kdata 2020-11-16T20:36:39.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:39.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj62 td/osd-backfill-space/4kdata 2020-11-16T20:36:39.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:39.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj63 td/osd-backfill-space/4kdata 2020-11-16T20:36:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj64 td/osd-backfill-space/4kdata 2020-11-16T20:36:39.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:39.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj65 td/osd-backfill-space/4kdata 2020-11-16T20:36:39.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:39.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj66 td/osd-backfill-space/4kdata 2020-11-16T20:36:39.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:39.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj67 td/osd-backfill-space/4kdata 2020-11-16T20:36:40.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:40.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj68 td/osd-backfill-space/4kdata 2020-11-16T20:36:40.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:40.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj69 td/osd-backfill-space/4kdata 2020-11-16T20:36:40.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:40.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj70 td/osd-backfill-space/4kdata 2020-11-16T20:36:40.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:40.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj71 td/osd-backfill-space/4kdata 2020-11-16T20:36:40.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:40.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj72 td/osd-backfill-space/4kdata 2020-11-16T20:36:41.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:41.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj73 td/osd-backfill-space/4kdata 2020-11-16T20:36:41.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:41.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj74 td/osd-backfill-space/4kdata 2020-11-16T20:36:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj75 td/osd-backfill-space/4kdata 2020-11-16T20:36:41.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:41.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj76 td/osd-backfill-space/4kdata 2020-11-16T20:36:41.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:41.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj77 td/osd-backfill-space/4kdata 2020-11-16T20:36:41.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:41.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj78 td/osd-backfill-space/4kdata 2020-11-16T20:36:42.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:42.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj79 td/osd-backfill-space/4kdata 2020-11-16T20:36:42.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:42.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj80 td/osd-backfill-space/4kdata 2020-11-16T20:36:42.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:42.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj81 td/osd-backfill-space/4kdata 2020-11-16T20:36:42.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:42.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj82 td/osd-backfill-space/4kdata 2020-11-16T20:36:42.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:42.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj83 td/osd-backfill-space/4kdata 2020-11-16T20:36:43.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:43.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj84 td/osd-backfill-space/4kdata 2020-11-16T20:36:43.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:43.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj85 td/osd-backfill-space/4kdata 2020-11-16T20:36:43.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:43.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj86 td/osd-backfill-space/4kdata 2020-11-16T20:36:43.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:43.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj87 td/osd-backfill-space/4kdata 2020-11-16T20:36:43.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:43.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj88 td/osd-backfill-space/4kdata 2020-11-16T20:36:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:43.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj89 td/osd-backfill-space/4kdata 2020-11-16T20:36:44.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:44.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj90 td/osd-backfill-space/4kdata 2020-11-16T20:36:44.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:44.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj91 td/osd-backfill-space/4kdata 2020-11-16T20:36:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj92 td/osd-backfill-space/4kdata 2020-11-16T20:36:44.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:44.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj93 td/osd-backfill-space/4kdata 2020-11-16T20:36:44.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj94 td/osd-backfill-space/4kdata 2020-11-16T20:36:44.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:44.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj95 td/osd-backfill-space/4kdata 2020-11-16T20:36:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj96 td/osd-backfill-space/4kdata 2020-11-16T20:36:45.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:45.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj97 td/osd-backfill-space/4kdata 2020-11-16T20:36:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj98 td/osd-backfill-space/4kdata 2020-11-16T20:36:45.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:45.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj99 td/osd-backfill-space/4kdata 2020-11-16T20:36:45.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:45.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj100 td/osd-backfill-space/4kdata 2020-11-16T20:36:46.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:46.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj101 td/osd-backfill-space/4kdata 2020-11-16T20:36:46.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:46.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj102 td/osd-backfill-space/4kdata 2020-11-16T20:36:46.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:46.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj103 td/osd-backfill-space/4kdata 2020-11-16T20:36:46.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:46.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj104 td/osd-backfill-space/4kdata 2020-11-16T20:36:46.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:46.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj105 td/osd-backfill-space/4kdata 2020-11-16T20:36:46.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:46.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj106 td/osd-backfill-space/4kdata 2020-11-16T20:36:47.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:47.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj107 td/osd-backfill-space/4kdata 2020-11-16T20:36:47.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:47.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj108 td/osd-backfill-space/4kdata 2020-11-16T20:36:47.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:47.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj109 td/osd-backfill-space/4kdata 2020-11-16T20:36:47.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:47.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj110 td/osd-backfill-space/4kdata 2020-11-16T20:36:47.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:47.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj111 td/osd-backfill-space/4kdata 2020-11-16T20:36:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:47.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj112 td/osd-backfill-space/4kdata 2020-11-16T20:36:48.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:48.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj113 td/osd-backfill-space/4kdata 2020-11-16T20:36:48.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:48.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj114 td/osd-backfill-space/4kdata 2020-11-16T20:36:48.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:48.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj115 td/osd-backfill-space/4kdata 2020-11-16T20:36:48.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:48.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj116 td/osd-backfill-space/4kdata 2020-11-16T20:36:48.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:48.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj117 td/osd-backfill-space/4kdata 2020-11-16T20:36:48.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:48.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj118 td/osd-backfill-space/4kdata 2020-11-16T20:36:49.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:49.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj119 td/osd-backfill-space/4kdata 2020-11-16T20:36:49.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:49.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj120 td/osd-backfill-space/4kdata 2020-11-16T20:36:49.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:49.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj121 td/osd-backfill-space/4kdata 2020-11-16T20:36:49.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:49.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj122 td/osd-backfill-space/4kdata 2020-11-16T20:36:49.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:49.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj123 td/osd-backfill-space/4kdata 2020-11-16T20:36:50.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:50.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj124 td/osd-backfill-space/4kdata 2020-11-16T20:36:50.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:50.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj125 td/osd-backfill-space/4kdata 2020-11-16T20:36:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj126 td/osd-backfill-space/4kdata 2020-11-16T20:36:50.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:50.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj127 td/osd-backfill-space/4kdata 2020-11-16T20:36:50.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:50.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj128 td/osd-backfill-space/4kdata 2020-11-16T20:36:50.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:50.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj129 td/osd-backfill-space/4kdata 2020-11-16T20:36:51.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:51.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj130 td/osd-backfill-space/4kdata 2020-11-16T20:36:51.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:51.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj131 td/osd-backfill-space/4kdata 2020-11-16T20:36:51.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:51.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj132 td/osd-backfill-space/4kdata 2020-11-16T20:36:51.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:51.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj133 td/osd-backfill-space/4kdata 2020-11-16T20:36:51.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:51.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj134 td/osd-backfill-space/4kdata 2020-11-16T20:36:51.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:51.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj135 td/osd-backfill-space/4kdata 2020-11-16T20:36:52.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:52.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj136 td/osd-backfill-space/4kdata 2020-11-16T20:36:52.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:52.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj137 td/osd-backfill-space/4kdata 2020-11-16T20:36:52.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:52.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj138 td/osd-backfill-space/4kdata 2020-11-16T20:36:52.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:52.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj139 td/osd-backfill-space/4kdata 2020-11-16T20:36:52.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:52.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj140 td/osd-backfill-space/4kdata 2020-11-16T20:36:53.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:53.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj141 td/osd-backfill-space/4kdata 2020-11-16T20:36:53.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:53.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj142 td/osd-backfill-space/4kdata 2020-11-16T20:36:53.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:53.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj143 td/osd-backfill-space/4kdata 2020-11-16T20:36:53.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:53.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj144 td/osd-backfill-space/4kdata 2020-11-16T20:36:53.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:53.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj145 td/osd-backfill-space/4kdata 2020-11-16T20:36:53.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:53.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj146 td/osd-backfill-space/4kdata 2020-11-16T20:36:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj147 td/osd-backfill-space/4kdata 2020-11-16T20:36:54.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:54.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj148 td/osd-backfill-space/4kdata 2020-11-16T20:36:54.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:54.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj149 td/osd-backfill-space/4kdata 2020-11-16T20:36:54.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:54.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj150 td/osd-backfill-space/4kdata 2020-11-16T20:36:54.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:54.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj151 td/osd-backfill-space/4kdata 2020-11-16T20:36:54.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:54.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj152 td/osd-backfill-space/4kdata 2020-11-16T20:36:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj153 td/osd-backfill-space/4kdata 2020-11-16T20:36:55.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:55.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj154 td/osd-backfill-space/4kdata 2020-11-16T20:36:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:55.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj155 td/osd-backfill-space/4kdata 2020-11-16T20:36:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:55.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj156 td/osd-backfill-space/4kdata 2020-11-16T20:36:55.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:55.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj157 td/osd-backfill-space/4kdata 2020-11-16T20:36:56.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:56.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj158 td/osd-backfill-space/4kdata 2020-11-16T20:36:56.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:56.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj159 td/osd-backfill-space/4kdata 2020-11-16T20:36:56.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:56.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj160 td/osd-backfill-space/4kdata 2020-11-16T20:36:56.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:56.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj161 td/osd-backfill-space/4kdata 2020-11-16T20:36:56.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:56.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj162 td/osd-backfill-space/4kdata 2020-11-16T20:36:57.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:57.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj163 td/osd-backfill-space/4kdata 2020-11-16T20:36:57.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:57.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj164 td/osd-backfill-space/4kdata 2020-11-16T20:36:57.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:36:57.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj165 td/osd-backfill-space/4kdata 2020-11-16T20:36:57.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj166 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj167 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj168 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj169 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj170 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj171 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj172 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj173 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj174 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj175 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj176 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj177 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj178 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj179 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj180 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj181 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj182 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj183 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj184 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj185 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj186 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj187 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj188 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj189 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj190 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj191 td/osd-backfill-space/4kdata 2020-11-16T20:37:01.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:01.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj192 td/osd-backfill-space/4kdata 2020-11-16T20:37:02.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:02.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj193 td/osd-backfill-space/4kdata 2020-11-16T20:37:02.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:02.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj194 td/osd-backfill-space/4kdata 2020-11-16T20:37:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:02.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj195 td/osd-backfill-space/4kdata 2020-11-16T20:37:02.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:02.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj196 td/osd-backfill-space/4kdata 2020-11-16T20:37:02.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:02.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj197 td/osd-backfill-space/4kdata 2020-11-16T20:37:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj198 td/osd-backfill-space/4kdata 2020-11-16T20:37:03.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:03.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj199 td/osd-backfill-space/4kdata 2020-11-16T20:37:03.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:03.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj200 td/osd-backfill-space/4kdata 2020-11-16T20:37:03.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:03.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj201 td/osd-backfill-space/4kdata 2020-11-16T20:37:03.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:03.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj202 td/osd-backfill-space/4kdata 2020-11-16T20:37:03.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:03.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj203 td/osd-backfill-space/4kdata 2020-11-16T20:37:04.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:04.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj204 td/osd-backfill-space/4kdata 2020-11-16T20:37:04.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:04.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj205 td/osd-backfill-space/4kdata 2020-11-16T20:37:04.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:04.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj206 td/osd-backfill-space/4kdata 2020-11-16T20:37:04.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:04.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj207 td/osd-backfill-space/4kdata 2020-11-16T20:37:04.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:04.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj208 td/osd-backfill-space/4kdata 2020-11-16T20:37:04.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:04.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj209 td/osd-backfill-space/4kdata 2020-11-16T20:37:05.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:05.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj210 td/osd-backfill-space/4kdata 2020-11-16T20:37:05.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:05.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj211 td/osd-backfill-space/4kdata 2020-11-16T20:37:05.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:05.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj212 td/osd-backfill-space/4kdata 2020-11-16T20:37:05.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:05.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj213 td/osd-backfill-space/4kdata 2020-11-16T20:37:05.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:05.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj214 td/osd-backfill-space/4kdata 2020-11-16T20:37:05.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:05.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj215 td/osd-backfill-space/4kdata 2020-11-16T20:37:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj216 td/osd-backfill-space/4kdata 2020-11-16T20:37:06.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:06.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj217 td/osd-backfill-space/4kdata 2020-11-16T20:37:06.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:06.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj218 td/osd-backfill-space/4kdata 2020-11-16T20:37:06.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:06.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj219 td/osd-backfill-space/4kdata 2020-11-16T20:37:06.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:06.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj220 td/osd-backfill-space/4kdata 2020-11-16T20:37:07.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:07.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj221 td/osd-backfill-space/4kdata 2020-11-16T20:37:07.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:07.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj222 td/osd-backfill-space/4kdata 2020-11-16T20:37:07.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:07.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj223 td/osd-backfill-space/4kdata 2020-11-16T20:37:07.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:07.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj224 td/osd-backfill-space/4kdata 2020-11-16T20:37:07.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:07.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj225 td/osd-backfill-space/4kdata 2020-11-16T20:37:07.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:07.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj226 td/osd-backfill-space/4kdata 2020-11-16T20:37:08.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:08.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj227 td/osd-backfill-space/4kdata 2020-11-16T20:37:08.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:08.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj228 td/osd-backfill-space/4kdata 2020-11-16T20:37:08.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:08.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj229 td/osd-backfill-space/4kdata 2020-11-16T20:37:08.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:08.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj230 td/osd-backfill-space/4kdata 2020-11-16T20:37:08.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:08.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj231 td/osd-backfill-space/4kdata 2020-11-16T20:37:08.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:08.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj232 td/osd-backfill-space/4kdata 2020-11-16T20:37:09.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:09.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj233 td/osd-backfill-space/4kdata 2020-11-16T20:37:09.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:09.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj234 td/osd-backfill-space/4kdata 2020-11-16T20:37:09.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:09.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj235 td/osd-backfill-space/4kdata 2020-11-16T20:37:09.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:09.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj236 td/osd-backfill-space/4kdata 2020-11-16T20:37:09.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:09.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj237 td/osd-backfill-space/4kdata 2020-11-16T20:37:09.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:09.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj238 td/osd-backfill-space/4kdata 2020-11-16T20:37:10.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:10.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj239 td/osd-backfill-space/4kdata 2020-11-16T20:37:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:10.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj240 td/osd-backfill-space/4kdata 2020-11-16T20:37:10.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:10.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj241 td/osd-backfill-space/4kdata 2020-11-16T20:37:10.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:10.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj242 td/osd-backfill-space/4kdata 2020-11-16T20:37:10.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:10.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj243 td/osd-backfill-space/4kdata 2020-11-16T20:37:11.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:11.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj244 td/osd-backfill-space/4kdata 2020-11-16T20:37:11.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:11.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj245 td/osd-backfill-space/4kdata 2020-11-16T20:37:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj246 td/osd-backfill-space/4kdata 2020-11-16T20:37:11.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:11.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj247 td/osd-backfill-space/4kdata 2020-11-16T20:37:11.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:11.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj248 td/osd-backfill-space/4kdata 2020-11-16T20:37:12.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:12.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj249 td/osd-backfill-space/4kdata 2020-11-16T20:37:12.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:12.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj250 td/osd-backfill-space/4kdata 2020-11-16T20:37:12.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:12.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj251 td/osd-backfill-space/4kdata 2020-11-16T20:37:12.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:12.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj252 td/osd-backfill-space/4kdata 2020-11-16T20:37:12.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:12.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj253 td/osd-backfill-space/4kdata 2020-11-16T20:37:12.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:12.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj254 td/osd-backfill-space/4kdata 2020-11-16T20:37:13.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:13.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj255 td/osd-backfill-space/4kdata 2020-11-16T20:37:13.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:13.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj256 td/osd-backfill-space/4kdata 2020-11-16T20:37:13.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:13.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj257 td/osd-backfill-space/4kdata 2020-11-16T20:37:13.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:13.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj258 td/osd-backfill-space/4kdata 2020-11-16T20:37:13.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:13.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj259 td/osd-backfill-space/4kdata 2020-11-16T20:37:13.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:13.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj260 td/osd-backfill-space/4kdata 2020-11-16T20:37:14.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:14.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj261 td/osd-backfill-space/4kdata 2020-11-16T20:37:14.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:14.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj262 td/osd-backfill-space/4kdata 2020-11-16T20:37:14.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:14.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj263 td/osd-backfill-space/4kdata 2020-11-16T20:37:14.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:14.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj264 td/osd-backfill-space/4kdata 2020-11-16T20:37:14.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:14.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj265 td/osd-backfill-space/4kdata 2020-11-16T20:37:15.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:15.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj266 td/osd-backfill-space/4kdata 2020-11-16T20:37:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj267 td/osd-backfill-space/4kdata 2020-11-16T20:37:15.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:15.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj268 td/osd-backfill-space/4kdata 2020-11-16T20:37:15.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:15.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj269 td/osd-backfill-space/4kdata 2020-11-16T20:37:15.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:15.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj270 td/osd-backfill-space/4kdata 2020-11-16T20:37:15.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:15.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj271 td/osd-backfill-space/4kdata 2020-11-16T20:37:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj272 td/osd-backfill-space/4kdata 2020-11-16T20:37:16.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:16.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj273 td/osd-backfill-space/4kdata 2020-11-16T20:37:16.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:16.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj274 td/osd-backfill-space/4kdata 2020-11-16T20:37:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj275 td/osd-backfill-space/4kdata 2020-11-16T20:37:16.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:16.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj276 td/osd-backfill-space/4kdata 2020-11-16T20:37:16.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:16.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj277 td/osd-backfill-space/4kdata 2020-11-16T20:37:17.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:17.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj278 td/osd-backfill-space/4kdata 2020-11-16T20:37:17.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:17.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj279 td/osd-backfill-space/4kdata 2020-11-16T20:37:17.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:17.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj280 td/osd-backfill-space/4kdata 2020-11-16T20:37:17.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:17.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj281 td/osd-backfill-space/4kdata 2020-11-16T20:37:17.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:17.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj282 td/osd-backfill-space/4kdata 2020-11-16T20:37:17.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:17.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj283 td/osd-backfill-space/4kdata 2020-11-16T20:37:18.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:18.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj284 td/osd-backfill-space/4kdata 2020-11-16T20:37:18.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:18.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj285 td/osd-backfill-space/4kdata 2020-11-16T20:37:18.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:18.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj286 td/osd-backfill-space/4kdata 2020-11-16T20:37:18.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj287 td/osd-backfill-space/4kdata 2020-11-16T20:37:18.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:18.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj288 td/osd-backfill-space/4kdata 2020-11-16T20:37:19.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:19.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj289 td/osd-backfill-space/4kdata 2020-11-16T20:37:19.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:19.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj290 td/osd-backfill-space/4kdata 2020-11-16T20:37:19.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:19.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj291 td/osd-backfill-space/4kdata 2020-11-16T20:37:19.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:19.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj292 td/osd-backfill-space/4kdata 2020-11-16T20:37:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:19.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj293 td/osd-backfill-space/4kdata 2020-11-16T20:37:19.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:19.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj294 td/osd-backfill-space/4kdata 2020-11-16T20:37:20.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:20.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj295 td/osd-backfill-space/4kdata 2020-11-16T20:37:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj296 td/osd-backfill-space/4kdata 2020-11-16T20:37:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj297 td/osd-backfill-space/4kdata 2020-11-16T20:37:20.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:20.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj298 td/osd-backfill-space/4kdata 2020-11-16T20:37:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj299 td/osd-backfill-space/4kdata 2020-11-16T20:37:20.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:20.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj300 td/osd-backfill-space/4kdata 2020-11-16T20:37:21.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:21.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj301 td/osd-backfill-space/4kdata 2020-11-16T20:37:21.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:21.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj302 td/osd-backfill-space/4kdata 2020-11-16T20:37:21.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:21.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj303 td/osd-backfill-space/4kdata 2020-11-16T20:37:21.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:21.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj304 td/osd-backfill-space/4kdata 2020-11-16T20:37:21.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:21.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj305 td/osd-backfill-space/4kdata 2020-11-16T20:37:22.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:22.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj306 td/osd-backfill-space/4kdata 2020-11-16T20:37:22.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:22.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj307 td/osd-backfill-space/4kdata 2020-11-16T20:37:22.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:22.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj308 td/osd-backfill-space/4kdata 2020-11-16T20:37:22.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:22.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj309 td/osd-backfill-space/4kdata 2020-11-16T20:37:22.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:22.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj310 td/osd-backfill-space/4kdata 2020-11-16T20:37:22.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:22.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj311 td/osd-backfill-space/4kdata 2020-11-16T20:37:23.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:23.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj312 td/osd-backfill-space/4kdata 2020-11-16T20:37:23.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:23.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj313 td/osd-backfill-space/4kdata 2020-11-16T20:37:23.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:23.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj314 td/osd-backfill-space/4kdata 2020-11-16T20:37:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj315 td/osd-backfill-space/4kdata 2020-11-16T20:37:23.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:23.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj316 td/osd-backfill-space/4kdata 2020-11-16T20:37:23.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:23.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj317 td/osd-backfill-space/4kdata 2020-11-16T20:37:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj318 td/osd-backfill-space/4kdata 2020-11-16T20:37:24.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:24.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj319 td/osd-backfill-space/4kdata 2020-11-16T20:37:24.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:24.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj320 td/osd-backfill-space/4kdata 2020-11-16T20:37:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj321 td/osd-backfill-space/4kdata 2020-11-16T20:37:24.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:24.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj322 td/osd-backfill-space/4kdata 2020-11-16T20:37:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj323 td/osd-backfill-space/4kdata 2020-11-16T20:37:25.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:25.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj324 td/osd-backfill-space/4kdata 2020-11-16T20:37:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj325 td/osd-backfill-space/4kdata 2020-11-16T20:37:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj326 td/osd-backfill-space/4kdata 2020-11-16T20:37:25.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:25.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj327 td/osd-backfill-space/4kdata 2020-11-16T20:37:25.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:25.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj328 td/osd-backfill-space/4kdata 2020-11-16T20:37:25.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:25.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj329 td/osd-backfill-space/4kdata 2020-11-16T20:37:26.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:26.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj330 td/osd-backfill-space/4kdata 2020-11-16T20:37:26.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:26.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj331 td/osd-backfill-space/4kdata 2020-11-16T20:37:26.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:26.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj332 td/osd-backfill-space/4kdata 2020-11-16T20:37:26.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:26.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj333 td/osd-backfill-space/4kdata 2020-11-16T20:37:27.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:27.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj334 td/osd-backfill-space/4kdata 2020-11-16T20:37:27.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:27.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj335 td/osd-backfill-space/4kdata 2020-11-16T20:37:27.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:27.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj336 td/osd-backfill-space/4kdata 2020-11-16T20:37:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj337 td/osd-backfill-space/4kdata 2020-11-16T20:37:27.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:27.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj338 td/osd-backfill-space/4kdata 2020-11-16T20:37:27.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:27.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj339 td/osd-backfill-space/4kdata 2020-11-16T20:37:28.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:28.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj340 td/osd-backfill-space/4kdata 2020-11-16T20:37:28.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:28.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj341 td/osd-backfill-space/4kdata 2020-11-16T20:37:28.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:28.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj342 td/osd-backfill-space/4kdata 2020-11-16T20:37:28.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:28.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj343 td/osd-backfill-space/4kdata 2020-11-16T20:37:28.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:28.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj344 td/osd-backfill-space/4kdata 2020-11-16T20:37:28.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:28.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj345 td/osd-backfill-space/4kdata 2020-11-16T20:37:29.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:29.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj346 td/osd-backfill-space/4kdata 2020-11-16T20:37:29.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:29.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj347 td/osd-backfill-space/4kdata 2020-11-16T20:37:29.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj348 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj349 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj350 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj351 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj352 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj353 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj354 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj355 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj356 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj357 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj358 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj359 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj360 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj361 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj362 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj363 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj364 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj365 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj366 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj367 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj368 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj369 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj370 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj371 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj372 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj373 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj374 td/osd-backfill-space/4kdata 2020-11-16T20:37:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:33.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj375 td/osd-backfill-space/4kdata 2020-11-16T20:37:34.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:34.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj376 td/osd-backfill-space/4kdata 2020-11-16T20:37:34.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj377 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj378 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj379 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj380 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj381 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj382 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj383 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj384 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj385 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj386 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj387 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj388 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj389 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj390 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj391 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj392 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj393 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj394 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj395 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj396 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj397 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj398 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj399 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj400 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj401 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj402 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj403 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj404 td/osd-backfill-space/4kdata 2020-11-16T20:37:38.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:38.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj405 td/osd-backfill-space/4kdata 2020-11-16T20:37:39.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:39.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj406 td/osd-backfill-space/4kdata 2020-11-16T20:37:39.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:39.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj407 td/osd-backfill-space/4kdata 2020-11-16T20:37:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:39.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj408 td/osd-backfill-space/4kdata 2020-11-16T20:37:39.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:39.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj409 td/osd-backfill-space/4kdata 2020-11-16T20:37:39.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:39.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj410 td/osd-backfill-space/4kdata 2020-11-16T20:37:39.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:39.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj411 td/osd-backfill-space/4kdata 2020-11-16T20:37:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj412 td/osd-backfill-space/4kdata 2020-11-16T20:37:40.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:40.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj413 td/osd-backfill-space/4kdata 2020-11-16T20:37:40.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:40.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj414 td/osd-backfill-space/4kdata 2020-11-16T20:37:40.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:40.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj415 td/osd-backfill-space/4kdata 2020-11-16T20:37:40.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:40.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj416 td/osd-backfill-space/4kdata 2020-11-16T20:37:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj417 td/osd-backfill-space/4kdata 2020-11-16T20:37:41.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:41.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj418 td/osd-backfill-space/4kdata 2020-11-16T20:37:41.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:41.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj419 td/osd-backfill-space/4kdata 2020-11-16T20:37:41.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:41.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj420 td/osd-backfill-space/4kdata 2020-11-16T20:37:41.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:41.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj421 td/osd-backfill-space/4kdata 2020-11-16T20:37:41.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:41.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj422 td/osd-backfill-space/4kdata 2020-11-16T20:37:41.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:41.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj423 td/osd-backfill-space/4kdata 2020-11-16T20:37:42.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:42.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj424 td/osd-backfill-space/4kdata 2020-11-16T20:37:42.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:42.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj425 td/osd-backfill-space/4kdata 2020-11-16T20:37:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj426 td/osd-backfill-space/4kdata 2020-11-16T20:37:42.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:42.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj427 td/osd-backfill-space/4kdata 2020-11-16T20:37:42.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:42.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj428 td/osd-backfill-space/4kdata 2020-11-16T20:37:42.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:42.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj429 td/osd-backfill-space/4kdata 2020-11-16T20:37:43.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:43.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj430 td/osd-backfill-space/4kdata 2020-11-16T20:37:43.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:43.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj431 td/osd-backfill-space/4kdata 2020-11-16T20:37:43.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:43.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj432 td/osd-backfill-space/4kdata 2020-11-16T20:37:43.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:43.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj433 td/osd-backfill-space/4kdata 2020-11-16T20:37:43.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:43.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj434 td/osd-backfill-space/4kdata 2020-11-16T20:37:43.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:43.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj435 td/osd-backfill-space/4kdata 2020-11-16T20:37:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj436 td/osd-backfill-space/4kdata 2020-11-16T20:37:44.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:44.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj437 td/osd-backfill-space/4kdata 2020-11-16T20:37:44.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:44.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj438 td/osd-backfill-space/4kdata 2020-11-16T20:37:44.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:44.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj439 td/osd-backfill-space/4kdata 2020-11-16T20:37:44.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:44.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj440 td/osd-backfill-space/4kdata 2020-11-16T20:37:44.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:44.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj441 td/osd-backfill-space/4kdata 2020-11-16T20:37:45.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:45.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj442 td/osd-backfill-space/4kdata 2020-11-16T20:37:45.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:45.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj443 td/osd-backfill-space/4kdata 2020-11-16T20:37:45.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:45.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj444 td/osd-backfill-space/4kdata 2020-11-16T20:37:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj445 td/osd-backfill-space/4kdata 2020-11-16T20:37:45.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:45.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj446 td/osd-backfill-space/4kdata 2020-11-16T20:37:45.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:45.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj447 td/osd-backfill-space/4kdata 2020-11-16T20:37:46.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:46.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj448 td/osd-backfill-space/4kdata 2020-11-16T20:37:46.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:46.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj449 td/osd-backfill-space/4kdata 2020-11-16T20:37:46.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:46.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj450 td/osd-backfill-space/4kdata 2020-11-16T20:37:46.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:46.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj451 td/osd-backfill-space/4kdata 2020-11-16T20:37:46.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:46.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj452 td/osd-backfill-space/4kdata 2020-11-16T20:37:46.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:46.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj453 td/osd-backfill-space/4kdata 2020-11-16T20:37:47.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:47.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj454 td/osd-backfill-space/4kdata 2020-11-16T20:37:47.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:47.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj455 td/osd-backfill-space/4kdata 2020-11-16T20:37:47.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:47.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj456 td/osd-backfill-space/4kdata 2020-11-16T20:37:47.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:47.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj457 td/osd-backfill-space/4kdata 2020-11-16T20:37:47.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:47.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj458 td/osd-backfill-space/4kdata 2020-11-16T20:37:47.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:47.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj459 td/osd-backfill-space/4kdata 2020-11-16T20:37:48.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:48.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj460 td/osd-backfill-space/4kdata 2020-11-16T20:37:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj461 td/osd-backfill-space/4kdata 2020-11-16T20:37:48.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:48.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj462 td/osd-backfill-space/4kdata 2020-11-16T20:37:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj463 td/osd-backfill-space/4kdata 2020-11-16T20:37:48.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:48.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj464 td/osd-backfill-space/4kdata 2020-11-16T20:37:48.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:48.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj465 td/osd-backfill-space/4kdata 2020-11-16T20:37:49.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:49.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj466 td/osd-backfill-space/4kdata 2020-11-16T20:37:49.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:49.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj467 td/osd-backfill-space/4kdata 2020-11-16T20:37:49.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:49.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj468 td/osd-backfill-space/4kdata 2020-11-16T20:37:49.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:49.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj469 td/osd-backfill-space/4kdata 2020-11-16T20:37:49.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:49.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj470 td/osd-backfill-space/4kdata 2020-11-16T20:37:49.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:49.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj471 td/osd-backfill-space/4kdata 2020-11-16T20:37:49.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj472 td/osd-backfill-space/4kdata 2020-11-16T20:37:50.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:50.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj473 td/osd-backfill-space/4kdata 2020-11-16T20:37:50.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:50.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj474 td/osd-backfill-space/4kdata 2020-11-16T20:37:50.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:50.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj475 td/osd-backfill-space/4kdata 2020-11-16T20:37:50.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:50.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj476 td/osd-backfill-space/4kdata 2020-11-16T20:37:50.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:50.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj477 td/osd-backfill-space/4kdata 2020-11-16T20:37:50.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:50.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj478 td/osd-backfill-space/4kdata 2020-11-16T20:37:51.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:51.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj479 td/osd-backfill-space/4kdata 2020-11-16T20:37:51.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:51.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj480 td/osd-backfill-space/4kdata 2020-11-16T20:37:51.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:51.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj481 td/osd-backfill-space/4kdata 2020-11-16T20:37:51.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:51.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj482 td/osd-backfill-space/4kdata 2020-11-16T20:37:51.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:51.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj483 td/osd-backfill-space/4kdata 2020-11-16T20:37:51.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:51.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj484 td/osd-backfill-space/4kdata 2020-11-16T20:37:52.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:52.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj485 td/osd-backfill-space/4kdata 2020-11-16T20:37:52.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:52.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj486 td/osd-backfill-space/4kdata 2020-11-16T20:37:52.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:52.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj487 td/osd-backfill-space/4kdata 2020-11-16T20:37:52.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:52.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj488 td/osd-backfill-space/4kdata 2020-11-16T20:37:52.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:52.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj489 td/osd-backfill-space/4kdata 2020-11-16T20:37:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:52.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj490 td/osd-backfill-space/4kdata 2020-11-16T20:37:53.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:53.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj491 td/osd-backfill-space/4kdata 2020-11-16T20:37:53.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:53.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj492 td/osd-backfill-space/4kdata 2020-11-16T20:37:53.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:53.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj493 td/osd-backfill-space/4kdata 2020-11-16T20:37:53.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:53.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj494 td/osd-backfill-space/4kdata 2020-11-16T20:37:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj495 td/osd-backfill-space/4kdata 2020-11-16T20:37:53.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:53.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj496 td/osd-backfill-space/4kdata 2020-11-16T20:37:54.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:54.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj497 td/osd-backfill-space/4kdata 2020-11-16T20:37:54.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:54.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj498 td/osd-backfill-space/4kdata 2020-11-16T20:37:54.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:54.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj499 td/osd-backfill-space/4kdata 2020-11-16T20:37:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj500 td/osd-backfill-space/4kdata 2020-11-16T20:37:54.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:54.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj501 td/osd-backfill-space/4kdata 2020-11-16T20:37:54.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:54.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj502 td/osd-backfill-space/4kdata 2020-11-16T20:37:55.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:55.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj503 td/osd-backfill-space/4kdata 2020-11-16T20:37:55.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:55.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj504 td/osd-backfill-space/4kdata 2020-11-16T20:37:55.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:55.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj505 td/osd-backfill-space/4kdata 2020-11-16T20:37:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj506 td/osd-backfill-space/4kdata 2020-11-16T20:37:55.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:55.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj507 td/osd-backfill-space/4kdata 2020-11-16T20:37:55.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:55.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj508 td/osd-backfill-space/4kdata 2020-11-16T20:37:56.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj509 td/osd-backfill-space/4kdata 2020-11-16T20:37:56.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:56.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj510 td/osd-backfill-space/4kdata 2020-11-16T20:37:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj511 td/osd-backfill-space/4kdata 2020-11-16T20:37:56.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:56.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj512 td/osd-backfill-space/4kdata 2020-11-16T20:37:56.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:56.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj513 td/osd-backfill-space/4kdata 2020-11-16T20:37:56.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:56.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj514 td/osd-backfill-space/4kdata 2020-11-16T20:37:57.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:57.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj515 td/osd-backfill-space/4kdata 2020-11-16T20:37:57.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:57.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj516 td/osd-backfill-space/4kdata 2020-11-16T20:37:57.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:57.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj517 td/osd-backfill-space/4kdata 2020-11-16T20:37:57.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:57.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj518 td/osd-backfill-space/4kdata 2020-11-16T20:37:57.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:57.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj519 td/osd-backfill-space/4kdata 2020-11-16T20:37:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj520 td/osd-backfill-space/4kdata 2020-11-16T20:37:58.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:58.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj521 td/osd-backfill-space/4kdata 2020-11-16T20:37:58.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:58.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj522 td/osd-backfill-space/4kdata 2020-11-16T20:37:58.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:58.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj523 td/osd-backfill-space/4kdata 2020-11-16T20:37:58.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:58.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj524 td/osd-backfill-space/4kdata 2020-11-16T20:37:58.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:58.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj525 td/osd-backfill-space/4kdata 2020-11-16T20:37:59.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:59.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj526 td/osd-backfill-space/4kdata 2020-11-16T20:37:59.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:59.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj527 td/osd-backfill-space/4kdata 2020-11-16T20:37:59.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:59.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj528 td/osd-backfill-space/4kdata 2020-11-16T20:37:59.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:59.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj529 td/osd-backfill-space/4kdata 2020-11-16T20:37:59.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:59.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj530 td/osd-backfill-space/4kdata 2020-11-16T20:37:59.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:37:59.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj531 td/osd-backfill-space/4kdata 2020-11-16T20:38:00.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:00.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj532 td/osd-backfill-space/4kdata 2020-11-16T20:38:00.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:00.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj533 td/osd-backfill-space/4kdata 2020-11-16T20:38:00.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:00.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj534 td/osd-backfill-space/4kdata 2020-11-16T20:38:00.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:00.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj535 td/osd-backfill-space/4kdata 2020-11-16T20:38:00.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:00.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj536 td/osd-backfill-space/4kdata 2020-11-16T20:38:00.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:00.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj537 td/osd-backfill-space/4kdata 2020-11-16T20:38:01.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:01.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj538 td/osd-backfill-space/4kdata 2020-11-16T20:38:01.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:01.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj539 td/osd-backfill-space/4kdata 2020-11-16T20:38:01.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:01.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj540 td/osd-backfill-space/4kdata 2020-11-16T20:38:01.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:01.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj541 td/osd-backfill-space/4kdata 2020-11-16T20:38:01.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:01.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj542 td/osd-backfill-space/4kdata 2020-11-16T20:38:01.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:01.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj543 td/osd-backfill-space/4kdata 2020-11-16T20:38:02.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:02.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj544 td/osd-backfill-space/4kdata 2020-11-16T20:38:02.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:02.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj545 td/osd-backfill-space/4kdata 2020-11-16T20:38:02.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:02.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj546 td/osd-backfill-space/4kdata 2020-11-16T20:38:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj547 td/osd-backfill-space/4kdata 2020-11-16T20:38:02.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:02.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj548 td/osd-backfill-space/4kdata 2020-11-16T20:38:02.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:02.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj549 td/osd-backfill-space/4kdata 2020-11-16T20:38:03.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:03.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj550 td/osd-backfill-space/4kdata 2020-11-16T20:38:03.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:03.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj551 td/osd-backfill-space/4kdata 2020-11-16T20:38:03.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:03.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj552 td/osd-backfill-space/4kdata 2020-11-16T20:38:03.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:03.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj553 td/osd-backfill-space/4kdata 2020-11-16T20:38:03.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:03.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj554 td/osd-backfill-space/4kdata 2020-11-16T20:38:03.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:03.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj555 td/osd-backfill-space/4kdata 2020-11-16T20:38:03.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:03.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj556 td/osd-backfill-space/4kdata 2020-11-16T20:38:04.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:04.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj557 td/osd-backfill-space/4kdata 2020-11-16T20:38:04.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:04.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj558 td/osd-backfill-space/4kdata 2020-11-16T20:38:04.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:04.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj559 td/osd-backfill-space/4kdata 2020-11-16T20:38:04.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:04.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj560 td/osd-backfill-space/4kdata 2020-11-16T20:38:04.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:04.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj561 td/osd-backfill-space/4kdata 2020-11-16T20:38:05.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:05.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj562 td/osd-backfill-space/4kdata 2020-11-16T20:38:05.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:05.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj563 td/osd-backfill-space/4kdata 2020-11-16T20:38:05.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:05.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj564 td/osd-backfill-space/4kdata 2020-11-16T20:38:05.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:05.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj565 td/osd-backfill-space/4kdata 2020-11-16T20:38:05.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:05.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj566 td/osd-backfill-space/4kdata 2020-11-16T20:38:05.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:05.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj567 td/osd-backfill-space/4kdata 2020-11-16T20:38:06.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:06.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj568 td/osd-backfill-space/4kdata 2020-11-16T20:38:06.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:06.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj569 td/osd-backfill-space/4kdata 2020-11-16T20:38:06.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:06.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj570 td/osd-backfill-space/4kdata 2020-11-16T20:38:06.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:06.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj571 td/osd-backfill-space/4kdata 2020-11-16T20:38:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:06.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj572 td/osd-backfill-space/4kdata 2020-11-16T20:38:06.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj573 td/osd-backfill-space/4kdata 2020-11-16T20:38:06.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:06.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj574 td/osd-backfill-space/4kdata 2020-11-16T20:38:07.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:07.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj575 td/osd-backfill-space/4kdata 2020-11-16T20:38:07.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:07.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj576 td/osd-backfill-space/4kdata 2020-11-16T20:38:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj577 td/osd-backfill-space/4kdata 2020-11-16T20:38:07.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:07.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj578 td/osd-backfill-space/4kdata 2020-11-16T20:38:07.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:07.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj579 td/osd-backfill-space/4kdata 2020-11-16T20:38:08.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:08.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj580 td/osd-backfill-space/4kdata 2020-11-16T20:38:08.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:08.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj581 td/osd-backfill-space/4kdata 2020-11-16T20:38:08.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:08.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj582 td/osd-backfill-space/4kdata 2020-11-16T20:38:08.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:08.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj583 td/osd-backfill-space/4kdata 2020-11-16T20:38:08.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:08.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj584 td/osd-backfill-space/4kdata 2020-11-16T20:38:08.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:08.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj585 td/osd-backfill-space/4kdata 2020-11-16T20:38:09.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:09.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj586 td/osd-backfill-space/4kdata 2020-11-16T20:38:09.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:09.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj587 td/osd-backfill-space/4kdata 2020-11-16T20:38:09.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:09.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj588 td/osd-backfill-space/4kdata 2020-11-16T20:38:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj589 td/osd-backfill-space/4kdata 2020-11-16T20:38:09.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:09.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj590 td/osd-backfill-space/4kdata 2020-11-16T20:38:09.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:09.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj591 td/osd-backfill-space/4kdata 2020-11-16T20:38:10.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:10.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj592 td/osd-backfill-space/4kdata 2020-11-16T20:38:10.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:10.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj593 td/osd-backfill-space/4kdata 2020-11-16T20:38:10.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:10.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj594 td/osd-backfill-space/4kdata 2020-11-16T20:38:10.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:10.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj595 td/osd-backfill-space/4kdata 2020-11-16T20:38:10.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:10.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj596 td/osd-backfill-space/4kdata 2020-11-16T20:38:10.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:10.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj597 td/osd-backfill-space/4kdata 2020-11-16T20:38:11.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:11.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj598 td/osd-backfill-space/4kdata 2020-11-16T20:38:11.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:11.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj599 td/osd-backfill-space/4kdata 2020-11-16T20:38:11.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2020-11-16T20:38:11.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj600 td/osd-backfill-space/4kdata 2020-11-16T20:38:11.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:583: TEST_backfill_grow: get_pg test obj1 2020-11-16T20:38:11.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T20:38:11.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T20:38:11.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T20:38:11.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T20:38:11.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:583: TEST_backfill_grow: local PG=1.0 2020-11-16T20:38:11.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:585: TEST_backfill_grow: get_primary test obj1 2020-11-16T20:38:11.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T20:38:11.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T20:38:11.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T20:38:11.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T20:38:12.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:585: TEST_backfill_grow: local primary=1 2020-11-16T20:38:12.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:586: TEST_backfill_grow: get_not_primary test obj1 2020-11-16T20:38:12.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2020-11-16T20:38:12.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2020-11-16T20:38:12.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2020-11-16T20:38:12.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T20:38:12.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T20:38:12.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T20:38:12.304 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T20:38:12.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T20:38:12.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2020-11-16T20:38:12.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T20:38:13.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:586: TEST_backfill_grow: local otherosd=0 2020-11-16T20:38:13.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:588: TEST_backfill_grow: ceph osd set noout 2020-11-16T20:38:13.878 INFO:tasks.workunit.client.0.smithi189.stderr:noout is set 2020-11-16T20:38:13.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:589: TEST_backfill_grow: kill_daemons td/osd-backfill-space TERM 0 2020-11-16T20:38:13.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:38:13.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:38:13.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:38:13.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:38:13.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:38:14.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:38:14.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:591: TEST_backfill_grow: expr 600 / 4 2020-11-16T20:38:14.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:591: TEST_backfill_grow: rmobjects=150 2020-11-16T20:38:14.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: seq 1 150 2020-11-16T20:38:14.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:14.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj1 2020-11-16T20:38:14.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:14.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj2 2020-11-16T20:38:15.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:15.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj3 2020-11-16T20:38:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj4 2020-11-16T20:38:15.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:15.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj5 2020-11-16T20:38:15.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:15.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj6 2020-11-16T20:38:15.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:15.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj7 2020-11-16T20:38:15.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:15.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj8 2020-11-16T20:38:15.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj9 2020-11-16T20:38:15.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:15.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj10 2020-11-16T20:38:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj11 2020-11-16T20:38:16.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:16.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj12 2020-11-16T20:38:16.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:16.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj13 2020-11-16T20:38:16.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:16.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj14 2020-11-16T20:38:16.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:16.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj15 2020-11-16T20:38:16.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:16.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj16 2020-11-16T20:38:16.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:16.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj17 2020-11-16T20:38:16.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:16.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj18 2020-11-16T20:38:17.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:17.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj19 2020-11-16T20:38:17.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:17.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj20 2020-11-16T20:38:17.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:17.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj21 2020-11-16T20:38:17.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:17.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj22 2020-11-16T20:38:17.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:17.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj23 2020-11-16T20:38:17.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:17.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj24 2020-11-16T20:38:17.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:17.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj25 2020-11-16T20:38:17.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:17.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj26 2020-11-16T20:38:18.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:18.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj27 2020-11-16T20:38:18.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:18.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj28 2020-11-16T20:38:18.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:18.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj29 2020-11-16T20:38:18.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:18.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj30 2020-11-16T20:38:18.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:18.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj31 2020-11-16T20:38:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj32 2020-11-16T20:38:18.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:18.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj33 2020-11-16T20:38:18.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:18.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj34 2020-11-16T20:38:19.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:19.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj35 2020-11-16T20:38:19.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:19.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj36 2020-11-16T20:38:19.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:19.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj37 2020-11-16T20:38:19.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:19.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj38 2020-11-16T20:38:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj39 2020-11-16T20:38:19.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:19.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj40 2020-11-16T20:38:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:19.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj41 2020-11-16T20:38:20.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:20.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj42 2020-11-16T20:38:20.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:20.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj43 2020-11-16T20:38:20.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:20.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj44 2020-11-16T20:38:20.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:20.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj45 2020-11-16T20:38:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj46 2020-11-16T20:38:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:20.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj47 2020-11-16T20:38:20.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:20.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj48 2020-11-16T20:38:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj49 2020-11-16T20:38:21.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:21.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj50 2020-11-16T20:38:21.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:21.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj51 2020-11-16T20:38:21.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:21.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj52 2020-11-16T20:38:21.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:21.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj53 2020-11-16T20:38:21.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:21.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj54 2020-11-16T20:38:21.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:21.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj55 2020-11-16T20:38:21.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:21.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj56 2020-11-16T20:38:21.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:21.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj57 2020-11-16T20:38:22.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:22.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj58 2020-11-16T20:38:22.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:22.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj59 2020-11-16T20:38:22.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:22.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj60 2020-11-16T20:38:22.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:22.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj61 2020-11-16T20:38:22.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:22.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj62 2020-11-16T20:38:22.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:22.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj63 2020-11-16T20:38:22.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:22.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj64 2020-11-16T20:38:22.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:22.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj65 2020-11-16T20:38:23.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:23.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj66 2020-11-16T20:38:23.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:23.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj67 2020-11-16T20:38:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj68 2020-11-16T20:38:23.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:23.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj69 2020-11-16T20:38:23.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:23.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj70 2020-11-16T20:38:23.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:23.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj71 2020-11-16T20:38:23.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:23.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj72 2020-11-16T20:38:23.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:23.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj73 2020-11-16T20:38:24.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:24.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj74 2020-11-16T20:38:24.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:24.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj75 2020-11-16T20:38:24.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj76 2020-11-16T20:38:24.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:24.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj77 2020-11-16T20:38:24.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:24.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj78 2020-11-16T20:38:24.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:24.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj79 2020-11-16T20:38:24.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:24.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj80 2020-11-16T20:38:24.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:24.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj81 2020-11-16T20:38:25.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:25.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj82 2020-11-16T20:38:25.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:25.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj83 2020-11-16T20:38:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj84 2020-11-16T20:38:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:25.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj85 2020-11-16T20:38:25.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:25.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj86 2020-11-16T20:38:25.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:25.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj87 2020-11-16T20:38:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:25.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj88 2020-11-16T20:38:25.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:25.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj89 2020-11-16T20:38:25.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:25.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj90 2020-11-16T20:38:26.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:26.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj91 2020-11-16T20:38:26.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:26.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj92 2020-11-16T20:38:26.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:26.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj93 2020-11-16T20:38:26.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:26.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj94 2020-11-16T20:38:26.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:26.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj95 2020-11-16T20:38:26.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:26.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj96 2020-11-16T20:38:26.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:26.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj97 2020-11-16T20:38:26.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:26.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj98 2020-11-16T20:38:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj99 2020-11-16T20:38:27.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:27.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj100 2020-11-16T20:38:27.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:27.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj101 2020-11-16T20:38:27.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:27.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj102 2020-11-16T20:38:27.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:27.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj103 2020-11-16T20:38:27.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:27.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj104 2020-11-16T20:38:27.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:27.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj105 2020-11-16T20:38:28.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:28.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj106 2020-11-16T20:38:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj107 2020-11-16T20:38:28.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:28.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj108 2020-11-16T20:38:28.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:28.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj109 2020-11-16T20:38:28.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:28.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj110 2020-11-16T20:38:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj111 2020-11-16T20:38:28.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:28.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj112 2020-11-16T20:38:29.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:29.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj113 2020-11-16T20:38:29.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:29.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj114 2020-11-16T20:38:29.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:29.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj115 2020-11-16T20:38:29.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:29.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj116 2020-11-16T20:38:29.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:29.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj117 2020-11-16T20:38:29.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:29.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj118 2020-11-16T20:38:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj119 2020-11-16T20:38:29.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:29.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj120 2020-11-16T20:38:30.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:30.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj121 2020-11-16T20:38:30.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:30.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj122 2020-11-16T20:38:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj123 2020-11-16T20:38:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj124 2020-11-16T20:38:30.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:30.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj125 2020-11-16T20:38:30.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:30.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj126 2020-11-16T20:38:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:30.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj127 2020-11-16T20:38:30.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:30.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj128 2020-11-16T20:38:31.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:31.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj129 2020-11-16T20:38:31.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:31.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj130 2020-11-16T20:38:31.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:31.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj131 2020-11-16T20:38:31.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:31.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj132 2020-11-16T20:38:31.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:31.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj133 2020-11-16T20:38:31.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj134 2020-11-16T20:38:31.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:31.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj135 2020-11-16T20:38:31.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:31.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj136 2020-11-16T20:38:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:32.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj137 2020-11-16T20:38:32.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:32.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj138 2020-11-16T20:38:32.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:32.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj139 2020-11-16T20:38:32.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:32.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj140 2020-11-16T20:38:32.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:32.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj141 2020-11-16T20:38:32.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:32.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj142 2020-11-16T20:38:32.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:32.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj143 2020-11-16T20:38:33.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:33.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj144 2020-11-16T20:38:33.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:33.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj145 2020-11-16T20:38:33.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:33.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj146 2020-11-16T20:38:33.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:33.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj147 2020-11-16T20:38:33.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:33.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj148 2020-11-16T20:38:33.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:33.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj149 2020-11-16T20:38:33.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T20:38:33.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj150 2020-11-16T20:38:34.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:597: TEST_backfill_grow: dd if=/dev/urandom of=td/osd-backfill-space/6kdata bs=6k count=1 2020-11-16T20:38:34.003 INFO:tasks.workunit.client.0.smithi189.stderr:1+0 records in 2020-11-16T20:38:34.003 INFO:tasks.workunit.client.0.smithi189.stderr:1+0 records out 2020-11-16T20:38:34.004 INFO:tasks.workunit.client.0.smithi189.stderr:6144 bytes (6.1 kB, 6.0 KiB) copied, 0.000253833 s, 24.2 MB/s 2020-11-16T20:38:34.005 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: expr 150 + 1 2020-11-16T20:38:34.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: seq 151 600 2020-11-16T20:38:34.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:34.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj151 td/osd-backfill-space/6kdata 2020-11-16T20:38:34.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:34.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj152 td/osd-backfill-space/6kdata 2020-11-16T20:38:34.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:34.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj153 td/osd-backfill-space/6kdata 2020-11-16T20:38:34.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:34.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj154 td/osd-backfill-space/6kdata 2020-11-16T20:38:34.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:34.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj155 td/osd-backfill-space/6kdata 2020-11-16T20:38:34.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:34.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj156 td/osd-backfill-space/6kdata 2020-11-16T20:38:34.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:34.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj157 td/osd-backfill-space/6kdata 2020-11-16T20:38:35.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:35.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj158 td/osd-backfill-space/6kdata 2020-11-16T20:38:35.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:35.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj159 td/osd-backfill-space/6kdata 2020-11-16T20:38:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj160 td/osd-backfill-space/6kdata 2020-11-16T20:38:35.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:35.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj161 td/osd-backfill-space/6kdata 2020-11-16T20:38:35.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:35.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj162 td/osd-backfill-space/6kdata 2020-11-16T20:38:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:35.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj163 td/osd-backfill-space/6kdata 2020-11-16T20:38:35.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:35.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj164 td/osd-backfill-space/6kdata 2020-11-16T20:38:36.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:36.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj165 td/osd-backfill-space/6kdata 2020-11-16T20:38:36.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:36.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj166 td/osd-backfill-space/6kdata 2020-11-16T20:38:36.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:36.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj167 td/osd-backfill-space/6kdata 2020-11-16T20:38:36.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:36.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj168 td/osd-backfill-space/6kdata 2020-11-16T20:38:36.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:36.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj169 td/osd-backfill-space/6kdata 2020-11-16T20:38:36.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:36.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj170 td/osd-backfill-space/6kdata 2020-11-16T20:38:37.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:37.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj171 td/osd-backfill-space/6kdata 2020-11-16T20:38:37.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:37.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj172 td/osd-backfill-space/6kdata 2020-11-16T20:38:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj173 td/osd-backfill-space/6kdata 2020-11-16T20:38:37.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:37.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj174 td/osd-backfill-space/6kdata 2020-11-16T20:38:37.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:37.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj175 td/osd-backfill-space/6kdata 2020-11-16T20:38:37.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:37.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj176 td/osd-backfill-space/6kdata 2020-11-16T20:38:38.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:38.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj177 td/osd-backfill-space/6kdata 2020-11-16T20:38:38.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:38.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj178 td/osd-backfill-space/6kdata 2020-11-16T20:38:38.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:38.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj179 td/osd-backfill-space/6kdata 2020-11-16T20:38:38.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:38.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj180 td/osd-backfill-space/6kdata 2020-11-16T20:38:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj181 td/osd-backfill-space/6kdata 2020-11-16T20:38:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj182 td/osd-backfill-space/6kdata 2020-11-16T20:38:39.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:39.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj183 td/osd-backfill-space/6kdata 2020-11-16T20:38:39.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:39.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj184 td/osd-backfill-space/6kdata 2020-11-16T20:38:39.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:39.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj185 td/osd-backfill-space/6kdata 2020-11-16T20:38:39.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:39.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj186 td/osd-backfill-space/6kdata 2020-11-16T20:38:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj187 td/osd-backfill-space/6kdata 2020-11-16T20:38:39.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:39.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj188 td/osd-backfill-space/6kdata 2020-11-16T20:38:39.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:39.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj189 td/osd-backfill-space/6kdata 2020-11-16T20:38:40.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:40.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj190 td/osd-backfill-space/6kdata 2020-11-16T20:38:40.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:40.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj191 td/osd-backfill-space/6kdata 2020-11-16T20:38:40.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:40.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj192 td/osd-backfill-space/6kdata 2020-11-16T20:38:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj193 td/osd-backfill-space/6kdata 2020-11-16T20:38:40.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:40.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj194 td/osd-backfill-space/6kdata 2020-11-16T20:38:40.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:40.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj195 td/osd-backfill-space/6kdata 2020-11-16T20:38:41.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:41.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj196 td/osd-backfill-space/6kdata 2020-11-16T20:38:41.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:41.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj197 td/osd-backfill-space/6kdata 2020-11-16T20:38:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj198 td/osd-backfill-space/6kdata 2020-11-16T20:38:41.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:41.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj199 td/osd-backfill-space/6kdata 2020-11-16T20:38:41.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:41.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj200 td/osd-backfill-space/6kdata 2020-11-16T20:38:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj201 td/osd-backfill-space/6kdata 2020-11-16T20:38:42.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:42.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj202 td/osd-backfill-space/6kdata 2020-11-16T20:38:42.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:42.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj203 td/osd-backfill-space/6kdata 2020-11-16T20:38:42.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:42.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj204 td/osd-backfill-space/6kdata 2020-11-16T20:38:42.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:42.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj205 td/osd-backfill-space/6kdata 2020-11-16T20:38:42.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:42.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj206 td/osd-backfill-space/6kdata 2020-11-16T20:38:42.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:42.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj207 td/osd-backfill-space/6kdata 2020-11-16T20:38:42.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:43.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj208 td/osd-backfill-space/6kdata 2020-11-16T20:38:43.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:43.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj209 td/osd-backfill-space/6kdata 2020-11-16T20:38:43.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:43.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj210 td/osd-backfill-space/6kdata 2020-11-16T20:38:43.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:43.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj211 td/osd-backfill-space/6kdata 2020-11-16T20:38:43.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:43.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj212 td/osd-backfill-space/6kdata 2020-11-16T20:38:43.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:43.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj213 td/osd-backfill-space/6kdata 2020-11-16T20:38:43.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:43.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj214 td/osd-backfill-space/6kdata 2020-11-16T20:38:44.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:44.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj215 td/osd-backfill-space/6kdata 2020-11-16T20:38:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj216 td/osd-backfill-space/6kdata 2020-11-16T20:38:44.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:44.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj217 td/osd-backfill-space/6kdata 2020-11-16T20:38:44.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:44.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj218 td/osd-backfill-space/6kdata 2020-11-16T20:38:44.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:44.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj219 td/osd-backfill-space/6kdata 2020-11-16T20:38:44.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:44.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj220 td/osd-backfill-space/6kdata 2020-11-16T20:38:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:45.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj221 td/osd-backfill-space/6kdata 2020-11-16T20:38:45.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:45.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj222 td/osd-backfill-space/6kdata 2020-11-16T20:38:45.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:45.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj223 td/osd-backfill-space/6kdata 2020-11-16T20:38:45.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj224 td/osd-backfill-space/6kdata 2020-11-16T20:38:45.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:45.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj225 td/osd-backfill-space/6kdata 2020-11-16T20:38:45.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:45.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj226 td/osd-backfill-space/6kdata 2020-11-16T20:38:46.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:46.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj227 td/osd-backfill-space/6kdata 2020-11-16T20:38:46.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:46.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj228 td/osd-backfill-space/6kdata 2020-11-16T20:38:46.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:46.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj229 td/osd-backfill-space/6kdata 2020-11-16T20:38:46.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:46.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj230 td/osd-backfill-space/6kdata 2020-11-16T20:38:46.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:46.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj231 td/osd-backfill-space/6kdata 2020-11-16T20:38:46.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:46.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj232 td/osd-backfill-space/6kdata 2020-11-16T20:38:46.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:46.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj233 td/osd-backfill-space/6kdata 2020-11-16T20:38:47.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:47.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj234 td/osd-backfill-space/6kdata 2020-11-16T20:38:47.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:47.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj235 td/osd-backfill-space/6kdata 2020-11-16T20:38:47.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:47.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj236 td/osd-backfill-space/6kdata 2020-11-16T20:38:47.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:47.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj237 td/osd-backfill-space/6kdata 2020-11-16T20:38:47.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:47.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj238 td/osd-backfill-space/6kdata 2020-11-16T20:38:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj239 td/osd-backfill-space/6kdata 2020-11-16T20:38:48.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:48.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj240 td/osd-backfill-space/6kdata 2020-11-16T20:38:48.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:48.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj241 td/osd-backfill-space/6kdata 2020-11-16T20:38:48.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:48.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj242 td/osd-backfill-space/6kdata 2020-11-16T20:38:48.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:48.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj243 td/osd-backfill-space/6kdata 2020-11-16T20:38:48.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj244 td/osd-backfill-space/6kdata 2020-11-16T20:38:48.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:48.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj245 td/osd-backfill-space/6kdata 2020-11-16T20:38:49.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:49.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj246 td/osd-backfill-space/6kdata 2020-11-16T20:38:49.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:49.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj247 td/osd-backfill-space/6kdata 2020-11-16T20:38:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj248 td/osd-backfill-space/6kdata 2020-11-16T20:38:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj249 td/osd-backfill-space/6kdata 2020-11-16T20:38:49.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:49.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj250 td/osd-backfill-space/6kdata 2020-11-16T20:38:49.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:49.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj251 td/osd-backfill-space/6kdata 2020-11-16T20:38:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj252 td/osd-backfill-space/6kdata 2020-11-16T20:38:50.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:50.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj253 td/osd-backfill-space/6kdata 2020-11-16T20:38:50.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:50.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj254 td/osd-backfill-space/6kdata 2020-11-16T20:38:50.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:50.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj255 td/osd-backfill-space/6kdata 2020-11-16T20:38:50.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:50.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj256 td/osd-backfill-space/6kdata 2020-11-16T20:38:50.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:50.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj257 td/osd-backfill-space/6kdata 2020-11-16T20:38:50.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:50.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj258 td/osd-backfill-space/6kdata 2020-11-16T20:38:51.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:51.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj259 td/osd-backfill-space/6kdata 2020-11-16T20:38:51.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:51.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj260 td/osd-backfill-space/6kdata 2020-11-16T20:38:51.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:51.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj261 td/osd-backfill-space/6kdata 2020-11-16T20:38:51.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:51.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj262 td/osd-backfill-space/6kdata 2020-11-16T20:38:51.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:51.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj263 td/osd-backfill-space/6kdata 2020-11-16T20:38:51.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:51.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj264 td/osd-backfill-space/6kdata 2020-11-16T20:38:52.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:52.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj265 td/osd-backfill-space/6kdata 2020-11-16T20:38:52.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:52.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj266 td/osd-backfill-space/6kdata 2020-11-16T20:38:52.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:52.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj267 td/osd-backfill-space/6kdata 2020-11-16T20:38:52.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:52.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj268 td/osd-backfill-space/6kdata 2020-11-16T20:38:52.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:52.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj269 td/osd-backfill-space/6kdata 2020-11-16T20:38:52.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:52.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj270 td/osd-backfill-space/6kdata 2020-11-16T20:38:53.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:53.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj271 td/osd-backfill-space/6kdata 2020-11-16T20:38:53.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:53.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj272 td/osd-backfill-space/6kdata 2020-11-16T20:38:53.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:53.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj273 td/osd-backfill-space/6kdata 2020-11-16T20:38:53.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:53.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj274 td/osd-backfill-space/6kdata 2020-11-16T20:38:53.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:53.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj275 td/osd-backfill-space/6kdata 2020-11-16T20:38:53.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:53.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj276 td/osd-backfill-space/6kdata 2020-11-16T20:38:53.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:53.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj277 td/osd-backfill-space/6kdata 2020-11-16T20:38:54.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:54.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj278 td/osd-backfill-space/6kdata 2020-11-16T20:38:54.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:54.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj279 td/osd-backfill-space/6kdata 2020-11-16T20:38:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj280 td/osd-backfill-space/6kdata 2020-11-16T20:38:54.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:54.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj281 td/osd-backfill-space/6kdata 2020-11-16T20:38:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj282 td/osd-backfill-space/6kdata 2020-11-16T20:38:54.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:54.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj283 td/osd-backfill-space/6kdata 2020-11-16T20:38:55.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:55.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj284 td/osd-backfill-space/6kdata 2020-11-16T20:38:55.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:55.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj285 td/osd-backfill-space/6kdata 2020-11-16T20:38:55.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:55.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj286 td/osd-backfill-space/6kdata 2020-11-16T20:38:55.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:55.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj287 td/osd-backfill-space/6kdata 2020-11-16T20:38:55.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:55.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj288 td/osd-backfill-space/6kdata 2020-11-16T20:38:55.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:55.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj289 td/osd-backfill-space/6kdata 2020-11-16T20:38:55.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:55.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj290 td/osd-backfill-space/6kdata 2020-11-16T20:38:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj291 td/osd-backfill-space/6kdata 2020-11-16T20:38:56.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:56.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj292 td/osd-backfill-space/6kdata 2020-11-16T20:38:56.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:56.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj293 td/osd-backfill-space/6kdata 2020-11-16T20:38:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj294 td/osd-backfill-space/6kdata 2020-11-16T20:38:56.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:56.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj295 td/osd-backfill-space/6kdata 2020-11-16T20:38:56.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:56.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj296 td/osd-backfill-space/6kdata 2020-11-16T20:38:56.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:56.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj297 td/osd-backfill-space/6kdata 2020-11-16T20:38:57.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:57.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj298 td/osd-backfill-space/6kdata 2020-11-16T20:38:57.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:57.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj299 td/osd-backfill-space/6kdata 2020-11-16T20:38:57.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:57.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj300 td/osd-backfill-space/6kdata 2020-11-16T20:38:57.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:57.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj301 td/osd-backfill-space/6kdata 2020-11-16T20:38:57.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:57.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj302 td/osd-backfill-space/6kdata 2020-11-16T20:38:57.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:57.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj303 td/osd-backfill-space/6kdata 2020-11-16T20:38:58.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:58.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj304 td/osd-backfill-space/6kdata 2020-11-16T20:38:58.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:58.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj305 td/osd-backfill-space/6kdata 2020-11-16T20:38:58.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:58.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj306 td/osd-backfill-space/6kdata 2020-11-16T20:38:58.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:58.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj307 td/osd-backfill-space/6kdata 2020-11-16T20:38:58.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:38:58.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj308 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj309 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj310 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj311 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj312 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj313 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj314 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj315 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj316 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj317 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj318 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj319 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj320 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj321 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj322 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj323 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj324 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj325 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj326 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj327 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj328 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj329 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj330 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj331 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj332 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj333 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj334 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj335 td/osd-backfill-space/6kdata 2020-11-16T20:39:02.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:02.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj336 td/osd-backfill-space/6kdata 2020-11-16T20:39:03.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:03.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj337 td/osd-backfill-space/6kdata 2020-11-16T20:39:03.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:03.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj338 td/osd-backfill-space/6kdata 2020-11-16T20:39:03.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:03.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj339 td/osd-backfill-space/6kdata 2020-11-16T20:39:03.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:03.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj340 td/osd-backfill-space/6kdata 2020-11-16T20:39:03.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:03.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj341 td/osd-backfill-space/6kdata 2020-11-16T20:39:03.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:03.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj342 td/osd-backfill-space/6kdata 2020-11-16T20:39:03.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:03.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj343 td/osd-backfill-space/6kdata 2020-11-16T20:39:04.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:04.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj344 td/osd-backfill-space/6kdata 2020-11-16T20:39:04.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:04.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj345 td/osd-backfill-space/6kdata 2020-11-16T20:39:04.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:04.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj346 td/osd-backfill-space/6kdata 2020-11-16T20:39:04.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:04.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj347 td/osd-backfill-space/6kdata 2020-11-16T20:39:04.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:04.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj348 td/osd-backfill-space/6kdata 2020-11-16T20:39:04.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:04.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj349 td/osd-backfill-space/6kdata 2020-11-16T20:39:04.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:04.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj350 td/osd-backfill-space/6kdata 2020-11-16T20:39:05.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:05.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj351 td/osd-backfill-space/6kdata 2020-11-16T20:39:05.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:05.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj352 td/osd-backfill-space/6kdata 2020-11-16T20:39:05.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:05.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj353 td/osd-backfill-space/6kdata 2020-11-16T20:39:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:05.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj354 td/osd-backfill-space/6kdata 2020-11-16T20:39:05.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:05.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj355 td/osd-backfill-space/6kdata 2020-11-16T20:39:05.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:05.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj356 td/osd-backfill-space/6kdata 2020-11-16T20:39:05.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:05.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj357 td/osd-backfill-space/6kdata 2020-11-16T20:39:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj358 td/osd-backfill-space/6kdata 2020-11-16T20:39:06.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:06.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj359 td/osd-backfill-space/6kdata 2020-11-16T20:39:06.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:06.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj360 td/osd-backfill-space/6kdata 2020-11-16T20:39:06.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:06.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj361 td/osd-backfill-space/6kdata 2020-11-16T20:39:06.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:06.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj362 td/osd-backfill-space/6kdata 2020-11-16T20:39:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj363 td/osd-backfill-space/6kdata 2020-11-16T20:39:07.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:07.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj364 td/osd-backfill-space/6kdata 2020-11-16T20:39:07.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:07.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj365 td/osd-backfill-space/6kdata 2020-11-16T20:39:07.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:07.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj366 td/osd-backfill-space/6kdata 2020-11-16T20:39:07.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:07.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj367 td/osd-backfill-space/6kdata 2020-11-16T20:39:07.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:07.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj368 td/osd-backfill-space/6kdata 2020-11-16T20:39:07.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:07.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj369 td/osd-backfill-space/6kdata 2020-11-16T20:39:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:07.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj370 td/osd-backfill-space/6kdata 2020-11-16T20:39:08.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:08.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj371 td/osd-backfill-space/6kdata 2020-11-16T20:39:08.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:08.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj372 td/osd-backfill-space/6kdata 2020-11-16T20:39:08.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:08.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj373 td/osd-backfill-space/6kdata 2020-11-16T20:39:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj374 td/osd-backfill-space/6kdata 2020-11-16T20:39:08.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:08.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj375 td/osd-backfill-space/6kdata 2020-11-16T20:39:08.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:08.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj376 td/osd-backfill-space/6kdata 2020-11-16T20:39:08.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:08.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj377 td/osd-backfill-space/6kdata 2020-11-16T20:39:09.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:09.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj378 td/osd-backfill-space/6kdata 2020-11-16T20:39:09.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:09.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj379 td/osd-backfill-space/6kdata 2020-11-16T20:39:09.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:09.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj380 td/osd-backfill-space/6kdata 2020-11-16T20:39:09.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:09.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj381 td/osd-backfill-space/6kdata 2020-11-16T20:39:09.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:09.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj382 td/osd-backfill-space/6kdata 2020-11-16T20:39:09.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:09.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj383 td/osd-backfill-space/6kdata 2020-11-16T20:39:10.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:10.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj384 td/osd-backfill-space/6kdata 2020-11-16T20:39:10.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:10.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj385 td/osd-backfill-space/6kdata 2020-11-16T20:39:10.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:10.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj386 td/osd-backfill-space/6kdata 2020-11-16T20:39:10.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:10.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj387 td/osd-backfill-space/6kdata 2020-11-16T20:39:10.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:10.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj388 td/osd-backfill-space/6kdata 2020-11-16T20:39:10.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:10.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj389 td/osd-backfill-space/6kdata 2020-11-16T20:39:10.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:10.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj390 td/osd-backfill-space/6kdata 2020-11-16T20:39:11.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:11.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj391 td/osd-backfill-space/6kdata 2020-11-16T20:39:11.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:11.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj392 td/osd-backfill-space/6kdata 2020-11-16T20:39:11.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:11.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj393 td/osd-backfill-space/6kdata 2020-11-16T20:39:11.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:11.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj394 td/osd-backfill-space/6kdata 2020-11-16T20:39:11.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:11.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj395 td/osd-backfill-space/6kdata 2020-11-16T20:39:11.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:11.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj396 td/osd-backfill-space/6kdata 2020-11-16T20:39:11.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:11.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj397 td/osd-backfill-space/6kdata 2020-11-16T20:39:12.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:12.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj398 td/osd-backfill-space/6kdata 2020-11-16T20:39:12.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:12.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj399 td/osd-backfill-space/6kdata 2020-11-16T20:39:12.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:12.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj400 td/osd-backfill-space/6kdata 2020-11-16T20:39:12.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:12.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj401 td/osd-backfill-space/6kdata 2020-11-16T20:39:12.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:12.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj402 td/osd-backfill-space/6kdata 2020-11-16T20:39:12.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:12.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj403 td/osd-backfill-space/6kdata 2020-11-16T20:39:13.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:13.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj404 td/osd-backfill-space/6kdata 2020-11-16T20:39:13.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:13.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj405 td/osd-backfill-space/6kdata 2020-11-16T20:39:13.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:13.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj406 td/osd-backfill-space/6kdata 2020-11-16T20:39:13.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:13.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj407 td/osd-backfill-space/6kdata 2020-11-16T20:39:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj408 td/osd-backfill-space/6kdata 2020-11-16T20:39:13.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:13.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj409 td/osd-backfill-space/6kdata 2020-11-16T20:39:13.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:13.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj410 td/osd-backfill-space/6kdata 2020-11-16T20:39:14.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:14.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj411 td/osd-backfill-space/6kdata 2020-11-16T20:39:14.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:14.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj412 td/osd-backfill-space/6kdata 2020-11-16T20:39:14.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:14.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj413 td/osd-backfill-space/6kdata 2020-11-16T20:39:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj414 td/osd-backfill-space/6kdata 2020-11-16T20:39:14.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:14.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj415 td/osd-backfill-space/6kdata 2020-11-16T20:39:14.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:14.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj416 td/osd-backfill-space/6kdata 2020-11-16T20:39:15.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:15.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj417 td/osd-backfill-space/6kdata 2020-11-16T20:39:15.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:15.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj418 td/osd-backfill-space/6kdata 2020-11-16T20:39:15.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:15.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj419 td/osd-backfill-space/6kdata 2020-11-16T20:39:15.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:15.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj420 td/osd-backfill-space/6kdata 2020-11-16T20:39:15.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:15.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj421 td/osd-backfill-space/6kdata 2020-11-16T20:39:15.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:15.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj422 td/osd-backfill-space/6kdata 2020-11-16T20:39:15.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:15.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj423 td/osd-backfill-space/6kdata 2020-11-16T20:39:16.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:16.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj424 td/osd-backfill-space/6kdata 2020-11-16T20:39:16.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:16.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj425 td/osd-backfill-space/6kdata 2020-11-16T20:39:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj426 td/osd-backfill-space/6kdata 2020-11-16T20:39:16.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:16.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj427 td/osd-backfill-space/6kdata 2020-11-16T20:39:16.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:16.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj428 td/osd-backfill-space/6kdata 2020-11-16T20:39:16.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:16.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj429 td/osd-backfill-space/6kdata 2020-11-16T20:39:17.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:17.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj430 td/osd-backfill-space/6kdata 2020-11-16T20:39:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj431 td/osd-backfill-space/6kdata 2020-11-16T20:39:17.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:17.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj432 td/osd-backfill-space/6kdata 2020-11-16T20:39:17.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:17.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj433 td/osd-backfill-space/6kdata 2020-11-16T20:39:17.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:17.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj434 td/osd-backfill-space/6kdata 2020-11-16T20:39:18.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:18.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj435 td/osd-backfill-space/6kdata 2020-11-16T20:39:18.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:18.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj436 td/osd-backfill-space/6kdata 2020-11-16T20:39:18.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:18.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj437 td/osd-backfill-space/6kdata 2020-11-16T20:39:18.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:18.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj438 td/osd-backfill-space/6kdata 2020-11-16T20:39:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj439 td/osd-backfill-space/6kdata 2020-11-16T20:39:18.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:18.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj440 td/osd-backfill-space/6kdata 2020-11-16T20:39:18.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:18.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj441 td/osd-backfill-space/6kdata 2020-11-16T20:39:19.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:19.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj442 td/osd-backfill-space/6kdata 2020-11-16T20:39:19.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:19.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj443 td/osd-backfill-space/6kdata 2020-11-16T20:39:19.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:19.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj444 td/osd-backfill-space/6kdata 2020-11-16T20:39:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj445 td/osd-backfill-space/6kdata 2020-11-16T20:39:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj446 td/osd-backfill-space/6kdata 2020-11-16T20:39:19.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:19.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj447 td/osd-backfill-space/6kdata 2020-11-16T20:39:20.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:20.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj448 td/osd-backfill-space/6kdata 2020-11-16T20:39:20.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:20.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj449 td/osd-backfill-space/6kdata 2020-11-16T20:39:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj450 td/osd-backfill-space/6kdata 2020-11-16T20:39:20.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:20.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj451 td/osd-backfill-space/6kdata 2020-11-16T20:39:20.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:20.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj452 td/osd-backfill-space/6kdata 2020-11-16T20:39:20.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:20.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj453 td/osd-backfill-space/6kdata 2020-11-16T20:39:20.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:20.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj454 td/osd-backfill-space/6kdata 2020-11-16T20:39:21.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:21.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj455 td/osd-backfill-space/6kdata 2020-11-16T20:39:21.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:21.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj456 td/osd-backfill-space/6kdata 2020-11-16T20:39:21.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:21.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj457 td/osd-backfill-space/6kdata 2020-11-16T20:39:21.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:21.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj458 td/osd-backfill-space/6kdata 2020-11-16T20:39:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj459 td/osd-backfill-space/6kdata 2020-11-16T20:39:21.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:21.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj460 td/osd-backfill-space/6kdata 2020-11-16T20:39:21.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:21.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj461 td/osd-backfill-space/6kdata 2020-11-16T20:39:22.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:22.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj462 td/osd-backfill-space/6kdata 2020-11-16T20:39:22.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:22.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj463 td/osd-backfill-space/6kdata 2020-11-16T20:39:22.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:22.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj464 td/osd-backfill-space/6kdata 2020-11-16T20:39:22.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:22.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj465 td/osd-backfill-space/6kdata 2020-11-16T20:39:22.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:22.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj466 td/osd-backfill-space/6kdata 2020-11-16T20:39:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:22.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj467 td/osd-backfill-space/6kdata 2020-11-16T20:39:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj468 td/osd-backfill-space/6kdata 2020-11-16T20:39:23.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:23.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj469 td/osd-backfill-space/6kdata 2020-11-16T20:39:23.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:23.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj470 td/osd-backfill-space/6kdata 2020-11-16T20:39:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj471 td/osd-backfill-space/6kdata 2020-11-16T20:39:23.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:23.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj472 td/osd-backfill-space/6kdata 2020-11-16T20:39:23.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:23.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj473 td/osd-backfill-space/6kdata 2020-11-16T20:39:24.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:24.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj474 td/osd-backfill-space/6kdata 2020-11-16T20:39:24.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:24.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj475 td/osd-backfill-space/6kdata 2020-11-16T20:39:24.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:24.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj476 td/osd-backfill-space/6kdata 2020-11-16T20:39:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj477 td/osd-backfill-space/6kdata 2020-11-16T20:39:24.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:24.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj478 td/osd-backfill-space/6kdata 2020-11-16T20:39:24.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:24.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj479 td/osd-backfill-space/6kdata 2020-11-16T20:39:24.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:24.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj480 td/osd-backfill-space/6kdata 2020-11-16T20:39:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj481 td/osd-backfill-space/6kdata 2020-11-16T20:39:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj482 td/osd-backfill-space/6kdata 2020-11-16T20:39:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj483 td/osd-backfill-space/6kdata 2020-11-16T20:39:25.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:25.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj484 td/osd-backfill-space/6kdata 2020-11-16T20:39:25.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:25.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj485 td/osd-backfill-space/6kdata 2020-11-16T20:39:25.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:25.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj486 td/osd-backfill-space/6kdata 2020-11-16T20:39:26.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:26.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj487 td/osd-backfill-space/6kdata 2020-11-16T20:39:26.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:26.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj488 td/osd-backfill-space/6kdata 2020-11-16T20:39:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj489 td/osd-backfill-space/6kdata 2020-11-16T20:39:26.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:26.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj490 td/osd-backfill-space/6kdata 2020-11-16T20:39:26.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:26.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj491 td/osd-backfill-space/6kdata 2020-11-16T20:39:26.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:26.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj492 td/osd-backfill-space/6kdata 2020-11-16T20:39:26.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:26.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj493 td/osd-backfill-space/6kdata 2020-11-16T20:39:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj494 td/osd-backfill-space/6kdata 2020-11-16T20:39:27.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:27.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj495 td/osd-backfill-space/6kdata 2020-11-16T20:39:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:27.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj496 td/osd-backfill-space/6kdata 2020-11-16T20:39:27.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:27.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj497 td/osd-backfill-space/6kdata 2020-11-16T20:39:27.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:27.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj498 td/osd-backfill-space/6kdata 2020-11-16T20:39:27.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:27.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj499 td/osd-backfill-space/6kdata 2020-11-16T20:39:27.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:27.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj500 td/osd-backfill-space/6kdata 2020-11-16T20:39:28.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:28.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj501 td/osd-backfill-space/6kdata 2020-11-16T20:39:28.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:28.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj502 td/osd-backfill-space/6kdata 2020-11-16T20:39:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj503 td/osd-backfill-space/6kdata 2020-11-16T20:39:28.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:28.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj504 td/osd-backfill-space/6kdata 2020-11-16T20:39:28.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:28.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj505 td/osd-backfill-space/6kdata 2020-11-16T20:39:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj506 td/osd-backfill-space/6kdata 2020-11-16T20:39:28.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:28.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj507 td/osd-backfill-space/6kdata 2020-11-16T20:39:29.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:29.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj508 td/osd-backfill-space/6kdata 2020-11-16T20:39:29.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:29.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj509 td/osd-backfill-space/6kdata 2020-11-16T20:39:29.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:29.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj510 td/osd-backfill-space/6kdata 2020-11-16T20:39:29.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:29.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj511 td/osd-backfill-space/6kdata 2020-11-16T20:39:29.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:29.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj512 td/osd-backfill-space/6kdata 2020-11-16T20:39:30.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:30.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj513 td/osd-backfill-space/6kdata 2020-11-16T20:39:30.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:30.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj514 td/osd-backfill-space/6kdata 2020-11-16T20:39:30.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:30.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj515 td/osd-backfill-space/6kdata 2020-11-16T20:39:30.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:30.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj516 td/osd-backfill-space/6kdata 2020-11-16T20:39:30.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:30.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj517 td/osd-backfill-space/6kdata 2020-11-16T20:39:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:30.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj518 td/osd-backfill-space/6kdata 2020-11-16T20:39:30.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:30.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj519 td/osd-backfill-space/6kdata 2020-11-16T20:39:31.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:31.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj520 td/osd-backfill-space/6kdata 2020-11-16T20:39:31.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:31.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj521 td/osd-backfill-space/6kdata 2020-11-16T20:39:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj522 td/osd-backfill-space/6kdata 2020-11-16T20:39:31.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:31.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj523 td/osd-backfill-space/6kdata 2020-11-16T20:39:31.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:31.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj524 td/osd-backfill-space/6kdata 2020-11-16T20:39:31.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:31.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj525 td/osd-backfill-space/6kdata 2020-11-16T20:39:32.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:32.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj526 td/osd-backfill-space/6kdata 2020-11-16T20:39:32.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:32.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj527 td/osd-backfill-space/6kdata 2020-11-16T20:39:32.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:32.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj528 td/osd-backfill-space/6kdata 2020-11-16T20:39:32.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:32.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj529 td/osd-backfill-space/6kdata 2020-11-16T20:39:32.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:32.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj530 td/osd-backfill-space/6kdata 2020-11-16T20:39:32.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:32.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj531 td/osd-backfill-space/6kdata 2020-11-16T20:39:32.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:32.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj532 td/osd-backfill-space/6kdata 2020-11-16T20:39:33.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:33.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj533 td/osd-backfill-space/6kdata 2020-11-16T20:39:33.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:33.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj534 td/osd-backfill-space/6kdata 2020-11-16T20:39:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj535 td/osd-backfill-space/6kdata 2020-11-16T20:39:33.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:33.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj536 td/osd-backfill-space/6kdata 2020-11-16T20:39:33.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:33.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj537 td/osd-backfill-space/6kdata 2020-11-16T20:39:33.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:33.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj538 td/osd-backfill-space/6kdata 2020-11-16T20:39:33.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:33.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj539 td/osd-backfill-space/6kdata 2020-11-16T20:39:34.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:34.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj540 td/osd-backfill-space/6kdata 2020-11-16T20:39:34.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:34.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj541 td/osd-backfill-space/6kdata 2020-11-16T20:39:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj542 td/osd-backfill-space/6kdata 2020-11-16T20:39:34.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:34.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj543 td/osd-backfill-space/6kdata 2020-11-16T20:39:34.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:34.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj544 td/osd-backfill-space/6kdata 2020-11-16T20:39:34.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:34.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj545 td/osd-backfill-space/6kdata 2020-11-16T20:39:35.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:35.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj546 td/osd-backfill-space/6kdata 2020-11-16T20:39:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:35.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj547 td/osd-backfill-space/6kdata 2020-11-16T20:39:35.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:35.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj548 td/osd-backfill-space/6kdata 2020-11-16T20:39:35.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:35.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj549 td/osd-backfill-space/6kdata 2020-11-16T20:39:35.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:35.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj550 td/osd-backfill-space/6kdata 2020-11-16T20:39:35.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:35.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj551 td/osd-backfill-space/6kdata 2020-11-16T20:39:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj552 td/osd-backfill-space/6kdata 2020-11-16T20:39:36.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:36.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj553 td/osd-backfill-space/6kdata 2020-11-16T20:39:36.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:36.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj554 td/osd-backfill-space/6kdata 2020-11-16T20:39:36.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:36.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj555 td/osd-backfill-space/6kdata 2020-11-16T20:39:36.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:36.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj556 td/osd-backfill-space/6kdata 2020-11-16T20:39:36.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:36.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj557 td/osd-backfill-space/6kdata 2020-11-16T20:39:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj558 td/osd-backfill-space/6kdata 2020-11-16T20:39:36.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:36.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj559 td/osd-backfill-space/6kdata 2020-11-16T20:39:37.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:37.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj560 td/osd-backfill-space/6kdata 2020-11-16T20:39:37.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:37.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj561 td/osd-backfill-space/6kdata 2020-11-16T20:39:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj562 td/osd-backfill-space/6kdata 2020-11-16T20:39:37.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:37.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj563 td/osd-backfill-space/6kdata 2020-11-16T20:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj564 td/osd-backfill-space/6kdata 2020-11-16T20:39:37.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:37.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj565 td/osd-backfill-space/6kdata 2020-11-16T20:39:38.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:38.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj566 td/osd-backfill-space/6kdata 2020-11-16T20:39:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:38.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj567 td/osd-backfill-space/6kdata 2020-11-16T20:39:38.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:38.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj568 td/osd-backfill-space/6kdata 2020-11-16T20:39:38.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:38.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj569 td/osd-backfill-space/6kdata 2020-11-16T20:39:38.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:38.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj570 td/osd-backfill-space/6kdata 2020-11-16T20:39:38.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:38.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj571 td/osd-backfill-space/6kdata 2020-11-16T20:39:39.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:39.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj572 td/osd-backfill-space/6kdata 2020-11-16T20:39:39.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:39.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj573 td/osd-backfill-space/6kdata 2020-11-16T20:39:39.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:39.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj574 td/osd-backfill-space/6kdata 2020-11-16T20:39:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj575 td/osd-backfill-space/6kdata 2020-11-16T20:39:39.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:39.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj576 td/osd-backfill-space/6kdata 2020-11-16T20:39:39.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:39.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj577 td/osd-backfill-space/6kdata 2020-11-16T20:39:39.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:39.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj578 td/osd-backfill-space/6kdata 2020-11-16T20:39:40.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:40.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj579 td/osd-backfill-space/6kdata 2020-11-16T20:39:40.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:40.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj580 td/osd-backfill-space/6kdata 2020-11-16T20:39:40.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:40.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj581 td/osd-backfill-space/6kdata 2020-11-16T20:39:40.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:40.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj582 td/osd-backfill-space/6kdata 2020-11-16T20:39:40.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:40.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj583 td/osd-backfill-space/6kdata 2020-11-16T20:39:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:40.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj584 td/osd-backfill-space/6kdata 2020-11-16T20:39:40.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:40.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj585 td/osd-backfill-space/6kdata 2020-11-16T20:39:41.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:41.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj586 td/osd-backfill-space/6kdata 2020-11-16T20:39:41.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:41.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj587 td/osd-backfill-space/6kdata 2020-11-16T20:39:41.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:41.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj588 td/osd-backfill-space/6kdata 2020-11-16T20:39:41.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:41.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj589 td/osd-backfill-space/6kdata 2020-11-16T20:39:41.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:41.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj590 td/osd-backfill-space/6kdata 2020-11-16T20:39:41.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:41.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj591 td/osd-backfill-space/6kdata 2020-11-16T20:39:42.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:42.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj592 td/osd-backfill-space/6kdata 2020-11-16T20:39:42.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:42.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj593 td/osd-backfill-space/6kdata 2020-11-16T20:39:42.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:42.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj594 td/osd-backfill-space/6kdata 2020-11-16T20:39:42.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:42.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj595 td/osd-backfill-space/6kdata 2020-11-16T20:39:42.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:42.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj596 td/osd-backfill-space/6kdata 2020-11-16T20:39:42.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:42.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj597 td/osd-backfill-space/6kdata 2020-11-16T20:39:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:42.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj598 td/osd-backfill-space/6kdata 2020-11-16T20:39:43.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:43.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj599 td/osd-backfill-space/6kdata 2020-11-16T20:39:43.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2020-11-16T20:39:43.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj600 td/osd-backfill-space/6kdata 2020-11-16T20:39:43.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:603: TEST_backfill_grow: activate_osd td/osd-backfill-space 0 2020-11-16T20:39:43.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2020-11-16T20:39:43.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T20:39:43.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T20:39:43.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T20:39:43.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/0 2020-11-16T20:39:43.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T20:39:43.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:39:43.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:39:43.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:39:43.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2020-11-16T20:39:43.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2020-11-16T20:39:43.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T20:39:43.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T20:39:43.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T20:39:43.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T20:39:43.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:39:43.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:39:43.354 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:39:43.355 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:39:43.355 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:39:43.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:39:43.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:39:43.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:39:43.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T20:39:43.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T20:39:43.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:39:43.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:39:43.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:39:43.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T20:39:43.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T20:39:43.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/0 2020-11-16T20:39:43.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T20:39:43.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:39:43.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/0/whoami 2020-11-16T20:39:43.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T20:39:43.358 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:39:43.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T20:39:43.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T20:39:43.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T20:39:43.387 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:39:43.384+0000 7f7877741d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:39:43.390 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:39:43.388+0000 7f7877741d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:39:43.392 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:39:43.388+0000 7f7877741d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:39:43.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T20:39:43.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:39:43.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:39:43.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:39:43.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:39:43.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:39:43.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:39:43.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:39:43.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:39:43.766 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:39:43.943 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:39:43.940+0000 7f7877741d80 -1 Falling back to public interface 2020-11-16T20:39:44.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:39:45.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:39:45.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:39:45.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:39:45.185 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:39:45.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:39:45.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:39:45.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:39:45.588 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:39:45.584+0000 7f7877741d80 -1 osd.0 23 log_to_monitors {default=true} 2020-11-16T20:39:46.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:39:46.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:39:46.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:39:46.583 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:39:46.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:39:46.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:39:46.973 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 28 up_thru 0 down_at 26 last_clean_interval [5,23) [v2:127.0.0.1:6802/165157,v1:127.0.0.1:6803/165157] [v2:127.0.0.1:6804/165157,v1:127.0.0.1:6805/165157] exists,up 63902c2d-77a1-4c5f-bcd4-ef7a61dc5208 2020-11-16T20:39:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:39:46.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:39:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:39:46.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:605: TEST_backfill_grow: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T20:39:47.143 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T20:39:47.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:607: TEST_backfill_grow: sleep 2 2020-11-16T20:39:49.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:609: TEST_backfill_grow: wait_for_clean 2020-11-16T20:39:49.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:39:49.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:39:49.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:39:49.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:39:49.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:39:49.156 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:39:49.157 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:39:49.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:39:49.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:39:49.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:39:49.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:39:49.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:39:49.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:39:49.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:39:49.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:39:49.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:39:49.754 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:39:49.755 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:39:49.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:39:49.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:39:49.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:39:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084291 2020-11-16T20:39:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084291 2020-11-16T20:39:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-120259084291' 2020-11-16T20:39:49.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:39:49.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:39:50.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673009 2020-11-16T20:39:50.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673009 2020-11-16T20:39:50.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-120259084291 1-42949673009' 2020-11-16T20:39:50.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:39:50.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:39:50.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509487 2020-11-16T20:39:50.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509487 2020-11-16T20:39:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-120259084291 1-42949673009 2-64424509487' 2020-11-16T20:39:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:39:50.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-120259084291 2020-11-16T20:39:50.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:39:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:39:50.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-120259084291 2020-11-16T20:39:50.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:39:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084291 2020-11-16T20:39:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2020-11-16T20:39:50.271 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 120259084291 2020-11-16T20:39:50.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:39:50.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084291 -lt 120259084291 2020-11-16T20:39:50.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:39:50.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673009 2020-11-16T20:39:50.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:39:50.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:39:50.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673009 2020-11-16T20:39:50.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:39:50.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673009 2020-11-16T20:39:50.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2020-11-16T20:39:50.717 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673009 2020-11-16T20:39:50.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:39:51.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673009 -lt 42949673009 2020-11-16T20:39:51.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:39:51.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509487 2020-11-16T20:39:51.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:39:51.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:39:51.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509487 2020-11-16T20:39:51.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:39:51.150 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509487 2020-11-16T20:39:51.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509487 2020-11-16T20:39:51.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509487' 2020-11-16T20:39:51.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:39:51.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509485 -lt 64424509487 2020-11-16T20:39:51.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T20:39:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T20:39:52.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:39:52.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509488 -lt 64424509487 2020-11-16T20:39:52.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:39:52.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:39:52.972 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:39:53.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T20:39:53.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:39:53.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:39:53.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:39:53.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:39:53.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:39:53.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:39:53.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:39:53.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:39:53.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:39:53.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:39:53.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:39:54.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:39:54.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T20:39:54.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T20:39:54.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T20:39:54.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:39:54.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:39:54.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:39:54.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:39:54.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:39:54.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:39:54.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:39:54.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:39:54.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:39:54.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:39:54.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:39:54.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:39:54.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:39:54.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:39:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:39:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:39:55.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:39:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:39:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:39:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:39:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:39:55.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:39:55.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:39:55.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21328 2020-11-16T20:39:55.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21328 '!=' null 2020-11-16T20:39:55.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:39:55.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:39:55.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:39:55.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:39:55.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:39:55.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:39:55.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:39:55.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:39:55.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:39:55.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:39:55.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:39:56.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:39:56.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:39:56.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:39:56.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:39:56.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:39:56.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:39:56.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:39:56.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:39:56.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:39:56.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:39:56.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:39:56.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:39:56.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:39:57.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16939 2020-11-16T20:39:57.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16939 '!=' null 2020-11-16T20:39:57.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:39:57.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:39:57.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:39:57.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:39:57.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:39:57.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:39:57.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:39:57.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:39:57.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:39:57.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:39:57.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:39:57.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:39:57.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:39:57.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:39:57.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:39:58.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:39:58.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:39:58.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:39:58.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:39:58.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:39:58.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:39:58.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:39:58.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:39:58.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:39:58.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14323 2020-11-16T20:39:58.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14323 '!=' null 2020-11-16T20:39:58.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:39:58.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:39:58.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:39:58.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:39:58.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:39:58.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:39:58.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:39:58.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:39:58.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:39:58.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:39:58.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:39:59.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:39:59.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:39:59.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:39:59.078 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:39:59.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:39:59.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:39:59.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:39:59.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:39:59.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:39:59.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:39:59.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:39:59.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:39:59.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:00.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14323 2020-11-16T20:40:00.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14323 '!=' null 2020-11-16T20:40:00.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:00.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:00.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:00.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:00.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:00.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:00.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:00.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:00.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:00.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:00.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:00.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:00.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:00.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:00.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:00.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:01.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26622 2020-11-16T20:40:01.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26622 '!=' null 2020-11-16T20:40:01.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:01.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:01.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:01.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:01.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:01.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:01.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:01.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:01.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:01.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:01.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:01.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:01.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:01.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:01.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:02.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:02.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:02.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:02.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:02.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:02.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:02.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:02.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:02.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23549 2020-11-16T20:40:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23549 '!=' null 2020-11-16T20:40:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:02.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:02.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:02.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:02.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:02.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:02.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:03.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:03.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:03.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:03.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=36348 2020-11-16T20:40:29.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 36348 '!=' null 2020-11-16T20:40:29.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=36348 2020-11-16T20:40:29.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 36348 '!=' null 2020-11-16T20:40:29.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27133 2020-11-16T20:40:29.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27133 '!=' null 2020-11-16T20:40:29.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27133 2020-11-16T20:40:29.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27133 '!=' null 2020-11-16T20:40:29.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27133 2020-11-16T20:40:29.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27133 '!=' null 2020-11-16T20:40:29.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40955 2020-11-16T20:40:29.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40955 '!=' null 2020-11-16T20:40:29.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26621 2020-11-16T20:40:29.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26621 '!=' null 2020-11-16T20:40:29.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=39419 2020-11-16T20:40:29.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 39419 '!=' null 2020-11-16T20:40:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=39419 2020-11-16T20:40:29.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 39419 '!=' null 2020-11-16T20:40:29.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26621 2020-11-16T20:40:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26621 '!=' null 2020-11-16T20:40:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26621 2020-11-16T20:40:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26621 '!=' null 2020-11-16T20:40:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40955 2020-11-16T20:40:29.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40955 '!=' null 2020-11-16T20:40:29.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40955 2020-11-16T20:40:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40955 '!=' null 2020-11-16T20:40:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27133 2020-11-16T20:40:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27133 '!=' null 2020-11-16T20:40:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40955 2020-11-16T20:40:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40955 '!=' null 2020-11-16T20:40:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28156 2020-11-16T20:40:29.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28156 '!=' null 2020-11-16T20:40:29.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28156 2020-11-16T20:40:29.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28156 '!=' null 2020-11-16T20:40:29.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:29.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:29.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:29.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:29.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:29.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:29.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:29.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:29.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:29.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:29.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28156 2020-11-16T20:40:29.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28156 '!=' null 2020-11-16T20:40:29.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:29.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:29.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:29.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:29.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:29.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:29.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:29.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:29.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:29.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:29.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:29.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:29.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:29.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:29.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:30.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:30.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:30.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:30.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:30.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:30.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42490 2020-11-16T20:40:30.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42490 '!=' null 2020-11-16T20:40:30.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:30.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:30.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:31.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:31.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:31.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:31.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:31.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:31.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:31.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:31.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:31.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:31.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:31.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:31.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:31.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:31.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:31.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:31.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:31.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:31.906 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:31.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:32.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28156 2020-11-16T20:40:32.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28156 '!=' null 2020-11-16T20:40:32.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:32.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:32.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:32.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:32.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:32.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:32.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:32.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:32.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:32.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:32.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:32.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:32.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:32.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:32.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:33.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:33.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:33.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:33.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:33.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28156 2020-11-16T20:40:33.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28156 '!=' null 2020-11-16T20:40:33.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:33.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:33.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:34.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:34.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:34.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:34.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:34.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:34.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:34.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:34.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:34.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:34.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:34.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:34.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:34.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:34.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:34.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:34.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:34.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:34.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:35.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40954 2020-11-16T20:40:35.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40954 '!=' null 2020-11-16T20:40:35.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:35.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:35.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:35.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:35.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:35.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:35.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:35.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:35.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:35.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:35.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:35.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:35.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:35.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:36.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:36.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:36.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27132 2020-11-16T20:40:36.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27132 '!=' null 2020-11-16T20:40:36.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:36.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:36.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:36.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:36.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:36.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:36.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:36.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:36.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:37.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:37.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:37.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:37.263 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:37.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:37.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:37.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:37.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:37.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:37.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:37.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:37.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:37.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:38.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27132 2020-11-16T20:40:38.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27132 '!=' null 2020-11-16T20:40:38.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:38.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:38.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:38.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:38.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:38.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:38.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:38.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:38.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:38.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:38.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:38.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:38.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:39.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:39.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:39.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27132 2020-11-16T20:40:39.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27132 '!=' null 2020-11-16T20:40:39.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:39.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:39.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:39.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:39.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:39.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:39.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:39.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:39.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:39.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:40.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:40.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:40.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:40.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:40.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:40.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:40.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:40.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:40.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:40.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:40.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:40.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:40.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:41.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41467 2020-11-16T20:40:41.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41467 '!=' null 2020-11-16T20:40:41.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:41.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:41.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:41.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:41.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:41.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:41.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:41.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:41.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:41.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:41.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:41.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:41.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:41.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:41.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:42.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:42.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:42.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:42.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:42.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:42.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:42.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:42.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:42.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:42.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27133 2020-11-16T20:40:42.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27133 '!=' null 2020-11-16T20:40:42.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:42.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:42.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:42.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:42.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:42.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:42.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:42.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:42.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:42.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:42.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:43.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:43.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:43.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:43.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:43.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:43.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:43.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:43.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:43.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:43.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:43.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:43.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:43.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:44.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27133 2020-11-16T20:40:44.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27133 '!=' null 2020-11-16T20:40:44.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:44.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:44.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:44.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:44.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:44.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:44.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:44.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:44.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:44.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:44.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:44.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:44.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:44.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:44.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:45.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:45.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:45.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:45.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:45.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:45.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:45.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:45.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:45.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:45.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40445 2020-11-16T20:40:45.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40445 '!=' null 2020-11-16T20:40:45.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:45.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:45.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:45.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:45.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:45.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:45.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:45.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:45.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:45.906 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:46.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:46.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:46.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:46.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:46.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:46.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:46.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:46.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:46.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:46.778 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:47.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27646 2020-11-16T20:40:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27646 '!=' null 2020-11-16T20:40:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:47.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:47.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:47.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:47.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:47.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:47.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:47.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:47.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:47.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:47.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:47.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:47.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:47.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:48.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:48.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:48.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27646 2020-11-16T20:40:48.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27646 '!=' null 2020-11-16T20:40:48.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:48.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:48.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:49.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:49.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:49.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:49.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:49.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:49.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:49.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:49.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:49.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:49.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:49.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:49.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:49.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:50.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27646 2020-11-16T20:40:50.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27646 '!=' null 2020-11-16T20:40:50.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:50.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:50.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:50.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:50.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:50.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:50.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:50.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:50.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:50.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:50.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:50.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:50.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:50.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:51.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:51.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:51.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41469 2020-11-16T20:40:51.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41469 '!=' null 2020-11-16T20:40:51.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:51.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:51.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:51.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:51.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:51.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:51.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:51.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:51.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:51.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:51.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:51.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:52.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:52.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:52.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:52.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:52.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:52.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:52.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:52.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:52.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:52.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:52.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:52.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:52.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27134 2020-11-16T20:40:52.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27134 '!=' null 2020-11-16T20:40:52.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:52.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:52.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:53.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:53.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:53.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:53.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:53.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:53.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:53.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:53.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:53.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:54.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41469 2020-11-16T20:40:54.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41469 '!=' null 2020-11-16T20:40:54.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:54.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:54.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:54.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:54.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:54.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:54.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:54.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:54.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:54.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:54.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:54.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:54.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:54.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:54.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:55.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:55.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:55.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:55.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:55.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:55.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:55.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:55.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:55.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:55.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41469 2020-11-16T20:40:55.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41469 '!=' null 2020-11-16T20:40:55.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:55.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:55.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:56.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:56.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:56.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:56.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:56.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:56.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:56.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:56.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:56.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:56.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:56.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:56.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:56.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:56.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:56.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:56.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:56.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:56.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:56.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:56.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:56.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:57.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28157 2020-11-16T20:40:57.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28157 '!=' null 2020-11-16T20:40:57.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:57.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:57.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:57.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:57.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:57.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:57.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:57.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:57.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:57.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:57.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:57.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:57.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:57.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:57.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:58.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:40:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28157 2020-11-16T20:40:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28157 '!=' null 2020-11-16T20:40:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:40:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:40:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:40:58.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:40:58.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:40:58.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:40:58.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:40:58.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:40:58.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:40:58.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:40:58.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:40:59.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:40:59.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:40:59.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:40:59.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:40:59.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:40:59.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:40:59.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:40:59.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:40:59.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:40:59.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:40:59.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:40:59.806 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:40:59.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:00.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42491 2020-11-16T20:41:00.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42491 '!=' null 2020-11-16T20:41:00.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:00.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:00.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:00.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:00.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:00.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:00.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:00.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:00.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:00.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:00.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:00.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:00.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:00.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:00.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:01.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:01.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:01.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:01.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:01.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:01.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:01.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:01.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:01.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:01.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42491 2020-11-16T20:41:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42491 '!=' null 2020-11-16T20:41:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:01.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:01.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:01.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:01.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:02.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:02.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:02.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:02.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:02.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:02.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:02.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:02.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:02.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:02.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:02.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:02.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:02.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:03.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28668 2020-11-16T20:41:03.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28668 '!=' null 2020-11-16T20:41:03.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:03.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:03.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:03.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:03.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:03.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:03.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:03.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:03.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:03.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:03.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:03.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:03.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:03.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:03.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:04.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:04.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:04.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:04.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41978 2020-11-16T20:41:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41978 '!=' null 2020-11-16T20:41:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:04.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:04.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:04.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:04.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:04.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:04.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:04.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:04.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:05.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:05.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:05.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:05.239 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:05.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:05.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:05.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:05.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:05.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:05.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:05.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:05.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:05.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:06.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41978 2020-11-16T20:41:06.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41978 '!=' null 2020-11-16T20:41:06.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:06.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:06.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:06.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:06.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:06.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:06.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:06.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:06.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:06.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:06.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:06.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:06.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:06.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:06.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:07.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27644 2020-11-16T20:41:07.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27644 '!=' null 2020-11-16T20:41:07.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:07.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:07.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:07.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:07.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:07.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:07.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:07.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:07.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:07.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:07.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:08.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:08.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:08.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:08.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:08.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:08.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:08.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:08.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:08.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:08.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:08.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:08.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:08.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:09.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27644 2020-11-16T20:41:09.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27644 '!=' null 2020-11-16T20:41:09.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:09.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:09.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:09.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:09.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:09.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:09.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:09.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:09.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:09.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:09.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:09.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:09.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:09.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:09.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:09.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:09.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:09.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:09.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:09.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:09.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:09.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:09.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:09.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:10.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40955 2020-11-16T20:41:10.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40955 '!=' null 2020-11-16T20:41:10.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:10.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:10.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:10.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:10.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:10.588 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:10.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:10.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:10.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:10.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:10.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:10.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:10.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:10.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:10.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:11.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:11.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:11.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:11.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:11.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:11.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:11.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40955 2020-11-16T20:41:11.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40955 '!=' null 2020-11-16T20:41:11.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:11.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:11.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:12.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:12.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:12.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:12.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:12.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:12.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:12.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:12.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:12.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:12.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:12.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:12.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:13.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:13.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:13.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:13.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:13.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:13.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:13.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:13.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:13.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26621 2020-11-16T20:41:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26621 '!=' null 2020-11-16T20:41:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:13.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:13.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:13.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:13.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:13.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:13.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:13.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:13.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:13.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:13.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:13.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:14.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:14.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:14.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:14.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:14.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:14.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:14.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:14.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:14.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:14.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=30717 2020-11-16T20:41:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 30717 '!=' null 2020-11-16T20:41:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:15.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:15.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:15.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:15.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:15.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:15.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:15.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:15.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:15.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:15.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:15.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:15.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:15.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:15.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:15.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:15.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:15.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:15.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:15.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:15.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:15.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:16.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=38399 2020-11-16T20:41:16.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 38399 '!=' null 2020-11-16T20:41:16.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:16.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:16.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:16.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:16.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:16.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:16.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:16.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:16.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:16.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:16.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:16.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:16.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:16.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:16.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:17.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:17.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:17.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=38399 2020-11-16T20:41:17.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 38399 '!=' null 2020-11-16T20:41:17.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:17.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:17.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:17.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:17.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:17.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:17.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:17.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:17.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:17.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:17.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:18.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:18.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:18.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:18.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:18.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:18.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:18.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:18.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:19.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18430 2020-11-16T20:41:19.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18430 '!=' null 2020-11-16T20:41:19.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:19.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:19.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:19.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:19.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:19.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:19.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:19.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:19.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:19.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T20:41:19.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:19.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:19.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:20.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T20:41:20.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T20:41:20.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T20:41:20.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T20:41:20.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T20:41:20.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T20:41:20.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T20:41:20.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T20:41:20.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T20:41:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18430 2020-11-16T20:41:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18430 '!=' null 2020-11-16T20:41:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T20:41:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T20:41:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T20:41:20.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T20:41:20.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:41:20.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:41:20.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:41:20.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:41:20.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:41:20.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:41:20.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:41:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T20:41:21.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:41:21.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:41:21.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:41:21.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T20:41:21.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:41:21.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:41:21.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:611: TEST_backfill_grow: delete_pool test 2020-11-16T20:41:21.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T20:41:21.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T20:41:22.230 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T20:41:22.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:612: TEST_backfill_grow: kill_daemons td/osd-backfill-space 2020-11-16T20:41:22.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:41:22.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:41:22.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:41:22.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:41:22.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:41:22.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:41:22.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:613: TEST_backfill_grow: grep -q 'num_bytes mismatch' td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log 2020-11-16T20:41:22.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2020-11-16T20:41:22.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T20:41:22.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:41:22.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T20:41:22.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:41:22.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:41:22.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:41:22.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:41:22.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:41:22.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:41:22.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:41:22.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:41:22.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:41:22.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:41:22.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:41:22.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:41:22.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:41:22.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:41:22.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:41:22.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:41:22.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:41:22.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:41:22.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T20:41:22.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:41:22.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:41:22.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:41:22.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T20:41:22.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:41:22.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:41:22.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2020-11-16T20:41:22.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2020-11-16T20:41:22.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2020-11-16T20:41:22.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2020-11-16T20:41:22.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T20:41:22.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T20:41:22.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T20:41:22.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T20:41:22.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T20:41:22.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T20:41:22.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T20:41:22.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T20:41:22.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T20:41:22.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T20:41:22.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T20:41:22.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T20:41:22.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T20:41:22.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T20:41:22.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T20:41:22.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:41:22.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T20:41:22.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T20:41:22.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T20:41:22.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T20:41:22.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T20:41:22.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T20:41:22.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T20:41:22.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:41:22.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:41:22.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T20:41:22.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T20:41:22.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T20:41:22.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2020-11-16T20:41:22.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T20:41:22.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:41:22.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:41:22.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137323 2020-11-16T20:41:22.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T20:41:22.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T20:41:22.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T20:41:22.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2020-11-16T20:41:22.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_backfill_multi_partial td/osd-backfill-space 2020-11-16T20:41:22.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:429: TEST_backfill_multi_partial: local dir=td/osd-backfill-space 2020-11-16T20:41:22.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:430: TEST_backfill_multi_partial: local EC= 2020-11-16T20:41:22.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:431: TEST_backfill_multi_partial: local pools=2 2020-11-16T20:41:22.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:432: TEST_backfill_multi_partial: local OSDS=3 2020-11-16T20:41:22.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:434: TEST_backfill_multi_partial: run_mon td/osd-backfill-space a 2020-11-16T20:41:22.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2020-11-16T20:41:22.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T20:41:22.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T20:41:22.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T20:41:22.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2020-11-16T20:41:22.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2020-11-16T20:41:22.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T20:41:22.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:41:22.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:41:22.856 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:41:22.856 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:41:22.857 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:41:22.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:41:22.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T20:41:23.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T20:41:23.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T20:41:23.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:41:23.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:41:23.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T20:41:23.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T20:41:23.070 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:41:23.070 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:41:23.070 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:41:23.071 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:41:23.071 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:41:23.071 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:41:23.071 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T20:41:23.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:41:23.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get fsid 2020-11-16T20:41:23.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T20:41:23.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T20:41:23.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T20:41:23.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T20:41:23.177 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T20:41:23.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T20:41:23.177 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T20:41:23.177 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T20:41:23.178 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T20:41:23.178 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:41:23.178 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:41:23.178 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T20:41:23.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T20:41:23.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get mon_host 2020-11-16T20:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:435: TEST_backfill_multi_partial: run_mgr td/osd-backfill-space x 2020-11-16T20:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2020-11-16T20:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T20:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T20:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T20:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2020-11-16T20:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T20:41:24.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T20:41:24.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:41:24.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:41:24.217 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:41:24.217 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:41:24.217 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:41:24.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:41:24.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:41:24.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T20:41:24.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:436: TEST_backfill_multi_partial: export CEPH_ARGS 2020-11-16T20:41:24.251 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:438: TEST_backfill_multi_partial: expr 3 - 1 2020-11-16T20:41:24.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:438: TEST_backfill_multi_partial: seq 0 2 2020-11-16T20:41:24.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:438: TEST_backfill_multi_partial: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:41:24.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:440: TEST_backfill_multi_partial: run_osd td/osd-backfill-space 0 2020-11-16T20:41:24.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T20:41:24.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:41:24.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T20:41:24.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:41:24.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2020-11-16T20:41:24.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T20:41:24.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:41:24.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:41:24.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:41:24.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2020-11-16T20:41:24.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2020-11-16T20:41:24.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:41:24.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:41:24.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T20:41:24.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:41:24.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:41:24.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:41:24.258 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:41:24.258 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:41:24.258 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:41:24.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:41:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:41:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:41:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:41:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:41:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T20:41:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T20:41:24.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:41:24.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:41:24.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:41:24.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:41:24.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:41:24.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2020-11-16T20:41:24.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:41:24.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0ff826ee-5e70-4431-b3dc-8bc756612724 2020-11-16T20:41:24.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 0ff826ee-5e70-4431-b3dc-8bc756612724' 2020-11-16T20:41:24.266 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 0ff826ee-5e70-4431-b3dc-8bc756612724 2020-11-16T20:41:24.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:41:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD047JfXts7ERAA0oICwhhzE2Mb8YVis4CZSQ== 2020-11-16T20:41:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD047JfXts7ERAA0oICwhhzE2Mb8YVis4CZSQ=="}' 2020-11-16T20:41:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0ff826ee-5e70-4431-b3dc-8bc756612724 -i td/osd-backfill-space/0/new.json 2020-11-16T20:41:24.631 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:41:24.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2020-11-16T20:41:24.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD047JfXts7ERAA0oICwhhzE2Mb8YVis4CZSQ== --osd-uuid 0ff826ee-5e70-4431-b3dc-8bc756612724 2020-11-16T20:41:24.675 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:24.674+0000 7fafbc718d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:24.677 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:24.678+0000 7fafbc718d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:24.678 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:24.678+0000 7fafbc718d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:24.679 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:24.678+0000 7fafbc718d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T20:41:24.679 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:24.678+0000 7fafbc718d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T20:41:24.679 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:24.678+0000 7fafbc718d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T20:41:24.717 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:24.718+0000 7fafbc718d80 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2020-11-16T20:41:29.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2020-11-16T20:41:29.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:41:29.304 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T20:41:29.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T20:41:29.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:41:29.809 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T20:41:29.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T20:41:29.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:41:29.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:41:29.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:41:29.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:41:29.842 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:29.842+0000 7f17f0276d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:29.845 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:29.842+0000 7f17f0276d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:29.846 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:29.846+0000 7f17f0276d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:30.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T20:41:30.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:41:30.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T20:41:30.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:41:30.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:41:30.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:41:30.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:41:30.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:41:30.192 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:41:30.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:41:30.391 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:30.390+0000 7f17f0276d80 -1 Falling back to public interface 2020-11-16T20:41:30.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:41:31.220 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:31.214+0000 7f17f0276d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T20:41:31.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:41:31.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:41:31.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:41:31.582 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:41:31.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:41:31.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T20:41:32.007 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/171628,v1:127.0.0.1:6803/171628] [v2:127.0.0.1:6804/171628,v1:127.0.0.1:6805/171628] exists,up 0ff826ee-5e70-4431-b3dc-8bc756612724 2020-11-16T20:41:32.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:41:32.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:41:32.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:41:32.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:438: TEST_backfill_multi_partial: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:41:32.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:440: TEST_backfill_multi_partial: run_osd td/osd-backfill-space 1 2020-11-16T20:41:32.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T20:41:32.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:41:32.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T20:41:32.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:41:32.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2020-11-16T20:41:32.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T20:41:32.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:41:32.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:41:32.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:41:32.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2020-11-16T20:41:32.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2020-11-16T20:41:32.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:41:32.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:41:32.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T20:41:32.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:41:32.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:41:32.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:41:32.012 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:41:32.012 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:41:32.013 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:41:32.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:41:32.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:41:32.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:41:32.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:41:32.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:41:32.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T20:41:32.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T20:41:32.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:41:32.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:41:32.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:41:32.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:41:32.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:41:32.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2020-11-16T20:41:32.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:41:32.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=929f76af-e446-4277-aece-f953facd3c63 2020-11-16T20:41:32.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 929f76af-e446-4277-aece-f953facd3c63' 2020-11-16T20:41:32.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:41:32.016 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 929f76af-e446-4277-aece-f953facd3c63 2020-11-16T20:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD847JfxyRXAhAAEJHbcW5Y9WEoqx1s/+UH8w== 2020-11-16T20:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD847JfxyRXAhAAEJHbcW5Y9WEoqx1s/+UH8w=="}' 2020-11-16T20:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 929f76af-e446-4277-aece-f953facd3c63 -i td/osd-backfill-space/1/new.json 2020-11-16T20:41:32.589 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:41:32.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2020-11-16T20:41:32.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD847JfxyRXAhAAEJHbcW5Y9WEoqx1s/+UH8w== --osd-uuid 929f76af-e446-4277-aece-f953facd3c63 2020-11-16T20:41:32.635 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:32.634+0000 7fcdcbfd6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:32.638 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:32.638+0000 7fcdcbfd6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:32.639 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:32.638+0000 7fcdcbfd6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:32.640 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:32.638+0000 7fcdcbfd6d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T20:41:32.640 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:32.638+0000 7fcdcbfd6d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T20:41:32.640 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:32.638+0000 7fcdcbfd6d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T20:41:32.689 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:32.686+0000 7fcdcbfd6d80 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2020-11-16T20:41:37.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2020-11-16T20:41:37.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:41:37.360 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T20:41:37.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T20:41:37.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:41:37.872 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T20:41:37.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T20:41:37.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:41:37.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:41:37.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:41:37.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:41:37.899 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:37.898+0000 7fb578215d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:37.902 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:37.902+0000 7fb578215d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:37.903 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:37.902+0000 7fb578215d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:38.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T20:41:38.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:41:38.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T20:41:38.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:41:38.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:41:38.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:41:38.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:41:38.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:41:38.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:41:38.294 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:41:38.450 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:38.450+0000 7fb578215d80 -1 Falling back to public interface 2020-11-16T20:41:38.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:41:39.310 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:39.310+0000 7fb578215d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T20:41:39.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:41:39.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:41:39.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:41:39.682 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:41:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:41:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T20:41:40.171 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/172622,v1:127.0.0.1:6811/172622] [v2:127.0.0.1:6812/172622,v1:127.0.0.1:6813/172622] exists,up 929f76af-e446-4277-aece-f953facd3c63 2020-11-16T20:41:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:41:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:41:40.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:41:40.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:438: TEST_backfill_multi_partial: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T20:41:40.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:440: TEST_backfill_multi_partial: run_osd td/osd-backfill-space 2 2020-11-16T20:41:40.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T20:41:40.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T20:41:40.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T20:41:40.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T20:41:40.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2020-11-16T20:41:40.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T20:41:40.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T20:41:40.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T20:41:40.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T20:41:40.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2020-11-16T20:41:40.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2020-11-16T20:41:40.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T20:41:40.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T20:41:40.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T20:41:40.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T20:41:40.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T20:41:40.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T20:41:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T20:41:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T20:41:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T20:41:40.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:41:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T20:41:40.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T20:41:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T20:41:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T20:41:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T20:41:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T20:41:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T20:41:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T20:41:40.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:41:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T20:41:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T20:41:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2020-11-16T20:41:40.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T20:41:40.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=30e48a20-7415-4187-bdb6-fc1f9fb3e8fb 2020-11-16T20:41:40.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 30e48a20-7415-4187-bdb6-fc1f9fb3e8fb' 2020-11-16T20:41:40.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T20:41:40.179 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 30e48a20-7415-4187-bdb6-fc1f9fb3e8fb 2020-11-16T20:41:40.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAE5LJfufTnCxAAKjZY1eXy1mHQZH0sZj5+IA== 2020-11-16T20:41:40.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAE5LJfufTnCxAAKjZY1eXy1mHQZH0sZj5+IA=="}' 2020-11-16T20:41:40.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 30e48a20-7415-4187-bdb6-fc1f9fb3e8fb -i td/osd-backfill-space/2/new.json 2020-11-16T20:41:40.696 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:41:40.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2020-11-16T20:41:40.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAE5LJfufTnCxAAKjZY1eXy1mHQZH0sZj5+IA== --osd-uuid 30e48a20-7415-4187-bdb6-fc1f9fb3e8fb 2020-11-16T20:41:40.739 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:40.738+0000 7fb59230ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:40.742+0000 7fb59230ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:40.742+0000 7fb59230ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:40.742+0000 7fb59230ed80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T20:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:40.742+0000 7fb59230ed80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T20:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:40.742+0000 7fb59230ed80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T20:41:40.790 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:40.790+0000 7fb59230ed80 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2020-11-16T20:41:45.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2020-11-16T20:41:45.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T20:41:45.248 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T20:41:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T20:41:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T20:41:45.752 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T20:41:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T20:41:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T20:41:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T20:41:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T20:41:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T20:41:45.784 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:45.782+0000 7ffabf3cbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:45.788 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:45.786+0000 7ffabf3cbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:45.789 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:45.786+0000 7ffabf3cbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T20:41:46.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T20:41:46.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T20:41:46.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T20:41:46.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T20:41:46.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T20:41:46.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:41:46.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T20:41:46.115 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T20:41:46.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:41:46.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:41:46.335 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:46.334+0000 7ffabf3cbd80 -1 Falling back to public interface 2020-11-16T20:41:46.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:41:47.248 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:41:47.242+0000 7ffabf3cbd80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T20:41:47.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:41:47.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:41:47.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T20:41:47.541 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T20:41:47.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:41:47.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:41:48.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T20:41:49.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T20:41:49.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T20:41:49.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T20:41:49.051 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T20:41:49.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T20:41:49.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T20:41:49.455 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/173616,v1:127.0.0.1:6819/173616] [v2:127.0.0.1:6820/173616,v1:127.0.0.1:6821/173616] exists,up 30e48a20-7415-4187-bdb6-fc1f9fb3e8fb 2020-11-16T20:41:49.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T20:41:49.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T20:41:49.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T20:41:49.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:443: TEST_backfill_multi_partial: ceph osd set-backfillfull-ratio .85 2020-11-16T20:41:50.224 INFO:tasks.workunit.client.0.smithi189.stderr:osd set-backfillfull-ratio 0.85 2020-11-16T20:41:50.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:445: TEST_backfill_multi_partial: ceph osd set-require-min-compat-client luminous 2020-11-16T20:41:50.942 INFO:tasks.workunit.client.0.smithi189.stderr:set require_min_compat_client to luminous 2020-11-16T20:41:50.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:446: TEST_backfill_multi_partial: create_pool fillpool 1 1 2020-11-16T20:41:50.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create fillpool 1 1 2020-11-16T20:41:51.590 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fillpool' already exists 2020-11-16T20:41:51.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:41:52.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:447: TEST_backfill_multi_partial: ceph osd pool set fillpool size 1 --yes-i-really-mean-it 2020-11-16T20:41:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 1 2020-11-16T20:41:53.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:448: TEST_backfill_multi_partial: seq 1 2 2020-11-16T20:41:53.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:448: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T20:41:53.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:450: TEST_backfill_multi_partial: create_pool test1 1 1 2020-11-16T20:41:53.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2020-11-16T20:41:53.961 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' already exists 2020-11-16T20:41:53.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:41:54.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:451: TEST_backfill_multi_partial: ceph osd pool set test1 size 2 2020-11-16T20:41:55.815 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 2 2020-11-16T20:41:55.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:448: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T20:41:55.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:450: TEST_backfill_multi_partial: create_pool test2 1 1 2020-11-16T20:41:55.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2020-11-16T20:41:56.559 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' already exists 2020-11-16T20:41:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T20:41:57.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:451: TEST_backfill_multi_partial: ceph osd pool set test2 size 2 2020-11-16T20:41:58.534 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 2 2020-11-16T20:41:58.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:454: TEST_backfill_multi_partial: wait_for_clean 2020-11-16T20:41:58.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T20:41:58.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T20:41:58.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T20:41:58.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T20:41:58.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T20:41:58.552 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T20:41:58.552 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T20:41:58.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T20:41:58.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T20:41:58.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T20:41:58.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T20:41:58.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T20:41:58.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T20:41:58.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T20:41:58.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T20:41:59.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T20:41:59.188 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T20:41:59.188 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T20:41:59.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T20:41:59.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:41:59.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T20:41:59.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836488 2020-11-16T20:41:59.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836488 2020-11-16T20:41:59.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488' 2020-11-16T20:41:59.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:41:59.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T20:41:59.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672967 2020-11-16T20:41:59.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672967 2020-11-16T20:41:59.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2020-11-16T20:41:59.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T20:41:59.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T20:41:59.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509445 2020-11-16T20:41:59.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509445 2020-11-16T20:41:59.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509445' 2020-11-16T20:41:59.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:41:59.707 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836488 2020-11-16T20:41:59.707 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:41:59.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T20:41:59.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2020-11-16T20:41:59.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:41:59.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836488 2020-11-16T20:41:59.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2020-11-16T20:41:59.711 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836488 2020-11-16T20:41:59.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T20:42:00.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836488 2020-11-16T20:42:00.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:42:00.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672967 2020-11-16T20:42:00.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:42:00.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T20:42:00.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672967 2020-11-16T20:42:00.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:42:00.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672967 2020-11-16T20:42:00.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-11-16T20:42:00.138 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672967 2020-11-16T20:42:00.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T20:42:00.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672967 -lt 42949672967 2020-11-16T20:42:00.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T20:42:00.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509445 2020-11-16T20:42:00.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T20:42:00.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T20:42:00.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509445 2020-11-16T20:42:00.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T20:42:00.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509445 2020-11-16T20:42:00.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2020-11-16T20:42:00.553 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509445 2020-11-16T20:42:00.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T20:42:00.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509445 -lt 64424509445 2020-11-16T20:42:00.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T20:42:00.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:42:00.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:42:01.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2020-11-16T20:42:01.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T20:42:01.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T20:42:01.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T20:42:01.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T20:42:01.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T20:42:01.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T20:42:01.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T20:42:01.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2020-11-16T20:42:01.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T20:42:01.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T20:42:01.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T20:42:02.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2020-11-16T20:42:02.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T20:42:02.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T20:42:02.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:461: TEST_backfill_multi_partial: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=2611 count=1 2020-11-16T20:42:02.325 INFO:tasks.workunit.client.0.smithi189.stderr:1+0 records in 2020-11-16T20:42:02.326 INFO:tasks.workunit.client.0.smithi189.stderr:1+0 records out 2020-11-16T20:42:02.326 INFO:tasks.workunit.client.0.smithi189.stderr:2611 bytes (2.6 kB, 2.5 KiB) copied, 0.000204859 s, 12.7 MB/s 2020-11-16T20:42:02.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: seq 1 600 2020-11-16T20:42:02.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:02.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-1 td/osd-backfill-space/datafile 2020-11-16T20:42:02.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:02.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-2 td/osd-backfill-space/datafile 2020-11-16T20:42:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-3 td/osd-backfill-space/datafile 2020-11-16T20:42:02.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:02.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-4 td/osd-backfill-space/datafile 2020-11-16T20:42:02.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:02.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-5 td/osd-backfill-space/datafile 2020-11-16T20:42:02.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:02.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-6 td/osd-backfill-space/datafile 2020-11-16T20:42:02.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:02.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-7 td/osd-backfill-space/datafile 2020-11-16T20:42:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:03.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-8 td/osd-backfill-space/datafile 2020-11-16T20:42:03.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:03.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-9 td/osd-backfill-space/datafile 2020-11-16T20:42:03.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:03.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-10 td/osd-backfill-space/datafile 2020-11-16T20:42:03.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:03.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-11 td/osd-backfill-space/datafile 2020-11-16T20:42:03.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:03.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-12 td/osd-backfill-space/datafile 2020-11-16T20:42:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-13 td/osd-backfill-space/datafile 2020-11-16T20:42:03.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:03.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-14 td/osd-backfill-space/datafile 2020-11-16T20:42:03.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:03.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-15 td/osd-backfill-space/datafile 2020-11-16T20:42:03.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:03.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-16 td/osd-backfill-space/datafile 2020-11-16T20:42:03.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:03.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-17 td/osd-backfill-space/datafile 2020-11-16T20:42:04.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:04.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-18 td/osd-backfill-space/datafile 2020-11-16T20:42:04.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:04.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-19 td/osd-backfill-space/datafile 2020-11-16T20:42:04.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:04.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-20 td/osd-backfill-space/datafile 2020-11-16T20:42:04.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:04.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-21 td/osd-backfill-space/datafile 2020-11-16T20:42:04.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:04.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-22 td/osd-backfill-space/datafile 2020-11-16T20:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-23 td/osd-backfill-space/datafile 2020-11-16T20:42:04.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:04.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-24 td/osd-backfill-space/datafile 2020-11-16T20:42:04.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:04.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-25 td/osd-backfill-space/datafile 2020-11-16T20:42:04.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:04.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-26 td/osd-backfill-space/datafile 2020-11-16T20:42:04.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:04.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-27 td/osd-backfill-space/datafile 2020-11-16T20:42:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-28 td/osd-backfill-space/datafile 2020-11-16T20:42:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-29 td/osd-backfill-space/datafile 2020-11-16T20:42:05.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:05.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-30 td/osd-backfill-space/datafile 2020-11-16T20:42:05.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:05.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-31 td/osd-backfill-space/datafile 2020-11-16T20:42:05.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:05.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-32 td/osd-backfill-space/datafile 2020-11-16T20:42:05.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:05.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-33 td/osd-backfill-space/datafile 2020-11-16T20:42:05.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:05.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-34 td/osd-backfill-space/datafile 2020-11-16T20:42:05.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:05.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-35 td/osd-backfill-space/datafile 2020-11-16T20:42:05.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:05.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-36 td/osd-backfill-space/datafile 2020-11-16T20:42:05.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:05.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-37 td/osd-backfill-space/datafile 2020-11-16T20:42:06.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:06.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-38 td/osd-backfill-space/datafile 2020-11-16T20:42:06.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-39 td/osd-backfill-space/datafile 2020-11-16T20:42:06.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:06.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-40 td/osd-backfill-space/datafile 2020-11-16T20:42:06.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:06.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-41 td/osd-backfill-space/datafile 2020-11-16T20:42:06.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:06.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-42 td/osd-backfill-space/datafile 2020-11-16T20:42:06.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:06.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-43 td/osd-backfill-space/datafile 2020-11-16T20:42:06.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:06.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-44 td/osd-backfill-space/datafile 2020-11-16T20:42:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:06.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-45 td/osd-backfill-space/datafile 2020-11-16T20:42:06.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:06.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-46 td/osd-backfill-space/datafile 2020-11-16T20:42:06.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:06.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-47 td/osd-backfill-space/datafile 2020-11-16T20:42:07.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-48 td/osd-backfill-space/datafile 2020-11-16T20:42:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-49 td/osd-backfill-space/datafile 2020-11-16T20:42:07.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:07.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-50 td/osd-backfill-space/datafile 2020-11-16T20:42:07.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:07.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-51 td/osd-backfill-space/datafile 2020-11-16T20:42:07.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:07.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-52 td/osd-backfill-space/datafile 2020-11-16T20:42:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-53 td/osd-backfill-space/datafile 2020-11-16T20:42:07.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:07.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-54 td/osd-backfill-space/datafile 2020-11-16T20:42:07.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:07.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-55 td/osd-backfill-space/datafile 2020-11-16T20:42:07.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:07.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-56 td/osd-backfill-space/datafile 2020-11-16T20:42:07.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:07.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-57 td/osd-backfill-space/datafile 2020-11-16T20:42:08.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:08.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-58 td/osd-backfill-space/datafile 2020-11-16T20:42:08.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:08.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-59 td/osd-backfill-space/datafile 2020-11-16T20:42:08.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:08.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-60 td/osd-backfill-space/datafile 2020-11-16T20:42:08.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:08.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-61 td/osd-backfill-space/datafile 2020-11-16T20:42:08.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:08.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-62 td/osd-backfill-space/datafile 2020-11-16T20:42:08.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:08.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-63 td/osd-backfill-space/datafile 2020-11-16T20:42:08.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:08.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-64 td/osd-backfill-space/datafile 2020-11-16T20:42:08.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:08.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-65 td/osd-backfill-space/datafile 2020-11-16T20:42:08.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:08.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-66 td/osd-backfill-space/datafile 2020-11-16T20:42:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-67 td/osd-backfill-space/datafile 2020-11-16T20:42:09.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:09.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-68 td/osd-backfill-space/datafile 2020-11-16T20:42:09.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:09.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-69 td/osd-backfill-space/datafile 2020-11-16T20:42:09.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:09.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-70 td/osd-backfill-space/datafile 2020-11-16T20:42:09.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:09.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-71 td/osd-backfill-space/datafile 2020-11-16T20:42:09.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:09.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-72 td/osd-backfill-space/datafile 2020-11-16T20:42:09.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:09.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-73 td/osd-backfill-space/datafile 2020-11-16T20:42:09.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:09.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-74 td/osd-backfill-space/datafile 2020-11-16T20:42:09.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:09.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-75 td/osd-backfill-space/datafile 2020-11-16T20:42:09.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:09.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-76 td/osd-backfill-space/datafile 2020-11-16T20:42:09.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:09.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-77 td/osd-backfill-space/datafile 2020-11-16T20:42:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-78 td/osd-backfill-space/datafile 2020-11-16T20:42:10.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:10.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-79 td/osd-backfill-space/datafile 2020-11-16T20:42:10.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:10.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-80 td/osd-backfill-space/datafile 2020-11-16T20:42:10.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:10.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-81 td/osd-backfill-space/datafile 2020-11-16T20:42:10.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:10.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-82 td/osd-backfill-space/datafile 2020-11-16T20:42:10.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-83 td/osd-backfill-space/datafile 2020-11-16T20:42:10.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:10.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-84 td/osd-backfill-space/datafile 2020-11-16T20:42:10.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:10.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-85 td/osd-backfill-space/datafile 2020-11-16T20:42:10.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:10.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-86 td/osd-backfill-space/datafile 2020-11-16T20:42:11.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:11.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-87 td/osd-backfill-space/datafile 2020-11-16T20:42:11.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:11.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-88 td/osd-backfill-space/datafile 2020-11-16T20:42:11.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:11.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-89 td/osd-backfill-space/datafile 2020-11-16T20:42:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:11.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-90 td/osd-backfill-space/datafile 2020-11-16T20:42:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-91 td/osd-backfill-space/datafile 2020-11-16T20:42:11.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:11.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-92 td/osd-backfill-space/datafile 2020-11-16T20:42:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-93 td/osd-backfill-space/datafile 2020-11-16T20:42:11.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:11.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-94 td/osd-backfill-space/datafile 2020-11-16T20:42:11.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:11.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-95 td/osd-backfill-space/datafile 2020-11-16T20:42:11.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:11.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-96 td/osd-backfill-space/datafile 2020-11-16T20:42:12.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:12.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-97 td/osd-backfill-space/datafile 2020-11-16T20:42:12.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:12.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-98 td/osd-backfill-space/datafile 2020-11-16T20:42:12.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:12.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-99 td/osd-backfill-space/datafile 2020-11-16T20:42:12.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:12.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-100 td/osd-backfill-space/datafile 2020-11-16T20:42:12.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:12.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-101 td/osd-backfill-space/datafile 2020-11-16T20:42:12.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:12.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-102 td/osd-backfill-space/datafile 2020-11-16T20:42:12.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:12.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-103 td/osd-backfill-space/datafile 2020-11-16T20:42:12.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:12.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-104 td/osd-backfill-space/datafile 2020-11-16T20:42:12.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:12.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-105 td/osd-backfill-space/datafile 2020-11-16T20:42:12.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:12.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-106 td/osd-backfill-space/datafile 2020-11-16T20:42:12.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:12.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-107 td/osd-backfill-space/datafile 2020-11-16T20:42:13.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:13.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-108 td/osd-backfill-space/datafile 2020-11-16T20:42:13.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:13.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-109 td/osd-backfill-space/datafile 2020-11-16T20:42:13.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:13.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-110 td/osd-backfill-space/datafile 2020-11-16T20:42:13.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-111 td/osd-backfill-space/datafile 2020-11-16T20:42:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:13.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-112 td/osd-backfill-space/datafile 2020-11-16T20:42:13.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:13.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-113 td/osd-backfill-space/datafile 2020-11-16T20:42:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-114 td/osd-backfill-space/datafile 2020-11-16T20:42:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-115 td/osd-backfill-space/datafile 2020-11-16T20:42:13.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:13.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-116 td/osd-backfill-space/datafile 2020-11-16T20:42:13.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:13.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-117 td/osd-backfill-space/datafile 2020-11-16T20:42:14.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:14.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-118 td/osd-backfill-space/datafile 2020-11-16T20:42:14.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:14.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-119 td/osd-backfill-space/datafile 2020-11-16T20:42:14.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:14.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-120 td/osd-backfill-space/datafile 2020-11-16T20:42:14.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:14.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-121 td/osd-backfill-space/datafile 2020-11-16T20:42:14.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:14.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-122 td/osd-backfill-space/datafile 2020-11-16T20:42:14.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:14.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-123 td/osd-backfill-space/datafile 2020-11-16T20:42:14.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:14.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-124 td/osd-backfill-space/datafile 2020-11-16T20:42:14.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:14.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-125 td/osd-backfill-space/datafile 2020-11-16T20:42:14.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:14.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-126 td/osd-backfill-space/datafile 2020-11-16T20:42:15.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:15.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-127 td/osd-backfill-space/datafile 2020-11-16T20:42:15.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:15.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-128 td/osd-backfill-space/datafile 2020-11-16T20:42:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-129 td/osd-backfill-space/datafile 2020-11-16T20:42:15.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:15.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-130 td/osd-backfill-space/datafile 2020-11-16T20:42:15.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:15.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-131 td/osd-backfill-space/datafile 2020-11-16T20:42:15.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:15.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-132 td/osd-backfill-space/datafile 2020-11-16T20:42:15.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:15.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-133 td/osd-backfill-space/datafile 2020-11-16T20:42:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:15.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-134 td/osd-backfill-space/datafile 2020-11-16T20:42:15.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:15.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-135 td/osd-backfill-space/datafile 2020-11-16T20:42:15.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:15.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-136 td/osd-backfill-space/datafile 2020-11-16T20:42:16.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:16.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-137 td/osd-backfill-space/datafile 2020-11-16T20:42:16.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:16.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-138 td/osd-backfill-space/datafile 2020-11-16T20:42:16.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:16.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-139 td/osd-backfill-space/datafile 2020-11-16T20:42:16.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:16.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-140 td/osd-backfill-space/datafile 2020-11-16T20:42:16.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:16.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-141 td/osd-backfill-space/datafile 2020-11-16T20:42:16.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:16.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-142 td/osd-backfill-space/datafile 2020-11-16T20:42:16.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:16.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-143 td/osd-backfill-space/datafile 2020-11-16T20:42:16.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:16.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-144 td/osd-backfill-space/datafile 2020-11-16T20:42:16.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:16.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-145 td/osd-backfill-space/datafile 2020-11-16T20:42:16.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:16.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-146 td/osd-backfill-space/datafile 2020-11-16T20:42:17.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:17.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-147 td/osd-backfill-space/datafile 2020-11-16T20:42:17.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:17.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-148 td/osd-backfill-space/datafile 2020-11-16T20:42:17.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:17.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-149 td/osd-backfill-space/datafile 2020-11-16T20:42:17.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:17.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-150 td/osd-backfill-space/datafile 2020-11-16T20:42:17.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:17.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-151 td/osd-backfill-space/datafile 2020-11-16T20:42:17.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:17.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-152 td/osd-backfill-space/datafile 2020-11-16T20:42:17.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:17.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-153 td/osd-backfill-space/datafile 2020-11-16T20:42:17.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:17.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-154 td/osd-backfill-space/datafile 2020-11-16T20:42:17.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:17.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-155 td/osd-backfill-space/datafile 2020-11-16T20:42:17.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:17.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-156 td/osd-backfill-space/datafile 2020-11-16T20:42:17.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:17.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-157 td/osd-backfill-space/datafile 2020-11-16T20:42:18.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:18.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-158 td/osd-backfill-space/datafile 2020-11-16T20:42:18.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:18.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-159 td/osd-backfill-space/datafile 2020-11-16T20:42:18.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:18.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-160 td/osd-backfill-space/datafile 2020-11-16T20:42:18.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:18.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-161 td/osd-backfill-space/datafile 2020-11-16T20:42:18.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:18.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-162 td/osd-backfill-space/datafile 2020-11-16T20:42:18.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:18.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-163 td/osd-backfill-space/datafile 2020-11-16T20:42:18.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:18.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-164 td/osd-backfill-space/datafile 2020-11-16T20:42:18.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:18.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-165 td/osd-backfill-space/datafile 2020-11-16T20:42:19.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:19.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-166 td/osd-backfill-space/datafile 2020-11-16T20:42:19.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:19.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-167 td/osd-backfill-space/datafile 2020-11-16T20:42:19.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:19.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-168 td/osd-backfill-space/datafile 2020-11-16T20:42:19.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:19.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-169 td/osd-backfill-space/datafile 2020-11-16T20:42:19.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:19.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-170 td/osd-backfill-space/datafile 2020-11-16T20:42:19.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:19.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-171 td/osd-backfill-space/datafile 2020-11-16T20:42:19.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:19.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-172 td/osd-backfill-space/datafile 2020-11-16T20:42:19.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:19.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-173 td/osd-backfill-space/datafile 2020-11-16T20:42:19.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:19.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-174 td/osd-backfill-space/datafile 2020-11-16T20:42:19.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:19.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-175 td/osd-backfill-space/datafile 2020-11-16T20:42:19.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:19.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-176 td/osd-backfill-space/datafile 2020-11-16T20:42:20.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:20.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-177 td/osd-backfill-space/datafile 2020-11-16T20:42:20.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:20.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-178 td/osd-backfill-space/datafile 2020-11-16T20:42:20.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:20.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-179 td/osd-backfill-space/datafile 2020-11-16T20:42:20.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:20.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-180 td/osd-backfill-space/datafile 2020-11-16T20:42:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-181 td/osd-backfill-space/datafile 2020-11-16T20:42:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-182 td/osd-backfill-space/datafile 2020-11-16T20:42:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:20.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-183 td/osd-backfill-space/datafile 2020-11-16T20:42:20.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:20.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-184 td/osd-backfill-space/datafile 2020-11-16T20:42:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-185 td/osd-backfill-space/datafile 2020-11-16T20:42:21.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:21.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-186 td/osd-backfill-space/datafile 2020-11-16T20:42:21.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:21.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-187 td/osd-backfill-space/datafile 2020-11-16T20:42:21.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:21.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-188 td/osd-backfill-space/datafile 2020-11-16T20:42:21.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:21.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-189 td/osd-backfill-space/datafile 2020-11-16T20:42:21.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:21.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-190 td/osd-backfill-space/datafile 2020-11-16T20:42:21.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:21.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-191 td/osd-backfill-space/datafile 2020-11-16T20:42:21.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:21.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-192 td/osd-backfill-space/datafile 2020-11-16T20:42:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:21.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-193 td/osd-backfill-space/datafile 2020-11-16T20:42:21.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:21.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-194 td/osd-backfill-space/datafile 2020-11-16T20:42:21.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:21.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-195 td/osd-backfill-space/datafile 2020-11-16T20:42:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:22.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-196 td/osd-backfill-space/datafile 2020-11-16T20:42:22.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:22.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-197 td/osd-backfill-space/datafile 2020-11-16T20:42:22.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:22.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-198 td/osd-backfill-space/datafile 2020-11-16T20:42:22.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:22.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-199 td/osd-backfill-space/datafile 2020-11-16T20:42:22.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:22.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-200 td/osd-backfill-space/datafile 2020-11-16T20:42:22.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:22.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-201 td/osd-backfill-space/datafile 2020-11-16T20:42:22.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:22.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-202 td/osd-backfill-space/datafile 2020-11-16T20:42:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:22.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-203 td/osd-backfill-space/datafile 2020-11-16T20:42:22.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:22.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-204 td/osd-backfill-space/datafile 2020-11-16T20:42:22.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:22.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-205 td/osd-backfill-space/datafile 2020-11-16T20:42:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-206 td/osd-backfill-space/datafile 2020-11-16T20:42:23.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:23.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-207 td/osd-backfill-space/datafile 2020-11-16T20:42:23.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:23.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-208 td/osd-backfill-space/datafile 2020-11-16T20:42:23.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:23.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-209 td/osd-backfill-space/datafile 2020-11-16T20:42:23.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:23.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-210 td/osd-backfill-space/datafile 2020-11-16T20:42:23.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:23.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-211 td/osd-backfill-space/datafile 2020-11-16T20:42:23.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:23.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-212 td/osd-backfill-space/datafile 2020-11-16T20:42:23.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:23.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-213 td/osd-backfill-space/datafile 2020-11-16T20:42:23.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:23.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-214 td/osd-backfill-space/datafile 2020-11-16T20:42:24.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:24.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-215 td/osd-backfill-space/datafile 2020-11-16T20:42:24.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:24.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-216 td/osd-backfill-space/datafile 2020-11-16T20:42:24.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:24.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-217 td/osd-backfill-space/datafile 2020-11-16T20:42:24.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:24.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-218 td/osd-backfill-space/datafile 2020-11-16T20:42:24.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:24.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-219 td/osd-backfill-space/datafile 2020-11-16T20:42:24.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:24.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-220 td/osd-backfill-space/datafile 2020-11-16T20:42:24.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:24.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-221 td/osd-backfill-space/datafile 2020-11-16T20:42:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-222 td/osd-backfill-space/datafile 2020-11-16T20:42:24.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:24.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-223 td/osd-backfill-space/datafile 2020-11-16T20:42:24.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:24.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-224 td/osd-backfill-space/datafile 2020-11-16T20:42:25.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:25.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-225 td/osd-backfill-space/datafile 2020-11-16T20:42:25.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:25.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-226 td/osd-backfill-space/datafile 2020-11-16T20:42:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-227 td/osd-backfill-space/datafile 2020-11-16T20:42:25.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:25.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-228 td/osd-backfill-space/datafile 2020-11-16T20:42:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-229 td/osd-backfill-space/datafile 2020-11-16T20:42:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-230 td/osd-backfill-space/datafile 2020-11-16T20:42:25.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:25.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-231 td/osd-backfill-space/datafile 2020-11-16T20:42:25.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:25.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-232 td/osd-backfill-space/datafile 2020-11-16T20:42:25.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:25.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-233 td/osd-backfill-space/datafile 2020-11-16T20:42:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:25.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-234 td/osd-backfill-space/datafile 2020-11-16T20:42:25.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:25.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-235 td/osd-backfill-space/datafile 2020-11-16T20:42:26.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:26.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-236 td/osd-backfill-space/datafile 2020-11-16T20:42:26.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:26.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-237 td/osd-backfill-space/datafile 2020-11-16T20:42:26.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:26.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-238 td/osd-backfill-space/datafile 2020-11-16T20:42:26.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:26.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-239 td/osd-backfill-space/datafile 2020-11-16T20:42:26.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:26.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-240 td/osd-backfill-space/datafile 2020-11-16T20:42:26.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:26.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-241 td/osd-backfill-space/datafile 2020-11-16T20:42:26.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:26.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-242 td/osd-backfill-space/datafile 2020-11-16T20:42:26.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:26.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-243 td/osd-backfill-space/datafile 2020-11-16T20:42:26.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:26.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-244 td/osd-backfill-space/datafile 2020-11-16T20:42:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-245 td/osd-backfill-space/datafile 2020-11-16T20:42:27.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:27.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-246 td/osd-backfill-space/datafile 2020-11-16T20:42:27.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:27.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-247 td/osd-backfill-space/datafile 2020-11-16T20:42:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-248 td/osd-backfill-space/datafile 2020-11-16T20:42:27.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:27.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-249 td/osd-backfill-space/datafile 2020-11-16T20:42:27.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:27.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-250 td/osd-backfill-space/datafile 2020-11-16T20:42:27.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:27.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-251 td/osd-backfill-space/datafile 2020-11-16T20:42:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-252 td/osd-backfill-space/datafile 2020-11-16T20:42:27.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:27.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-253 td/osd-backfill-space/datafile 2020-11-16T20:42:27.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:27.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-254 td/osd-backfill-space/datafile 2020-11-16T20:42:28.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:28.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-255 td/osd-backfill-space/datafile 2020-11-16T20:42:28.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:28.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-256 td/osd-backfill-space/datafile 2020-11-16T20:42:28.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:28.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-257 td/osd-backfill-space/datafile 2020-11-16T20:42:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:28.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-258 td/osd-backfill-space/datafile 2020-11-16T20:42:28.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:28.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-259 td/osd-backfill-space/datafile 2020-11-16T20:42:28.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:28.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-260 td/osd-backfill-space/datafile 2020-11-16T20:42:28.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:28.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-261 td/osd-backfill-space/datafile 2020-11-16T20:42:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:29.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-262 td/osd-backfill-space/datafile 2020-11-16T20:42:29.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:29.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-263 td/osd-backfill-space/datafile 2020-11-16T20:42:29.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:29.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-264 td/osd-backfill-space/datafile 2020-11-16T20:42:29.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:29.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-265 td/osd-backfill-space/datafile 2020-11-16T20:42:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:29.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-266 td/osd-backfill-space/datafile 2020-11-16T20:42:29.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:29.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-267 td/osd-backfill-space/datafile 2020-11-16T20:42:29.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:29.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-268 td/osd-backfill-space/datafile 2020-11-16T20:42:29.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:29.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-269 td/osd-backfill-space/datafile 2020-11-16T20:42:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-270 td/osd-backfill-space/datafile 2020-11-16T20:42:30.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:30.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-271 td/osd-backfill-space/datafile 2020-11-16T20:42:30.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:30.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-272 td/osd-backfill-space/datafile 2020-11-16T20:42:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-273 td/osd-backfill-space/datafile 2020-11-16T20:42:30.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:30.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-274 td/osd-backfill-space/datafile 2020-11-16T20:42:30.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:30.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-275 td/osd-backfill-space/datafile 2020-11-16T20:42:30.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:30.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-276 td/osd-backfill-space/datafile 2020-11-16T20:42:30.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:30.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-277 td/osd-backfill-space/datafile 2020-11-16T20:42:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-278 td/osd-backfill-space/datafile 2020-11-16T20:42:30.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:30.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-279 td/osd-backfill-space/datafile 2020-11-16T20:42:30.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:30.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-280 td/osd-backfill-space/datafile 2020-11-16T20:42:31.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:31.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-281 td/osd-backfill-space/datafile 2020-11-16T20:42:31.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:31.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-282 td/osd-backfill-space/datafile 2020-11-16T20:42:31.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:31.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-283 td/osd-backfill-space/datafile 2020-11-16T20:42:31.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:31.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-284 td/osd-backfill-space/datafile 2020-11-16T20:42:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:31.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-285 td/osd-backfill-space/datafile 2020-11-16T20:42:31.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:31.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-286 td/osd-backfill-space/datafile 2020-11-16T20:42:31.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:31.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-287 td/osd-backfill-space/datafile 2020-11-16T20:42:31.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:31.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-288 td/osd-backfill-space/datafile 2020-11-16T20:42:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:31.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-289 td/osd-backfill-space/datafile 2020-11-16T20:42:31.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:31.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-290 td/osd-backfill-space/datafile 2020-11-16T20:42:32.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:32.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-291 td/osd-backfill-space/datafile 2020-11-16T20:42:32.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:32.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-292 td/osd-backfill-space/datafile 2020-11-16T20:42:32.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:32.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-293 td/osd-backfill-space/datafile 2020-11-16T20:42:32.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:32.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-294 td/osd-backfill-space/datafile 2020-11-16T20:42:32.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:32.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-295 td/osd-backfill-space/datafile 2020-11-16T20:42:32.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:32.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-296 td/osd-backfill-space/datafile 2020-11-16T20:42:32.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:32.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-297 td/osd-backfill-space/datafile 2020-11-16T20:42:32.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:32.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-298 td/osd-backfill-space/datafile 2020-11-16T20:42:32.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:32.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-299 td/osd-backfill-space/datafile 2020-11-16T20:42:32.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:32.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-300 td/osd-backfill-space/datafile 2020-11-16T20:42:33.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:33.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-301 td/osd-backfill-space/datafile 2020-11-16T20:42:33.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:33.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-302 td/osd-backfill-space/datafile 2020-11-16T20:42:33.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:33.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-303 td/osd-backfill-space/datafile 2020-11-16T20:42:33.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:33.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-304 td/osd-backfill-space/datafile 2020-11-16T20:42:33.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:33.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-305 td/osd-backfill-space/datafile 2020-11-16T20:42:33.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:33.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-306 td/osd-backfill-space/datafile 2020-11-16T20:42:33.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:33.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-307 td/osd-backfill-space/datafile 2020-11-16T20:42:33.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:33.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-308 td/osd-backfill-space/datafile 2020-11-16T20:42:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-309 td/osd-backfill-space/datafile 2020-11-16T20:42:34.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:34.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-310 td/osd-backfill-space/datafile 2020-11-16T20:42:34.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:34.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-311 td/osd-backfill-space/datafile 2020-11-16T20:42:34.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:34.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-312 td/osd-backfill-space/datafile 2020-11-16T20:42:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-313 td/osd-backfill-space/datafile 2020-11-16T20:42:34.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:34.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-314 td/osd-backfill-space/datafile 2020-11-16T20:42:34.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:34.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-315 td/osd-backfill-space/datafile 2020-11-16T20:42:34.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:34.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-316 td/osd-backfill-space/datafile 2020-11-16T20:42:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:34.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-317 td/osd-backfill-space/datafile 2020-11-16T20:42:34.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:34.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-318 td/osd-backfill-space/datafile 2020-11-16T20:42:35.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:35.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-319 td/osd-backfill-space/datafile 2020-11-16T20:42:35.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:35.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-320 td/osd-backfill-space/datafile 2020-11-16T20:42:35.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:35.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-321 td/osd-backfill-space/datafile 2020-11-16T20:42:35.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:35.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-322 td/osd-backfill-space/datafile 2020-11-16T20:42:35.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:35.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-323 td/osd-backfill-space/datafile 2020-11-16T20:42:35.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:35.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-324 td/osd-backfill-space/datafile 2020-11-16T20:42:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:35.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-325 td/osd-backfill-space/datafile 2020-11-16T20:42:35.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:35.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-326 td/osd-backfill-space/datafile 2020-11-16T20:42:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-327 td/osd-backfill-space/datafile 2020-11-16T20:42:35.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:35.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-328 td/osd-backfill-space/datafile 2020-11-16T20:42:36.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:36.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-329 td/osd-backfill-space/datafile 2020-11-16T20:42:36.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:36.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-330 td/osd-backfill-space/datafile 2020-11-16T20:42:36.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:36.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-331 td/osd-backfill-space/datafile 2020-11-16T20:42:36.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:36.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-332 td/osd-backfill-space/datafile 2020-11-16T20:42:36.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:36.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-333 td/osd-backfill-space/datafile 2020-11-16T20:42:36.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:36.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-334 td/osd-backfill-space/datafile 2020-11-16T20:42:36.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:36.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-335 td/osd-backfill-space/datafile 2020-11-16T20:42:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-336 td/osd-backfill-space/datafile 2020-11-16T20:42:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:36.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-337 td/osd-backfill-space/datafile 2020-11-16T20:42:36.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:36.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-338 td/osd-backfill-space/datafile 2020-11-16T20:42:37.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:37.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-339 td/osd-backfill-space/datafile 2020-11-16T20:42:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-340 td/osd-backfill-space/datafile 2020-11-16T20:42:37.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:37.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-341 td/osd-backfill-space/datafile 2020-11-16T20:42:37.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:37.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-342 td/osd-backfill-space/datafile 2020-11-16T20:42:37.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:37.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-343 td/osd-backfill-space/datafile 2020-11-16T20:42:37.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:37.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-344 td/osd-backfill-space/datafile 2020-11-16T20:42:37.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:37.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-345 td/osd-backfill-space/datafile 2020-11-16T20:42:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:37.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-346 td/osd-backfill-space/datafile 2020-11-16T20:42:37.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:37.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-347 td/osd-backfill-space/datafile 2020-11-16T20:42:38.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:38.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-348 td/osd-backfill-space/datafile 2020-11-16T20:42:38.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:38.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-349 td/osd-backfill-space/datafile 2020-11-16T20:42:38.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:38.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-350 td/osd-backfill-space/datafile 2020-11-16T20:42:38.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:38.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-351 td/osd-backfill-space/datafile 2020-11-16T20:42:38.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:38.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-352 td/osd-backfill-space/datafile 2020-11-16T20:42:38.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:38.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-353 td/osd-backfill-space/datafile 2020-11-16T20:42:38.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:38.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-354 td/osd-backfill-space/datafile 2020-11-16T20:42:38.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:38.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-355 td/osd-backfill-space/datafile 2020-11-16T20:42:38.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:38.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-356 td/osd-backfill-space/datafile 2020-11-16T20:42:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-357 td/osd-backfill-space/datafile 2020-11-16T20:42:39.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:39.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-358 td/osd-backfill-space/datafile 2020-11-16T20:42:39.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:39.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-359 td/osd-backfill-space/datafile 2020-11-16T20:42:39.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:39.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-360 td/osd-backfill-space/datafile 2020-11-16T20:42:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-361 td/osd-backfill-space/datafile 2020-11-16T20:42:39.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:39.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-362 td/osd-backfill-space/datafile 2020-11-16T20:42:39.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:39.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-363 td/osd-backfill-space/datafile 2020-11-16T20:42:39.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:39.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-364 td/osd-backfill-space/datafile 2020-11-16T20:42:39.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:39.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-365 td/osd-backfill-space/datafile 2020-11-16T20:42:39.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:39.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-366 td/osd-backfill-space/datafile 2020-11-16T20:42:40.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:40.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-367 td/osd-backfill-space/datafile 2020-11-16T20:42:40.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:40.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-368 td/osd-backfill-space/datafile 2020-11-16T20:42:40.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:40.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-369 td/osd-backfill-space/datafile 2020-11-16T20:42:40.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:40.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-370 td/osd-backfill-space/datafile 2020-11-16T20:42:40.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:40.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-371 td/osd-backfill-space/datafile 2020-11-16T20:42:40.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:40.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-372 td/osd-backfill-space/datafile 2020-11-16T20:42:40.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:40.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-373 td/osd-backfill-space/datafile 2020-11-16T20:42:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-374 td/osd-backfill-space/datafile 2020-11-16T20:42:40.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-375 td/osd-backfill-space/datafile 2020-11-16T20:42:40.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:40.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-376 td/osd-backfill-space/datafile 2020-11-16T20:42:41.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:41.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-377 td/osd-backfill-space/datafile 2020-11-16T20:42:41.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:41.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-378 td/osd-backfill-space/datafile 2020-11-16T20:42:41.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:41.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-379 td/osd-backfill-space/datafile 2020-11-16T20:42:41.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:41.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-380 td/osd-backfill-space/datafile 2020-11-16T20:42:41.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:41.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-381 td/osd-backfill-space/datafile 2020-11-16T20:42:41.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:41.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-382 td/osd-backfill-space/datafile 2020-11-16T20:42:41.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:41.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-383 td/osd-backfill-space/datafile 2020-11-16T20:42:41.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:41.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-384 td/osd-backfill-space/datafile 2020-11-16T20:42:41.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:41.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-385 td/osd-backfill-space/datafile 2020-11-16T20:42:41.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:41.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-386 td/osd-backfill-space/datafile 2020-11-16T20:42:42.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:42.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-387 td/osd-backfill-space/datafile 2020-11-16T20:42:42.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:42.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-388 td/osd-backfill-space/datafile 2020-11-16T20:42:42.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:42.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-389 td/osd-backfill-space/datafile 2020-11-16T20:42:42.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:42.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-390 td/osd-backfill-space/datafile 2020-11-16T20:42:42.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:42.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-391 td/osd-backfill-space/datafile 2020-11-16T20:42:42.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:42.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-392 td/osd-backfill-space/datafile 2020-11-16T20:42:42.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:42.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-393 td/osd-backfill-space/datafile 2020-11-16T20:42:42.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:42.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-394 td/osd-backfill-space/datafile 2020-11-16T20:42:42.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:42.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-395 td/osd-backfill-space/datafile 2020-11-16T20:42:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:42.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-396 td/osd-backfill-space/datafile 2020-11-16T20:42:43.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:43.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-397 td/osd-backfill-space/datafile 2020-11-16T20:42:43.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:43.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-398 td/osd-backfill-space/datafile 2020-11-16T20:42:43.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:43.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-399 td/osd-backfill-space/datafile 2020-11-16T20:42:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:43.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-400 td/osd-backfill-space/datafile 2020-11-16T20:42:43.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:43.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-401 td/osd-backfill-space/datafile 2020-11-16T20:42:43.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:43.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-402 td/osd-backfill-space/datafile 2020-11-16T20:42:43.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:43.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-403 td/osd-backfill-space/datafile 2020-11-16T20:42:43.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:43.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-404 td/osd-backfill-space/datafile 2020-11-16T20:42:43.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:43.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-405 td/osd-backfill-space/datafile 2020-11-16T20:42:44.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:44.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-406 td/osd-backfill-space/datafile 2020-11-16T20:42:44.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:44.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-407 td/osd-backfill-space/datafile 2020-11-16T20:42:44.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:44.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-408 td/osd-backfill-space/datafile 2020-11-16T20:42:44.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:44.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-409 td/osd-backfill-space/datafile 2020-11-16T20:42:44.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:44.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-410 td/osd-backfill-space/datafile 2020-11-16T20:42:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:44.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-411 td/osd-backfill-space/datafile 2020-11-16T20:42:44.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:44.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-412 td/osd-backfill-space/datafile 2020-11-16T20:42:44.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:44.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-413 td/osd-backfill-space/datafile 2020-11-16T20:42:44.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:44.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-414 td/osd-backfill-space/datafile 2020-11-16T20:42:44.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:44.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-415 td/osd-backfill-space/datafile 2020-11-16T20:42:45.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:45.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-416 td/osd-backfill-space/datafile 2020-11-16T20:42:45.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:45.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-417 td/osd-backfill-space/datafile 2020-11-16T20:42:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-418 td/osd-backfill-space/datafile 2020-11-16T20:42:45.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:45.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-419 td/osd-backfill-space/datafile 2020-11-16T20:42:45.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:45.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-420 td/osd-backfill-space/datafile 2020-11-16T20:42:45.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:45.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-421 td/osd-backfill-space/datafile 2020-11-16T20:42:45.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:45.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-422 td/osd-backfill-space/datafile 2020-11-16T20:42:45.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:45.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-423 td/osd-backfill-space/datafile 2020-11-16T20:42:45.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:45.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-424 td/osd-backfill-space/datafile 2020-11-16T20:42:45.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:45.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-425 td/osd-backfill-space/datafile 2020-11-16T20:42:46.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:46.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-426 td/osd-backfill-space/datafile 2020-11-16T20:42:46.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:46.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-427 td/osd-backfill-space/datafile 2020-11-16T20:42:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-428 td/osd-backfill-space/datafile 2020-11-16T20:42:46.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:46.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-429 td/osd-backfill-space/datafile 2020-11-16T20:42:46.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:46.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-430 td/osd-backfill-space/datafile 2020-11-16T20:42:46.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-431 td/osd-backfill-space/datafile 2020-11-16T20:42:46.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:46.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-432 td/osd-backfill-space/datafile 2020-11-16T20:42:46.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:46.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-433 td/osd-backfill-space/datafile 2020-11-16T20:42:46.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:46.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-434 td/osd-backfill-space/datafile 2020-11-16T20:42:46.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:46.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-435 td/osd-backfill-space/datafile 2020-11-16T20:42:47.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:47.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-436 td/osd-backfill-space/datafile 2020-11-16T20:42:47.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:47.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-437 td/osd-backfill-space/datafile 2020-11-16T20:42:47.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:47.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-438 td/osd-backfill-space/datafile 2020-11-16T20:42:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-439 td/osd-backfill-space/datafile 2020-11-16T20:42:47.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:47.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-440 td/osd-backfill-space/datafile 2020-11-16T20:42:47.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:47.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-441 td/osd-backfill-space/datafile 2020-11-16T20:42:47.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:47.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-442 td/osd-backfill-space/datafile 2020-11-16T20:42:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-443 td/osd-backfill-space/datafile 2020-11-16T20:42:47.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:47.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-444 td/osd-backfill-space/datafile 2020-11-16T20:42:47.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:47.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-445 td/osd-backfill-space/datafile 2020-11-16T20:42:48.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:48.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-446 td/osd-backfill-space/datafile 2020-11-16T20:42:48.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:48.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-447 td/osd-backfill-space/datafile 2020-11-16T20:42:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:48.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-448 td/osd-backfill-space/datafile 2020-11-16T20:42:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-449 td/osd-backfill-space/datafile 2020-11-16T20:42:48.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:48.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-450 td/osd-backfill-space/datafile 2020-11-16T20:42:48.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:48.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-451 td/osd-backfill-space/datafile 2020-11-16T20:42:48.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:48.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-452 td/osd-backfill-space/datafile 2020-11-16T20:42:48.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:48.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-453 td/osd-backfill-space/datafile 2020-11-16T20:42:48.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:48.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-454 td/osd-backfill-space/datafile 2020-11-16T20:42:48.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:48.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-455 td/osd-backfill-space/datafile 2020-11-16T20:42:49.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:49.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-456 td/osd-backfill-space/datafile 2020-11-16T20:42:49.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:49.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-457 td/osd-backfill-space/datafile 2020-11-16T20:42:49.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:49.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-458 td/osd-backfill-space/datafile 2020-11-16T20:42:49.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:49.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-459 td/osd-backfill-space/datafile 2020-11-16T20:42:49.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:49.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-460 td/osd-backfill-space/datafile 2020-11-16T20:42:49.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:49.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-461 td/osd-backfill-space/datafile 2020-11-16T20:42:49.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:49.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-462 td/osd-backfill-space/datafile 2020-11-16T20:42:49.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:49.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-463 td/osd-backfill-space/datafile 2020-11-16T20:42:49.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:49.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-464 td/osd-backfill-space/datafile 2020-11-16T20:42:50.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T20:42:50.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-465 td/osd-backfill-space/datafile 2020-11-16T20:42:50.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:05.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-466 td/osd-backfill-space/datafile 2020-11-16T21:12:06.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-467 td/osd-backfill-space/datafile 2020-11-16T21:12:06.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-468 td/osd-backfill-space/datafile 2020-11-16T21:12:06.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-469 td/osd-backfill-space/datafile 2020-11-16T21:12:06.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-470 td/osd-backfill-space/datafile 2020-11-16T21:12:06.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-471 td/osd-backfill-space/datafile 2020-11-16T21:12:06.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-472 td/osd-backfill-space/datafile 2020-11-16T21:12:06.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-473 td/osd-backfill-space/datafile 2020-11-16T21:12:06.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-474 td/osd-backfill-space/datafile 2020-11-16T21:12:06.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-475 td/osd-backfill-space/datafile 2020-11-16T21:12:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-476 td/osd-backfill-space/datafile 2020-11-16T21:12:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-477 td/osd-backfill-space/datafile 2020-11-16T21:12:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-478 td/osd-backfill-space/datafile 2020-11-16T21:12:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-479 td/osd-backfill-space/datafile 2020-11-16T21:12:06.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-480 td/osd-backfill-space/datafile 2020-11-16T21:12:06.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-481 td/osd-backfill-space/datafile 2020-11-16T21:12:06.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-482 td/osd-backfill-space/datafile 2020-11-16T21:12:06.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-483 td/osd-backfill-space/datafile 2020-11-16T21:12:06.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-484 td/osd-backfill-space/datafile 2020-11-16T21:12:06.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-485 td/osd-backfill-space/datafile 2020-11-16T21:12:06.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-486 td/osd-backfill-space/datafile 2020-11-16T21:12:06.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-487 td/osd-backfill-space/datafile 2020-11-16T21:12:06.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-488 td/osd-backfill-space/datafile 2020-11-16T21:12:06.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-489 td/osd-backfill-space/datafile 2020-11-16T21:12:06.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-490 td/osd-backfill-space/datafile 2020-11-16T21:12:06.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-491 td/osd-backfill-space/datafile 2020-11-16T21:12:06.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-492 td/osd-backfill-space/datafile 2020-11-16T21:12:06.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-493 td/osd-backfill-space/datafile 2020-11-16T21:12:06.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-494 td/osd-backfill-space/datafile 2020-11-16T21:12:06.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-495 td/osd-backfill-space/datafile 2020-11-16T21:12:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-496 td/osd-backfill-space/datafile 2020-11-16T21:12:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-497 td/osd-backfill-space/datafile 2020-11-16T21:12:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-498 td/osd-backfill-space/datafile 2020-11-16T21:12:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-499 td/osd-backfill-space/datafile 2020-11-16T21:12:06.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-500 td/osd-backfill-space/datafile 2020-11-16T21:12:06.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-501 td/osd-backfill-space/datafile 2020-11-16T21:12:06.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-502 td/osd-backfill-space/datafile 2020-11-16T21:12:06.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-503 td/osd-backfill-space/datafile 2020-11-16T21:12:06.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-504 td/osd-backfill-space/datafile 2020-11-16T21:12:06.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-505 td/osd-backfill-space/datafile 2020-11-16T21:12:06.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-506 td/osd-backfill-space/datafile 2020-11-16T21:12:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-507 td/osd-backfill-space/datafile 2020-11-16T21:12:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-508 td/osd-backfill-space/datafile 2020-11-16T21:12:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-509 td/osd-backfill-space/datafile 2020-11-16T21:12:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-510 td/osd-backfill-space/datafile 2020-11-16T21:12:06.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-511 td/osd-backfill-space/datafile 2020-11-16T21:12:06.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-512 td/osd-backfill-space/datafile 2020-11-16T21:12:06.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-513 td/osd-backfill-space/datafile 2020-11-16T21:12:06.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-514 td/osd-backfill-space/datafile 2020-11-16T21:12:06.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-515 td/osd-backfill-space/datafile 2020-11-16T21:12:06.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-516 td/osd-backfill-space/datafile 2020-11-16T21:12:06.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-517 td/osd-backfill-space/datafile 2020-11-16T21:12:06.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-518 td/osd-backfill-space/datafile 2020-11-16T21:12:06.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-519 td/osd-backfill-space/datafile 2020-11-16T21:12:06.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-520 td/osd-backfill-space/datafile 2020-11-16T21:12:06.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-521 td/osd-backfill-space/datafile 2020-11-16T21:12:06.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-522 td/osd-backfill-space/datafile 2020-11-16T21:12:06.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-523 td/osd-backfill-space/datafile 2020-11-16T21:12:06.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-524 td/osd-backfill-space/datafile 2020-11-16T21:12:06.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-525 td/osd-backfill-space/datafile 2020-11-16T21:12:06.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-526 td/osd-backfill-space/datafile 2020-11-16T21:12:06.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-527 td/osd-backfill-space/datafile 2020-11-16T21:12:06.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-528 td/osd-backfill-space/datafile 2020-11-16T21:12:06.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-529 td/osd-backfill-space/datafile 2020-11-16T21:12:06.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-530 td/osd-backfill-space/datafile 2020-11-16T21:12:06.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-531 td/osd-backfill-space/datafile 2020-11-16T21:12:06.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-532 td/osd-backfill-space/datafile 2020-11-16T21:12:06.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-533 td/osd-backfill-space/datafile 2020-11-16T21:12:06.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-534 td/osd-backfill-space/datafile 2020-11-16T21:12:06.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-535 td/osd-backfill-space/datafile 2020-11-16T21:12:06.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-536 td/osd-backfill-space/datafile 2020-11-16T21:12:06.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-537 td/osd-backfill-space/datafile 2020-11-16T21:12:06.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-538 td/osd-backfill-space/datafile 2020-11-16T21:12:06.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-539 td/osd-backfill-space/datafile 2020-11-16T21:12:06.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-540 td/osd-backfill-space/datafile 2020-11-16T21:12:06.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-541 td/osd-backfill-space/datafile 2020-11-16T21:12:06.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-542 td/osd-backfill-space/datafile 2020-11-16T21:12:06.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-543 td/osd-backfill-space/datafile 2020-11-16T21:12:06.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-544 td/osd-backfill-space/datafile 2020-11-16T21:12:06.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-545 td/osd-backfill-space/datafile 2020-11-16T21:12:06.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-546 td/osd-backfill-space/datafile 2020-11-16T21:12:06.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-547 td/osd-backfill-space/datafile 2020-11-16T21:12:06.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-548 td/osd-backfill-space/datafile 2020-11-16T21:12:06.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-549 td/osd-backfill-space/datafile 2020-11-16T21:12:06.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-550 td/osd-backfill-space/datafile 2020-11-16T21:12:06.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-551 td/osd-backfill-space/datafile 2020-11-16T21:12:06.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-552 td/osd-backfill-space/datafile 2020-11-16T21:12:06.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-553 td/osd-backfill-space/datafile 2020-11-16T21:12:06.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-554 td/osd-backfill-space/datafile 2020-11-16T21:12:06.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-555 td/osd-backfill-space/datafile 2020-11-16T21:12:06.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-556 td/osd-backfill-space/datafile 2020-11-16T21:12:06.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-557 td/osd-backfill-space/datafile 2020-11-16T21:12:06.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-558 td/osd-backfill-space/datafile 2020-11-16T21:12:06.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-559 td/osd-backfill-space/datafile 2020-11-16T21:12:06.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-560 td/osd-backfill-space/datafile 2020-11-16T21:12:06.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-561 td/osd-backfill-space/datafile 2020-11-16T21:12:06.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-562 td/osd-backfill-space/datafile 2020-11-16T21:12:06.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-563 td/osd-backfill-space/datafile 2020-11-16T21:12:06.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-564 td/osd-backfill-space/datafile 2020-11-16T21:12:06.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-565 td/osd-backfill-space/datafile 2020-11-16T21:12:06.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-566 td/osd-backfill-space/datafile 2020-11-16T21:12:06.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-567 td/osd-backfill-space/datafile 2020-11-16T21:12:06.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-568 td/osd-backfill-space/datafile 2020-11-16T21:12:06.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-569 td/osd-backfill-space/datafile 2020-11-16T21:12:06.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-570 td/osd-backfill-space/datafile 2020-11-16T21:12:06.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-571 td/osd-backfill-space/datafile 2020-11-16T21:12:06.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-572 td/osd-backfill-space/datafile 2020-11-16T21:12:06.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-573 td/osd-backfill-space/datafile 2020-11-16T21:12:06.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-574 td/osd-backfill-space/datafile 2020-11-16T21:12:06.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-575 td/osd-backfill-space/datafile 2020-11-16T21:12:06.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-576 td/osd-backfill-space/datafile 2020-11-16T21:12:06.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-577 td/osd-backfill-space/datafile 2020-11-16T21:12:06.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-578 td/osd-backfill-space/datafile 2020-11-16T21:12:06.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-579 td/osd-backfill-space/datafile 2020-11-16T21:12:06.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-580 td/osd-backfill-space/datafile 2020-11-16T21:12:06.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-581 td/osd-backfill-space/datafile 2020-11-16T21:12:06.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-582 td/osd-backfill-space/datafile 2020-11-16T21:12:06.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-583 td/osd-backfill-space/datafile 2020-11-16T21:12:06.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-584 td/osd-backfill-space/datafile 2020-11-16T21:12:06.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-585 td/osd-backfill-space/datafile 2020-11-16T21:12:06.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-586 td/osd-backfill-space/datafile 2020-11-16T21:12:06.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-587 td/osd-backfill-space/datafile 2020-11-16T21:12:06.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-588 td/osd-backfill-space/datafile 2020-11-16T21:12:06.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-589 td/osd-backfill-space/datafile 2020-11-16T21:12:06.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-590 td/osd-backfill-space/datafile 2020-11-16T21:12:06.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-591 td/osd-backfill-space/datafile 2020-11-16T21:12:06.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-592 td/osd-backfill-space/datafile 2020-11-16T21:12:06.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-593 td/osd-backfill-space/datafile 2020-11-16T21:12:06.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-594 td/osd-backfill-space/datafile 2020-11-16T21:12:06.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-595 td/osd-backfill-space/datafile 2020-11-16T21:12:06.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-596 td/osd-backfill-space/datafile 2020-11-16T21:12:06.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-597 td/osd-backfill-space/datafile 2020-11-16T21:12:06.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-598 td/osd-backfill-space/datafile 2020-11-16T21:12:06.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-599 td/osd-backfill-space/datafile 2020-11-16T21:12:06.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-600 td/osd-backfill-space/datafile 2020-11-16T21:12:06.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:467: TEST_backfill_multi_partial: get_primary fillpool obj-fill-1 2020-11-16T21:12:06.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=fillpool 2020-11-16T21:12:06.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj-fill-1 2020-11-16T21:12:06.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map fillpool obj-fill-1 2020-11-16T21:12:06.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:12:06.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:467: TEST_backfill_multi_partial: local fillosd=1 2020-11-16T21:12:06.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:468: TEST_backfill_multi_partial: expr 1 + 1 2020-11-16T21:12:06.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:468: TEST_backfill_multi_partial: osd=2 2020-11-16T21:12:06.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:469: TEST_backfill_multi_partial: '[' 2 = 3 ']' 2020-11-16T21:12:06.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:473: TEST_backfill_multi_partial: kill_daemon td/osd-backfill-space/osd.1.pid TERM 2020-11-16T21:12:06.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/osd-backfill-space/osd.1.pid 2020-11-16T21:12:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=172622 2020-11-16T21:12:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2020-11-16T21:12:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2020-11-16T21:12:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2020-11-16T21:12:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2020-11-16T21:12:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 172622 2020-11-16T21:12:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2020-11-16T21:12:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2020-11-16T21:12:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2020-11-16T21:12:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2020-11-16T21:12:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 172622 2020-11-16T21:12:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2020-11-16T21:12:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2020-11-16T21:12:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2020-11-16T21:12:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:474: TEST_backfill_multi_partial: ceph osd out osd.1 2020-11-16T21:12:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. 2020-11-16T21:12:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:476: TEST_backfill_multi_partial: _objectstore_tool_nodown td/osd-backfill-space 1 --op export-remove --pgid 1.0 --file td/osd-backfill-space/fillexport.out 2020-11-16T21:12:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-space 2020-11-16T21:12:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T21:12:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-11-16T21:12:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T21:12:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-space/1 2020-11-16T21:12:06.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-space/1 --op export-remove --pgid 1.0 --file td/osd-backfill-space/fillexport.out 2020-11-16T21:12:06.708 INFO:tasks.workunit.client.0.smithi189.stderr:Exporting 1.0 info 1.0( v 36'600 (36'500,36'600] local-lis/les=23/24 n=600 ec=21/21 lis/c=23/23 les/c/f=24/24/0 sis=23) 2020-11-16T21:12:06.708 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0002d87b:::obj-fill-318:head# 2020-11-16T21:12:06.708 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:00676868:::obj-fill-583:head# 2020-11-16T21:12:06.708 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:00f4514e:::obj-fill-547:head# 2020-11-16T21:12:06.708 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:01596a3c:::obj-fill-533:head# 2020-11-16T21:12:06.708 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0285e89c:::obj-fill-190:head# 2020-11-16T21:12:06.709 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:02bd591e:::obj-fill-463:head# 2020-11-16T21:12:06.709 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:02c01ffe:::obj-fill-28:head# 2020-11-16T21:12:06.709 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:03ada31e:::obj-fill-285:head# 2020-11-16T21:12:06.709 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:03b07c3f:::obj-fill-568:head# 2020-11-16T21:12:06.709 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:03b85770:::obj-fill-573:head# 2020-11-16T21:12:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:03dcbe89:::obj-fill-375:head# 2020-11-16T21:12:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:04231826:::obj-fill-594:head# 2020-11-16T21:12:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:04d48684:::obj-fill-421:head# 2020-11-16T21:12:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:05385fcc:::obj-fill-299:head# 2020-11-16T21:12:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:05785d38:::obj-fill-159:head# 2020-11-16T21:12:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:057ec231:::obj-fill-138:head# 2020-11-16T21:12:06.710 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:06058b5e:::obj-fill-569:head# 2020-11-16T21:12:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:06d2a3d9:::obj-fill-284:head# 2020-11-16T21:12:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:07597b63:::obj-fill-459:head# 2020-11-16T21:12:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:075f4069:::obj-fill-550:head# 2020-11-16T21:12:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:08234052:::obj-fill-17:head# 2020-11-16T21:12:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:091ce2ec:::obj-fill-472:head# 2020-11-16T21:12:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:095fd509:::obj-fill-210:head# 2020-11-16T21:12:06.711 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0a52a594:::obj-fill-116:head# 2020-11-16T21:12:06.712 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0a63716b:::obj-fill-225:head# 2020-11-16T21:12:06.712 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0b0fa6fb:::obj-fill-541:head# 2020-11-16T21:12:06.712 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0b673f72:::obj-fill-232:head# 2020-11-16T21:12:06.712 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0bbcd1b2:::obj-fill-582:head# 2020-11-16T21:12:06.712 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0caf8ce3:::obj-fill-214:head# 2020-11-16T21:12:06.712 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0d94b9fb:::obj-fill-415:head# 2020-11-16T21:12:06.712 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0de4e6f7:::obj-fill-94:head# 2020-11-16T21:12:06.713 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0ef75768:::obj-fill-489:head# 2020-11-16T21:12:06.713 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0f67284f:::obj-fill-213:head# 2020-11-16T21:12:06.713 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:0f823df2:::obj-fill-234:head# 2020-11-16T21:12:06.713 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1005c08d:::obj-fill-490:head# 2020-11-16T21:12:06.713 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1039525f:::obj-fill-369:head# 2020-11-16T21:12:06.713 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:105fb36c:::obj-fill-226:head# 2020-11-16T21:12:06.713 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:111abdeb:::obj-fill-554:head# 2020-11-16T21:12:06.713 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:11392f3d:::obj-fill-453:head# 2020-11-16T21:12:06.714 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1169976c:::obj-fill-417:head# 2020-11-16T21:12:06.714 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:11840a2a:::obj-fill-477:head# 2020-11-16T21:12:06.714 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:11a4832e:::obj-fill-279:head# 2020-11-16T21:12:06.714 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:11a4d2f6:::obj-fill-256:head# 2020-11-16T21:12:06.714 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:12d52a31:::obj-fill-22:head# 2020-11-16T21:12:06.714 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:133c45db:::obj-fill-257:head# 2020-11-16T21:12:06.714 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:13563256:::obj-fill-150:head# 2020-11-16T21:12:06.715 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:13c09e00:::obj-fill-71:head# 2020-11-16T21:12:06.715 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:13e9fbed:::obj-fill-242:head# 2020-11-16T21:12:06.715 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:13fae6c2:::obj-fill-163:head# 2020-11-16T21:12:06.715 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:145025e0:::obj-fill-543:head# 2020-11-16T21:12:06.715 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:14517c84:::obj-fill-13:head# 2020-11-16T21:12:06.715 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:14802e9c:::obj-fill-127:head# 2020-11-16T21:12:06.715 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:150f1c96:::obj-fill-62:head# 2020-11-16T21:12:06.715 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:155e90a2:::obj-fill-537:head# 2020-11-16T21:12:06.716 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1632c776:::obj-fill-262:head# 2020-11-16T21:12:06.716 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:163bde28:::obj-fill-523:head# 2020-11-16T21:12:06.716 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:169c4ad0:::obj-fill-85:head# 2020-11-16T21:12:06.716 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:16c4da1d:::obj-fill-436:head# 2020-11-16T21:12:06.716 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:17b28c32:::obj-fill-345:head# 2020-11-16T21:12:06.716 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:17b34255:::obj-fill-577:head# 2020-11-16T21:12:06.716 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:17d2541f:::obj-fill-426:head# 2020-11-16T21:12:06.717 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:17ea19f2:::obj-fill-542:head# 2020-11-16T21:12:06.717 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:189155ea:::obj-fill-496:head# 2020-11-16T21:12:06.717 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:191a6d81:::obj-fill-548:head# 2020-11-16T21:12:06.717 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1a964993:::obj-fill-576:head# 2020-11-16T21:12:06.718 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1aa623dc:::obj-fill-356:head# 2020-11-16T21:12:06.718 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1abe5c38:::obj-fill-561:head# 2020-11-16T21:12:06.718 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1bddc06b:::obj-fill-7:head# 2020-11-16T21:12:06.718 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1be86b71:::obj-fill-295:head# 2020-11-16T21:12:06.718 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1c499bca:::obj-fill-240:head# 2020-11-16T21:12:06.718 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1c545e27:::obj-fill-509:head# 2020-11-16T21:12:06.718 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1c63c8f4:::obj-fill-4:head# 2020-11-16T21:12:06.719 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1d5a8f59:::obj-fill-377:head# 2020-11-16T21:12:06.719 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1d85b5df:::obj-fill-201:head# 2020-11-16T21:12:06.719 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1d9a0d5b:::obj-fill-581:head# 2020-11-16T21:12:06.719 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1dcf5667:::obj-fill-488:head# 2020-11-16T21:12:06.719 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1fa3090d:::obj-fill-458:head# 2020-11-16T21:12:06.719 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1fe937f6:::obj-fill-337:head# 2020-11-16T21:12:06.719 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:1ff54542:::obj-fill-349:head# 2020-11-16T21:12:06.720 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:20d9e0ca:::obj-fill-367:head# 2020-11-16T21:12:06.720 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:21b8e3bb:::obj-fill-538:head# 2020-11-16T21:12:06.720 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:227183d4:::obj-fill-235:head# 2020-11-16T21:12:06.720 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:229e5e5d:::obj-fill-522:head# 2020-11-16T21:12:06.720 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:234f8861:::obj-fill-476:head# 2020-11-16T21:12:06.720 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:23ac2c9d:::obj-fill-471:head# 2020-11-16T21:12:06.720 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:25db1742:::obj-fill-200:head# 2020-11-16T21:12:06.720 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:264f8545:::obj-fill-230:head# 2020-11-16T21:12:06.721 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:267ba97f:::obj-fill-493:head# 2020-11-16T21:12:06.721 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:26b067e8:::obj-fill-208:head# 2020-11-16T21:12:06.721 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:26f521cb:::obj-fill-300:head# 2020-11-16T21:12:06.721 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:27e70a11:::obj-fill-368:head# 2020-11-16T21:12:06.721 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:29357d4a:::obj-fill-456:head# 2020-11-16T21:12:06.721 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:29f2567e:::obj-fill-414:head# 2020-11-16T21:12:06.721 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2a10cbb6:::obj-fill-228:head# 2020-11-16T21:12:06.722 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2a1519a6:::obj-fill-148:head# 2020-11-16T21:12:06.722 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2a686ca6:::obj-fill-411:head# 2020-11-16T21:12:06.722 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2ab3f384:::obj-fill-77:head# 2020-11-16T21:12:06.722 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2ae6a1d0:::obj-fill-187:head# 2020-11-16T21:12:06.722 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2b890959:::obj-fill-222:head# 2020-11-16T21:12:06.722 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2bcc26c4:::obj-fill-114:head# 2020-11-16T21:12:06.722 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2bd3a3a2:::obj-fill-281:head# 2020-11-16T21:12:06.722 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2c1a21d7:::obj-fill-505:head# 2020-11-16T21:12:06.723 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2c255347:::obj-fill-95:head# 2020-11-16T21:12:06.723 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2c2a3f62:::obj-fill-131:head# 2020-11-16T21:12:06.723 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2c7d92ce:::obj-fill-442:head# 2020-11-16T21:12:06.723 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2d1ad117:::obj-fill-443:head# 2020-11-16T21:12:06.723 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2d57b2f7:::obj-fill-192:head# 2020-11-16T21:12:06.723 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2d7b53ea:::obj-fill-79:head# 2020-11-16T21:12:06.723 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2db25266:::obj-fill-121:head# 2020-11-16T21:12:06.724 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2e27853e:::obj-fill-473:head# 2020-11-16T21:12:06.724 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2f12cb21:::obj-fill-386:head# 2020-11-16T21:12:06.724 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2f957332:::obj-fill-422:head# 2020-11-16T21:12:06.724 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:2feae493:::obj-fill-301:head# 2020-11-16T21:12:06.724 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:302c95d2:::obj-fill-110:head# 2020-11-16T21:12:06.724 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:302d6b76:::obj-fill-374:head# 2020-11-16T21:12:06.724 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:30581f34:::obj-fill-313:head# 2020-11-16T21:12:06.724 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:307e86ac:::obj-fill-381:head# 2020-11-16T21:12:06.725 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:30ab77bf:::obj-fill-275:head# 2020-11-16T21:12:06.725 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:31127e6d:::obj-fill-250:head# 2020-11-16T21:12:06.725 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:32c93500:::obj-fill-155:head# 2020-11-16T21:12:06.725 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:339880b7:::obj-fill-227:head# 2020-11-16T21:12:06.725 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:34069e1e:::obj-fill-292:head# 2020-11-16T21:12:06.725 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:346dff31:::obj-fill-157:head# 2020-11-16T21:12:06.725 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:34ade0f1:::obj-fill-575:head# 2020-11-16T21:12:06.725 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:34bbae43:::obj-fill-90:head# 2020-11-16T21:12:06.726 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:34e97cf8:::obj-fill-334:head# 2020-11-16T21:12:06.726 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:35d7ccc2:::obj-fill-510:head# 2020-11-16T21:12:06.726 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:35f519b6:::obj-fill-513:head# 2020-11-16T21:12:06.726 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:364727e8:::obj-fill-130:head# 2020-11-16T21:12:06.726 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:364c0361:::obj-fill-586:head# 2020-11-16T21:12:06.726 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:368e7baf:::obj-fill-546:head# 2020-11-16T21:12:06.726 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:36ffb224:::obj-fill-82:head# 2020-11-16T21:12:06.727 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:371cb800:::obj-fill-330:head# 2020-11-16T21:12:06.727 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:37d5378d:::obj-fill-175:head# 2020-11-16T21:12:06.727 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:385593f3:::obj-fill-325:head# 2020-11-16T21:12:06.727 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:388d5073:::obj-fill-317:head# 2020-11-16T21:12:06.727 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:394af079:::obj-fill-59:head# 2020-11-16T21:12:06.727 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3a41ad37:::obj-fill-409:head# 2020-11-16T21:12:06.727 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3b3c63d2:::obj-fill-136:head# 2020-11-16T21:12:06.727 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3bb50131:::obj-fill-512:head# 2020-11-16T21:12:06.728 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3bb52bb5:::obj-fill-188:head# 2020-11-16T21:12:06.728 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3c0683d1:::obj-fill-372:head# 2020-11-16T21:12:06.728 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3c67d3f4:::obj-fill-249:head# 2020-11-16T21:12:06.728 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3c6a4a4c:::obj-fill-495:head# 2020-11-16T21:12:06.728 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3c7fc8c9:::obj-fill-324:head# 2020-11-16T21:12:06.728 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3d940c50:::obj-fill-591:head# 2020-11-16T21:12:06.728 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3ed5508e:::obj-fill-558:head# 2020-11-16T21:12:06.729 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:3f598316:::obj-fill-389:head# 2020-11-16T21:12:06.729 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:401cd0ce:::obj-fill-251:head# 2020-11-16T21:12:06.729 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:40fae006:::obj-fill-14:head# 2020-11-16T21:12:06.729 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4120b10f:::obj-fill-165:head# 2020-11-16T21:12:06.729 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:41b382c2:::obj-fill-289:head# 2020-11-16T21:12:06.729 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4277cdea:::obj-fill-319:head# 2020-11-16T21:12:06.729 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:42978c3c:::obj-fill-174:head# 2020-11-16T21:12:06.729 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:42b08dc2:::obj-fill-553:head# 2020-11-16T21:12:06.730 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:42ec4bf6:::obj-fill-209:head# 2020-11-16T21:12:06.730 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:430c129d:::obj-fill-448:head# 2020-11-16T21:12:06.730 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4335d553:::obj-fill-600:head# 2020-11-16T21:12:06.730 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:44804f62:::obj-fill-532:head# 2020-11-16T21:12:06.730 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:44954d54:::obj-fill-2:head# 2020-11-16T21:12:06.730 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4502f904:::obj-fill-429:head# 2020-11-16T21:12:06.730 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4568ddcb:::obj-fill-382:head# 2020-11-16T21:12:06.731 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:45939974:::obj-fill-563:head# 2020-11-16T21:12:06.731 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:462a5bbd:::obj-fill-567:head# 2020-11-16T21:12:06.731 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:47773ae4:::obj-fill-307:head# 2020-11-16T21:12:06.731 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4802b263:::obj-fill-387:head# 2020-11-16T21:12:06.731 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:483d938e:::obj-fill-128:head# 2020-11-16T21:12:06.731 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:48a7ad29:::obj-fill-147:head# 2020-11-16T21:12:06.731 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:493b61ce:::obj-fill-454:head# 2020-11-16T21:12:06.731 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4afb70fe:::obj-fill-186:head# 2020-11-16T21:12:06.732 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4bff08e3:::obj-fill-352:head# 2020-11-16T21:12:06.732 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4c01eabe:::obj-fill-370:head# 2020-11-16T21:12:06.732 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4c131560:::obj-fill-6:head# 2020-11-16T21:12:06.732 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4c4c5e14:::obj-fill-350:head# 2020-11-16T21:12:06.732 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4c80dc2c:::obj-fill-357:head# 2020-11-16T21:12:06.732 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4c9dee9b:::obj-fill-540:head# 2020-11-16T21:12:06.732 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4ce40c18:::obj-fill-280:head# 2020-11-16T21:12:06.732 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4d854fc3:::obj-fill-585:head# 2020-11-16T21:12:06.733 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:4f615b53:::obj-fill-101:head# 2020-11-16T21:12:06.733 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5042b30f:::obj-fill-84:head# 2020-11-16T21:12:06.733 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:507dc08b:::obj-fill-336:head# 2020-11-16T21:12:06.733 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:512fcf5a:::obj-fill-169:head# 2020-11-16T21:12:06.733 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:51a0c8f8:::obj-fill-578:head# 2020-11-16T21:12:06.733 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:51fd05a2:::obj-fill-425:head# 2020-11-16T21:12:06.733 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:52400ebc:::obj-fill-81:head# 2020-11-16T21:12:06.734 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5288d40c:::obj-fill-16:head# 2020-11-16T21:12:06.734 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5295b0ee:::obj-fill-224:head# 2020-11-16T21:12:06.734 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:531e4c60:::obj-fill-103:head# 2020-11-16T21:12:06.734 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:54259f28:::obj-fill-584:head# 2020-11-16T21:12:06.734 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:542d619c:::obj-fill-528:head# 2020-11-16T21:12:06.734 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5503ca3e:::obj-fill-494:head# 2020-11-16T21:12:06.734 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5555c48b:::obj-fill-403:head# 2020-11-16T21:12:06.734 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:558d1ec7:::obj-fill-447:head# 2020-11-16T21:12:06.735 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:55aa2233:::obj-fill-21:head# 2020-11-16T21:12:06.735 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:55aaf8c4:::obj-fill-29:head# 2020-11-16T21:12:06.735 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:55e246e1:::obj-fill-65:head# 2020-11-16T21:12:06.735 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:563b14a8:::obj-fill-120:head# 2020-11-16T21:12:06.735 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:56e1df25:::obj-fill-305:head# 2020-11-16T21:12:06.735 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5702979c:::obj-fill-73:head# 2020-11-16T21:12:06.735 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:570722ba:::obj-fill-219:head# 2020-11-16T21:12:06.736 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:57f7a09b:::obj-fill-571:head# 2020-11-16T21:12:06.736 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:585309ba:::obj-fill-144:head# 2020-11-16T21:12:06.736 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:585c7c9c:::obj-fill-437:head# 2020-11-16T21:12:06.736 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:587c63db:::obj-fill-182:head# 2020-11-16T21:12:06.736 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:58f34d6b:::obj-fill-598:head# 2020-11-16T21:12:06.736 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:58f40088:::obj-fill-298:head# 2020-11-16T21:12:06.736 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:591e150c:::obj-fill-32:head# 2020-11-16T21:12:06.736 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:59971ddb:::obj-fill-559:head# 2020-11-16T21:12:06.737 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5adafca4:::obj-fill-56:head# 2020-11-16T21:12:06.737 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5b32b04f:::obj-fill-484:head# 2020-11-16T21:12:06.737 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5bdaeb16:::obj-fill-99:head# 2020-11-16T21:12:06.737 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5c1d6f4b:::obj-fill-133:head# 2020-11-16T21:12:06.737 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5c92b8f1:::obj-fill-255:head# 2020-11-16T21:12:06.737 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5ca4850f:::obj-fill-238:head# 2020-11-16T21:12:06.737 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5d928a5b:::obj-fill-395:head# 2020-11-16T21:12:06.737 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5dd49320:::obj-fill-423:head# 2020-11-16T21:12:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5e7bc368:::obj-fill-506:head# 2020-11-16T21:12:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5ebd5ee3:::obj-fill-322:head# 2020-11-16T21:12:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5edeedf1:::obj-fill-557:head# 2020-11-16T21:12:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:5facf8e2:::obj-fill-15:head# 2020-11-16T21:12:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:603a2c76:::obj-fill-439:head# 2020-11-16T21:12:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:60b30714:::obj-fill-53:head# 2020-11-16T21:12:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:612b4a8a:::obj-fill-412:head# 2020-11-16T21:12:06.739 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:61a111b6:::obj-fill-87:head# 2020-11-16T21:12:06.739 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:63a0e8a2:::obj-fill-266:head# 2020-11-16T21:12:06.739 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:63aa4d60:::obj-fill-68:head# 2020-11-16T21:12:06.739 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:63ac5b95:::obj-fill-405:head# 2020-11-16T21:12:06.739 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6415a9e3:::obj-fill-400:head# 2020-11-16T21:12:06.739 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:64ba03ae:::obj-fill-20:head# 2020-11-16T21:12:06.739 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6512f015:::obj-fill-597:head# 2020-11-16T21:12:06.739 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:65433104:::obj-fill-504:head# 2020-11-16T21:12:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:65950c58:::obj-fill-86:head# 2020-11-16T21:12:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:65b6b2d7:::obj-fill-39:head# 2020-11-16T21:12:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:661c8afd:::obj-fill-267:head# 2020-11-16T21:12:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:66f87c94:::obj-fill-229:head# 2020-11-16T21:12:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:67292cef:::obj-fill-93:head# 2020-11-16T21:12:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:67f0e12c:::obj-fill-363:head# 2020-11-16T21:12:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:696eb83b:::obj-fill-185:head# 2020-11-16T21:12:06.741 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6b33981c:::obj-fill-259:head# 2020-11-16T21:12:06.741 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6b5a527c:::obj-fill-485:head# 2020-11-16T21:12:06.741 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6b6f1852:::obj-fill-390:head# 2020-11-16T21:12:06.741 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6ba5c5f8:::obj-fill-304:head# 2020-11-16T21:12:06.741 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6bce929d:::obj-fill-518:head# 2020-11-16T21:12:06.741 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6c8ffc04:::obj-fill-61:head# 2020-11-16T21:12:06.741 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6ca7e551:::obj-fill-40:head# 2020-11-16T21:12:06.741 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6caff2cd:::obj-fill-332:head# 2020-11-16T21:12:06.742 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6ce4adc7:::obj-fill-574:head# 2020-11-16T21:12:06.742 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6d09d770:::obj-fill-74:head# 2020-11-16T21:12:06.742 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6e039822:::obj-fill-278:head# 2020-11-16T21:12:06.742 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6f2cd109:::obj-fill-297:head# 2020-11-16T21:12:06.742 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6f3754c4:::obj-fill-88:head# 2020-11-16T21:12:06.742 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6f86cfd2:::obj-fill-511:head# 2020-11-16T21:12:06.742 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:6fe3f3f7:::obj-fill-193:head# 2020-11-16T21:12:06.743 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7002c57d:::obj-fill-50:head# 2020-11-16T21:12:06.743 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7030b2a3:::obj-fill-217:head# 2020-11-16T21:12:06.743 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:70665087:::obj-fill-51:head# 2020-11-16T21:12:06.743 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:70a0f5e0:::obj-fill-11:head# 2020-11-16T21:12:06.743 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:71f4643d:::obj-fill-236:head# 2020-11-16T21:12:06.743 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:722fbd0f:::obj-fill-302:head# 2020-11-16T21:12:06.743 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:72584da9:::obj-fill-445:head# 2020-11-16T21:12:06.743 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:72d91d24:::obj-fill-218:head# 2020-11-16T21:12:06.744 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:72f7268b:::obj-fill-106:head# 2020-11-16T21:12:06.744 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:74077abf:::obj-fill-435:head# 2020-11-16T21:12:06.744 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:740ca050:::obj-fill-112:head# 2020-11-16T21:12:06.744 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7435e881:::obj-fill-78:head# 2020-11-16T21:12:06.744 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:74ba0bb7:::obj-fill-478:head# 2020-11-16T21:12:06.744 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:74d82091:::obj-fill-96:head# 2020-11-16T21:12:06.744 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7515f56c:::obj-fill-483:head# 2020-11-16T21:12:06.745 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7550d6fe:::obj-fill-286:head# 2020-11-16T21:12:06.745 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:75807edc:::obj-fill-223:head# 2020-11-16T21:12:06.745 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:762ba324:::obj-fill-172:head# 2020-11-16T21:12:06.745 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:76369bda:::obj-fill-570:head# 2020-11-16T21:12:06.745 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7667a5b0:::obj-fill-196:head# 2020-11-16T21:12:06.745 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:77736c0c:::obj-fill-566:head# 2020-11-16T21:12:06.745 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:77fb47a4:::obj-fill-314:head# 2020-11-16T21:12:06.745 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:785ad894:::obj-fill-80:head# 2020-11-16T21:12:06.746 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:785e3d14:::obj-fill-530:head# 2020-11-16T21:12:06.746 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:788efef4:::obj-fill-595:head# 2020-11-16T21:12:06.746 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:789ec5ab:::obj-fill-44:head# 2020-11-16T21:12:06.746 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:798a3fa8:::obj-fill-517:head# 2020-11-16T21:12:06.746 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:79a105ef:::obj-fill-315:head# 2020-11-16T21:12:06.746 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:79d02f04:::obj-fill-243:head# 2020-11-16T21:12:06.747 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7a00dfcb:::obj-fill-402:head# 2020-11-16T21:12:06.747 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7b982684:::obj-fill-168:head# 2020-11-16T21:12:06.747 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7bb36851:::obj-fill-440:head# 2020-11-16T21:12:06.747 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7bb49d8c:::obj-fill-55:head# 2020-11-16T21:12:06.747 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7c77b783:::obj-fill-580:head# 2020-11-16T21:12:06.747 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7c89d75b:::obj-fill-555:head# 2020-11-16T21:12:06.748 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7d12fb67:::obj-fill-399:head# 2020-11-16T21:12:06.748 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7d139961:::obj-fill-92:head# 2020-11-16T21:12:06.748 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7d39ffd0:::obj-fill-135:head# 2020-11-16T21:12:06.748 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7e534cc4:::obj-fill-282:head# 2020-11-16T21:12:06.748 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7e7a08e0:::obj-fill-108:head# 2020-11-16T21:12:06.748 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7e8e2b67:::obj-fill-244:head# 2020-11-16T21:12:06.748 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7eea215e:::obj-fill-139:head# 2020-11-16T21:12:06.748 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7efa3c6e:::obj-fill-408:head# 2020-11-16T21:12:06.749 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7f03a24b:::obj-fill-45:head# 2020-11-16T21:12:06.749 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7f6a2e2b:::obj-fill-427:head# 2020-11-16T21:12:06.749 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7fb0abd6:::obj-fill-479:head# 2020-11-16T21:12:06.749 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:7fb77de1:::obj-fill-167:head# 2020-11-16T21:12:06.749 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:80012a03:::obj-fill-105:head# 2020-11-16T21:12:06.749 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:80075ae5:::obj-fill-376:head# 2020-11-16T21:12:06.749 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8089baa5:::obj-fill-97:head# 2020-11-16T21:12:06.749 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:80d0b4c5:::obj-fill-123:head# 2020-11-16T21:12:06.750 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:810f319f:::obj-fill-552:head# 2020-11-16T21:12:06.750 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:81427ef2:::obj-fill-474:head# 2020-11-16T21:12:06.750 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:818d780c:::obj-fill-33:head# 2020-11-16T21:12:06.750 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:819f31b9:::obj-fill-366:head# 2020-11-16T21:12:06.750 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8205549d:::obj-fill-125:head# 2020-11-16T21:12:06.750 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:821f56e5:::obj-fill-339:head# 2020-11-16T21:12:06.750 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:822807c6:::obj-fill-69:head# 2020-11-16T21:12:06.751 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:822fcf61:::obj-fill-560:head# 2020-11-16T21:12:06.751 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:82ad696b:::obj-fill-149:head# 2020-11-16T21:12:06.751 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:835cc771:::obj-fill-446:head# 2020-11-16T21:12:06.751 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8386bee1:::obj-fill-137:head# 2020-11-16T21:12:06.751 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:83c1303a:::obj-fill-311:head# 2020-11-16T21:12:06.751 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:83cf1c2a:::obj-fill-360:head# 2020-11-16T21:12:06.751 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:83d70321:::obj-fill-351:head# 2020-11-16T21:12:06.752 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:83ef162d:::obj-fill-331:head# 2020-11-16T21:12:06.752 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:840f8f20:::obj-fill-398:head# 2020-11-16T21:12:06.752 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:84d2e456:::obj-fill-207:head# 2020-11-16T21:12:06.752 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:856697ff:::obj-fill-333:head# 2020-11-16T21:12:06.752 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:858be937:::obj-fill-418:head# 2020-11-16T21:12:06.752 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:85a0b302:::obj-fill-438:head# 2020-11-16T21:12:06.752 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:85fa942e:::obj-fill-272:head# 2020-11-16T21:12:06.752 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:863eb6f8:::obj-fill-320:head# 2020-11-16T21:12:06.753 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:868d4588:::obj-fill-19:head# 2020-11-16T21:12:06.753 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8759d023:::obj-fill-466:head# 2020-11-16T21:12:06.753 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:88072111:::obj-fill-312:head# 2020-11-16T21:12:06.753 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:886edcef:::obj-fill-380:head# 2020-11-16T21:12:06.753 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:88ce753e:::obj-fill-43:head# 2020-11-16T21:12:06.753 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:890e1d83:::obj-fill-590:head# 2020-11-16T21:12:06.753 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:891566a3:::obj-fill-341:head# 2020-11-16T21:12:06.753 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:891ca235:::obj-fill-177:head# 2020-11-16T21:12:06.754 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:897fa9d0:::obj-fill-283:head# 2020-11-16T21:12:06.754 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:89cd304b:::obj-fill-592:head# 2020-11-16T21:12:06.754 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:89d04c4a:::obj-fill-388:head# 2020-11-16T21:12:06.754 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:89db45d9:::obj-fill-247:head# 2020-11-16T21:12:06.754 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8ad879c5:::obj-fill-492:head# 2020-11-16T21:12:06.754 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8af4b5de:::obj-fill-37:head# 2020-11-16T21:12:06.754 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8b132e86:::obj-fill-35:head# 2020-11-16T21:12:06.755 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8b80a60a:::obj-fill-100:head# 2020-11-16T21:12:06.755 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8bb64fa6:::obj-fill-599:head# 2020-11-16T21:12:06.755 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8bed03f5:::obj-fill-5:head# 2020-11-16T21:12:06.755 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8c60e135:::obj-fill-470:head# 2020-11-16T21:12:06.755 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8c774623:::obj-fill-38:head# 2020-11-16T21:12:06.755 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8cc528ae:::obj-fill-519:head# 2020-11-16T21:12:06.755 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8cf73826:::obj-fill-460:head# 2020-11-16T21:12:06.756 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8d3564b3:::obj-fill-452:head# 2020-11-16T21:12:06.756 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8dfe1ca2:::obj-fill-469:head# 2020-11-16T21:12:06.756 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8e527672:::obj-fill-246:head# 2020-11-16T21:12:06.756 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8e96103f:::obj-fill-444:head# 2020-11-16T21:12:06.756 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8ed6417f:::obj-fill-146:head# 2020-11-16T21:12:06.756 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8eda2ffc:::obj-fill-539:head# 2020-11-16T21:12:06.756 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8f08dd48:::obj-fill-26:head# 2020-11-16T21:12:06.756 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8f111b25:::obj-fill-18:head# 2020-11-16T21:12:06.757 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:8fde63f8:::obj-fill-338:head# 2020-11-16T21:12:06.757 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9030746d:::obj-fill-468:head# 2020-11-16T21:12:06.757 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9115d8a1:::obj-fill-260:head# 2020-11-16T21:12:06.757 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:91649909:::obj-fill-140:head# 2020-11-16T21:12:06.757 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:91c1081d:::obj-fill-404:head# 2020-11-16T21:12:06.757 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:91f42503:::obj-fill-216:head# 2020-11-16T21:12:06.757 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:927b34a0:::obj-fill-63:head# 2020-11-16T21:12:06.757 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:92a00a91:::obj-fill-467:head# 2020-11-16T21:12:06.758 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:930a7268:::obj-fill-450:head# 2020-11-16T21:12:06.758 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:93c3d0bf:::obj-fill-118:head# 2020-11-16T21:12:06.758 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:93ca7c9e:::obj-fill-132:head# 2020-11-16T21:12:06.758 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:942435d5:::obj-fill-378:head# 2020-11-16T21:12:06.758 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9445f8c5:::obj-fill-245:head# 2020-11-16T21:12:06.758 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9457c8d9:::obj-fill-535:head# 2020-11-16T21:12:06.758 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:950dc56e:::obj-fill-348:head# 2020-11-16T21:12:06.759 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9559ee1d:::obj-fill-545:head# 2020-11-16T21:12:06.759 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:957c5cc4:::obj-fill-212:head# 2020-11-16T21:12:06.759 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:95bdac02:::obj-fill-588:head# 2020-11-16T21:12:06.759 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:95f202af:::obj-fill-126:head# 2020-11-16T21:12:06.759 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:96789003:::obj-fill-83:head# 2020-11-16T21:12:06.759 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:96d494a8:::obj-fill-316:head# 2020-11-16T21:12:06.759 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:97adb85b:::obj-fill-233:head# 2020-11-16T21:12:06.759 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:980d998c:::obj-fill-197:head# 2020-11-16T21:12:06.760 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:98235c44:::obj-fill-396:head# 2020-11-16T21:12:06.760 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:987d3b87:::obj-fill-327:head# 2020-11-16T21:12:06.760 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:990a5e24:::obj-fill-515:head# 2020-11-16T21:12:06.760 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:995fec05:::obj-fill-191:head# 2020-11-16T21:12:06.760 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9a769b75:::obj-fill-587:head# 2020-11-16T21:12:06.760 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9b141990:::obj-fill-394:head# 2020-11-16T21:12:06.760 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9bb108fb:::obj-fill-579:head# 2020-11-16T21:12:06.761 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9ce19d92:::obj-fill-361:head# 2020-11-16T21:12:06.761 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9d8c3f95:::obj-fill-572:head# 2020-11-16T21:12:06.761 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9e283e00:::obj-fill-254:head# 2020-11-16T21:12:06.761 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9e3a1a61:::obj-fill-58:head# 2020-11-16T21:12:06.761 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9e8f6029:::obj-fill-41:head# 2020-11-16T21:12:06.761 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9ef117aa:::obj-fill-520:head# 2020-11-16T21:12:06.761 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9f3bdfc2:::obj-fill-184:head# 2020-11-16T21:12:06.761 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9fd89320:::obj-fill-364:head# 2020-11-16T21:12:06.762 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:9feb7062:::obj-fill-419:head# 2020-11-16T21:12:06.762 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a07504c4:::obj-fill-516:head# 2020-11-16T21:12:06.762 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a1897e34:::obj-fill-153:head# 2020-11-16T21:12:06.762 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a19a9d76:::obj-fill-30:head# 2020-11-16T21:12:06.762 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a1a49c98:::obj-fill-98:head# 2020-11-16T21:12:06.762 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a1ea911a:::obj-fill-36:head# 2020-11-16T21:12:06.762 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a24bad98:::obj-fill-335:head# 2020-11-16T21:12:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a263b16f:::obj-fill-326:head# 2020-11-16T21:12:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a27d661e:::obj-fill-152:head# 2020-11-16T21:12:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a282a129:::obj-fill-451:head# 2020-11-16T21:12:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a28f05ba:::obj-fill-241:head# 2020-11-16T21:12:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a2cdc5cc:::obj-fill-180:head# 2020-11-16T21:12:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a30777ea:::obj-fill-486:head# 2020-11-16T21:12:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a3a65661:::obj-fill-151:head# 2020-11-16T21:12:06.763 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a3cb2967:::obj-fill-195:head# 2020-11-16T21:12:06.764 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a3ffc098:::obj-fill-293:head# 2020-11-16T21:12:06.764 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a4a31ccc:::obj-fill-413:head# 2020-11-16T21:12:06.764 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a51c3677:::obj-fill-384:head# 2020-11-16T21:12:06.764 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a5e941a7:::obj-fill-141:head# 2020-11-16T21:12:06.764 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a6cdd8b2:::obj-fill-342:head# 2020-11-16T21:12:06.764 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a6e20b86:::obj-fill-593:head# 2020-11-16T21:12:06.764 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a7a9ccf9:::obj-fill-129:head# 2020-11-16T21:12:06.765 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a7b702d3:::obj-fill-49:head# 2020-11-16T21:12:06.765 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a8daf8bd:::obj-fill-391:head# 2020-11-16T21:12:06.765 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:a99c3cef:::obj-fill-171:head# 2020-11-16T21:12:06.765 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:aabb5718:::obj-fill-521:head# 2020-11-16T21:12:06.765 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ab0cb851:::obj-fill-183:head# 2020-11-16T21:12:06.765 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ab7a05e4:::obj-fill-203:head# 2020-11-16T21:12:06.765 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ab7dda83:::obj-fill-113:head# 2020-11-16T21:12:06.765 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:abb3502c:::obj-fill-461:head# 2020-11-16T21:12:06.766 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ac1c41be:::obj-fill-162:head# 2020-11-16T21:12:06.766 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ac2114ba:::obj-fill-239:head# 2020-11-16T21:12:06.766 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ad355fca:::obj-fill-271:head# 2020-11-16T21:12:06.766 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ad4634a0:::obj-fill-487:head# 2020-11-16T21:12:06.766 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:add2bb13:::obj-fill-31:head# 2020-11-16T21:12:06.766 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:aeb6258d:::obj-fill-124:head# 2020-11-16T21:12:06.766 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:af17f7d7:::obj-fill-464:head# 2020-11-16T21:12:06.767 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:af1e1cf0:::obj-fill-237:head# 2020-11-16T21:12:06.767 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:afc439ce:::obj-fill-248:head# 2020-11-16T21:12:06.767 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:afceab4d:::obj-fill-527:head# 2020-11-16T21:12:06.767 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b03af40c:::obj-fill-143:head# 2020-11-16T21:12:06.767 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b04cf0db:::obj-fill-507:head# 2020-11-16T21:12:06.767 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b07896f0:::obj-fill-269:head# 2020-11-16T21:12:06.768 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b0b954d5:::obj-fill-524:head# 2020-11-16T21:12:06.768 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b1cbfcb7:::obj-fill-134:head# 2020-11-16T21:12:06.768 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b2978fd3:::obj-fill-27:head# 2020-11-16T21:12:06.768 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b394798f:::obj-fill-156:head# 2020-11-16T21:12:06.768 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b3c6f36d:::obj-fill-401:head# 2020-11-16T21:12:06.768 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b3f8e5ac:::obj-fill-290:head# 2020-11-16T21:12:06.768 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b41ed5f8:::obj-fill-76:head# 2020-11-16T21:12:06.768 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b4878e76:::obj-fill-253:head# 2020-11-16T21:12:06.769 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b4e19741:::obj-fill-503:head# 2020-11-16T21:12:06.769 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b53a3690:::obj-fill-264:head# 2020-11-16T21:12:06.769 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b545ab4f:::obj-fill-24:head# 2020-11-16T21:12:06.769 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b55af5e6:::obj-fill-491:head# 2020-11-16T21:12:06.769 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b57edf82:::obj-fill-526:head# 2020-11-16T21:12:06.769 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b589d2be:::obj-fill-323:head# 2020-11-16T21:12:06.769 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b6207f32:::obj-fill-344:head# 2020-11-16T21:12:06.770 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b757ee2e:::obj-fill-433:head# 2020-11-16T21:12:06.770 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b75cc479:::obj-fill-34:head# 2020-11-16T21:12:06.770 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b7b03117:::obj-fill-42:head# 2020-11-16T21:12:06.770 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b7e6205d:::obj-fill-501:head# 2020-11-16T21:12:06.770 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b8d18b7a:::obj-fill-353:head# 2020-11-16T21:12:06.770 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b9189c29:::obj-fill-328:head# 2020-11-16T21:12:06.770 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b98e07aa:::obj-fill-54:head# 2020-11-16T21:12:06.771 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:b9cd1e9a:::obj-fill-371:head# 2020-11-16T21:12:06.771 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:bc3322bb:::obj-fill-340:head# 2020-11-16T21:12:06.771 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:bcf43161:::obj-fill-294:head# 2020-11-16T21:12:06.771 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:bd07fc4f:::obj-fill-549:head# 2020-11-16T21:12:06.771 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:bdf48e47:::obj-fill-430:head# 2020-11-16T21:12:06.771 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:be01c111:::obj-fill-1:head# 2020-11-16T21:12:06.771 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:be3a4bc7:::obj-fill-102:head# 2020-11-16T21:12:06.771 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:be8e1fcd:::obj-fill-321:head# 2020-11-16T21:12:06.772 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:bf5fcbac:::obj-fill-287:head# 2020-11-16T21:12:06.772 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:bf806d14:::obj-fill-115:head# 2020-11-16T21:12:06.772 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:bfb28823:::obj-fill-72:head# 2020-11-16T21:12:06.772 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:bfef805e:::obj-fill-119:head# 2020-11-16T21:12:06.772 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c02f8e10:::obj-fill-497:head# 2020-11-16T21:12:06.772 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c050cd6e:::obj-fill-8:head# 2020-11-16T21:12:06.772 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c218e297:::obj-fill-173:head# 2020-11-16T21:12:06.773 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c2bc66fe:::obj-fill-252:head# 2020-11-16T21:12:06.773 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c3b55903:::obj-fill-428:head# 2020-11-16T21:12:06.773 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c51907b0:::obj-fill-457:head# 2020-11-16T21:12:06.773 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c561dc83:::obj-fill-75:head# 2020-11-16T21:12:06.773 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c5d1e1a9:::obj-fill-268:head# 2020-11-16T21:12:06.773 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c60ab0d0:::obj-fill-303:head# 2020-11-16T21:12:06.773 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c644ce52:::obj-fill-514:head# 2020-11-16T21:12:06.773 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c6ab3a11:::obj-fill-407:head# 2020-11-16T21:12:06.774 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c74377bf:::obj-fill-432:head# 2020-11-16T21:12:06.774 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c767fff6:::obj-fill-531:head# 2020-11-16T21:12:06.774 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c7a9adad:::obj-fill-475:head# 2020-11-16T21:12:06.774 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c7e3a9c2:::obj-fill-465:head# 2020-11-16T21:12:06.774 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c850aed7:::obj-fill-424:head# 2020-11-16T21:12:06.774 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c8a0088a:::obj-fill-329:head# 2020-11-16T21:12:06.774 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:c9d419f9:::obj-fill-176:head# 2020-11-16T21:12:06.775 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:caf25ec2:::obj-fill-274:head# 2020-11-16T21:12:06.775 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cb625830:::obj-fill-70:head# 2020-11-16T21:12:06.775 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cbb65125:::obj-fill-529:head# 2020-11-16T21:12:06.775 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cbe2544c:::obj-fill-3:head# 2020-11-16T21:12:06.775 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cbfce4ee:::obj-fill-455:head# 2020-11-16T21:12:06.775 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cc62419c:::obj-fill-498:head# 2020-11-16T21:12:06.775 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cce831fa:::obj-fill-160:head# 2020-11-16T21:12:06.775 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cda71af4:::obj-fill-122:head# 2020-11-16T21:12:06.776 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cea7a728:::obj-fill-265:head# 2020-11-16T21:12:06.776 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cee19f0a:::obj-fill-142:head# 2020-11-16T21:12:06.776 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cf22117e:::obj-fill-499:head# 2020-11-16T21:12:06.776 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cfbfa66d:::obj-fill-66:head# 2020-11-16T21:12:06.776 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cfc2552c:::obj-fill-211:head# 2020-11-16T21:12:06.777 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:cfc563e8:::obj-fill-392:head# 2020-11-16T21:12:06.777 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d02d8880:::obj-fill-362:head# 2020-11-16T21:12:06.777 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d054947e:::obj-fill-67:head# 2020-11-16T21:12:06.777 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d061c977:::obj-fill-164:head# 2020-11-16T21:12:06.777 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d06a5f57:::obj-fill-296:head# 2020-11-16T21:12:06.777 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d110062a:::obj-fill-343:head# 2020-11-16T21:12:06.777 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d1b8ad47:::obj-fill-383:head# 2020-11-16T21:12:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d1e80db6:::obj-fill-117:head# 2020-11-16T21:12:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d210027e:::obj-fill-231:head# 2020-11-16T21:12:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d24428ed:::obj-fill-199:head# 2020-11-16T21:12:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d27c3c4b:::obj-fill-154:head# 2020-11-16T21:12:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d28957e3:::obj-fill-263:head# 2020-11-16T21:12:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d2d44da7:::obj-fill-544:head# 2020-11-16T21:12:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d31004a4:::obj-fill-434:head# 2020-11-16T21:12:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d32487c3:::obj-fill-385:head# 2020-11-16T21:12:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d340891c:::obj-fill-12:head# 2020-11-16T21:12:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d4135728:::obj-fill-310:head# 2020-11-16T21:12:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d5a43972:::obj-fill-9:head# 2020-11-16T21:12:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d5efeb15:::obj-fill-347:head# 2020-11-16T21:12:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d6463f70:::obj-fill-536:head# 2020-11-16T21:12:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d666d9b7:::obj-fill-420:head# 2020-11-16T21:12:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d6e86688:::obj-fill-166:head# 2020-11-16T21:12:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d730370d:::obj-fill-206:head# 2020-11-16T21:12:06.780 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d783551c:::obj-fill-109:head# 2020-11-16T21:12:06.780 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d7f77e5f:::obj-fill-551:head# 2020-11-16T21:12:06.780 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d843056c:::obj-fill-291:head# 2020-11-16T21:12:06.780 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d84eb173:::obj-fill-221:head# 2020-11-16T21:12:06.780 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d8dd069c:::obj-fill-60:head# 2020-11-16T21:12:06.780 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d8f2f750:::obj-fill-562:head# 2020-11-16T21:12:06.780 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d8f80ac2:::obj-fill-179:head# 2020-11-16T21:12:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:d9f904cf:::obj-fill-308:head# 2020-11-16T21:12:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:db0a534b:::obj-fill-410:head# 2020-11-16T21:12:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:dc4403f4:::obj-fill-416:head# 2020-11-16T21:12:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:dc66c3ca:::obj-fill-220:head# 2020-11-16T21:12:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:dc8198dc:::obj-fill-596:head# 2020-11-16T21:12:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:dcbade2b:::obj-fill-10:head# 2020-11-16T21:12:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ddb787ea:::obj-fill-556:head# 2020-11-16T21:12:06.781 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:de58065b:::obj-fill-306:head# 2020-11-16T21:12:06.782 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:deaa6fab:::obj-fill-194:head# 2020-11-16T21:12:06.782 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:dfee7714:::obj-fill-202:head# 2020-11-16T21:12:06.782 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:dff5df4a:::obj-fill-565:head# 2020-11-16T21:12:06.782 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e2053442:::obj-fill-107:head# 2020-11-16T21:12:06.782 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e32055b8:::obj-fill-57:head# 2020-11-16T21:12:06.782 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e37fac1e:::obj-fill-161:head# 2020-11-16T21:12:06.782 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e388f5aa:::obj-fill-273:head# 2020-11-16T21:12:06.783 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e38b1f4d:::obj-fill-393:head# 2020-11-16T21:12:06.783 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e4792474:::obj-fill-158:head# 2020-11-16T21:12:06.783 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e4899cb0:::obj-fill-270:head# 2020-11-16T21:12:06.783 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e62ae9f1:::obj-fill-288:head# 2020-11-16T21:12:06.783 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e64fd95a:::obj-fill-449:head# 2020-11-16T21:12:06.783 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e6682cfc:::obj-fill-373:head# 2020-11-16T21:12:06.783 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e716717b:::obj-fill-178:head# 2020-11-16T21:12:06.783 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e80226af:::obj-fill-354:head# 2020-11-16T21:12:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e83a5940:::obj-fill-564:head# 2020-11-16T21:12:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e854bfbe:::obj-fill-589:head# 2020-11-16T21:12:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e8b197bc:::obj-fill-89:head# 2020-11-16T21:12:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e92fca88:::obj-fill-397:head# 2020-11-16T21:12:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e94d8f7b:::obj-fill-261:head# 2020-11-16T21:12:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:e98cc0a4:::obj-fill-525:head# 2020-11-16T21:12:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ea5a9459:::obj-fill-431:head# 2020-11-16T21:12:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ebc54d47:::obj-fill-258:head# 2020-11-16T21:12:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ec0002dd:::obj-fill-205:head# 2020-11-16T21:12:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ec740b5c:::obj-fill-46:head# 2020-11-16T21:12:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ed35dda4:::obj-fill-170:head# 2020-11-16T21:12:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ed51b8f7:::obj-fill-91:head# 2020-11-16T21:12:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:edd96161:::obj-fill-48:head# 2020-11-16T21:12:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ee3f0d4b:::obj-fill-47:head# 2020-11-16T21:12:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ee4784dd:::obj-fill-480:head# 2020-11-16T21:12:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ee5a8271:::obj-fill-198:head# 2020-11-16T21:12:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ef745b66:::obj-fill-145:head# 2020-11-16T21:12:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:efa5309d:::obj-fill-215:head# 2020-11-16T21:12:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:efbdd3a1:::obj-fill-276:head# 2020-11-16T21:12:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f059d4c0:::obj-fill-482:head# 2020-11-16T21:12:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f0f9fb7a:::obj-fill-379:head# 2020-11-16T21:12:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f16a3656:::obj-fill-462:head# 2020-11-16T21:12:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f1b395ae:::obj-fill-25:head# 2020-11-16T21:12:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f5876dad:::obj-fill-359:head# 2020-11-16T21:12:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f5a283cd:::obj-fill-534:head# 2020-11-16T21:12:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f5cd905a:::obj-fill-358:head# 2020-11-16T21:12:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f5e48302:::obj-fill-406:head# 2020-11-16T21:12:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f5ea65fc:::obj-fill-52:head# 2020-11-16T21:12:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f642de6b:::obj-fill-204:head# 2020-11-16T21:12:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f716eafd:::obj-fill-502:head# 2020-11-16T21:12:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f752e9cb:::obj-fill-23:head# 2020-11-16T21:12:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f85fe42c:::obj-fill-346:head# 2020-11-16T21:12:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:f882fd3b:::obj-fill-355:head# 2020-11-16T21:12:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:fa133332:::obj-fill-309:head# 2020-11-16T21:12:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:fa875e52:::obj-fill-64:head# 2020-11-16T21:12:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:fad91a22:::obj-fill-481:head# 2020-11-16T21:12:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:fbbc3122:::obj-fill-111:head# 2020-11-16T21:12:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:fc539ddb:::obj-fill-181:head# 2020-11-16T21:12:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:fc54128d:::obj-fill-104:head# 2020-11-16T21:12:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:fe00aded:::obj-fill-365:head# 2020-11-16T21:12:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:fe1e9971:::obj-fill-189:head# 2020-11-16T21:12:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ff0e0563:::obj-fill-508:head# 2020-11-16T21:12:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ff4e156c:::obj-fill-441:head# 2020-11-16T21:12:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:ffa355f7:::obj-fill-500:head# 2020-11-16T21:12:06.790 INFO:tasks.workunit.client.0.smithi189.stderr:Read #1:fff6c93b:::obj-fill-277:head# 2020-11-16T21:12:06.790 INFO:tasks.workunit.client.0.smithi189.stderr:Export successful 2020-11-16T21:12:06.790 INFO:tasks.workunit.client.0.smithi189.stderr:Remove successful 2020-11-16T21:12:06.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:477: TEST_backfill_multi_partial: activate_osd td/osd-backfill-space 1 2020-11-16T21:12:06.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:06.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T21:12:06.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T21:12:06.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T21:12:06.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/1 2020-11-16T21:12:06.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:06.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:06.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:06.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:06.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2020-11-16T21:12:06.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2020-11-16T21:12:06.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:06.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T21:12:06.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:06.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T21:12:06.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:06.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:06.792 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:06.793 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:06.793 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:06.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:06.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:06.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:06.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:06.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:06.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:06.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:06.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:06.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T21:12:06.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T21:12:06.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/1 2020-11-16T21:12:06.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T21:12:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:06.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/1/whoami 2020-11-16T21:12:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T21:12:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T21:12:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T21:12:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T21:12:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:43:09.172+0000 7f823260cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:43:09.176+0000 7f823260cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:43:09.176+0000 7f823260cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T21:12:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:12:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:12:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:43:09.704+0000 7f823260cd80 -1 Falling back to public interface 2020-11-16T21:12:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:43:10.916+0000 7f823260cd80 -1 osd.1 36 log_to_monitors {default=true} 2020-11-16T21:12:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:12:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:479: TEST_backfill_multi_partial: ceph pg dump pgs 2020-11-16T21:12:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:12:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:481: TEST_backfill_multi_partial: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=1 2020-11-16T21:12:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:1+0 records in 2020-11-16T21:12:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:1+0 records out 2020-11-16T21:12:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000255583 s, 4.0 MB/s 2020-11-16T21:12:06.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: seq 1 600 2020-11-16T21:12:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-1 td/osd-backfill-space/datafile 2020-11-16T21:12:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-2 td/osd-backfill-space/datafile 2020-11-16T21:12:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-3 td/osd-backfill-space/datafile 2020-11-16T21:12:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-4 td/osd-backfill-space/datafile 2020-11-16T21:12:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-5 td/osd-backfill-space/datafile 2020-11-16T21:12:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-6 td/osd-backfill-space/datafile 2020-11-16T21:12:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-7 td/osd-backfill-space/datafile 2020-11-16T21:12:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-8 td/osd-backfill-space/datafile 2020-11-16T21:12:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-9 td/osd-backfill-space/datafile 2020-11-16T21:12:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-10 td/osd-backfill-space/datafile 2020-11-16T21:12:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-11 td/osd-backfill-space/datafile 2020-11-16T21:12:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-12 td/osd-backfill-space/datafile 2020-11-16T21:12:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-13 td/osd-backfill-space/datafile 2020-11-16T21:12:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-14 td/osd-backfill-space/datafile 2020-11-16T21:12:06.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-15 td/osd-backfill-space/datafile 2020-11-16T21:12:06.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-16 td/osd-backfill-space/datafile 2020-11-16T21:12:06.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-17 td/osd-backfill-space/datafile 2020-11-16T21:12:06.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-18 td/osd-backfill-space/datafile 2020-11-16T21:12:06.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-19 td/osd-backfill-space/datafile 2020-11-16T21:12:06.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-20 td/osd-backfill-space/datafile 2020-11-16T21:12:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-21 td/osd-backfill-space/datafile 2020-11-16T21:12:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-22 td/osd-backfill-space/datafile 2020-11-16T21:12:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-23 td/osd-backfill-space/datafile 2020-11-16T21:12:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-24 td/osd-backfill-space/datafile 2020-11-16T21:12:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-25 td/osd-backfill-space/datafile 2020-11-16T21:12:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-26 td/osd-backfill-space/datafile 2020-11-16T21:12:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-27 td/osd-backfill-space/datafile 2020-11-16T21:12:06.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-28 td/osd-backfill-space/datafile 2020-11-16T21:12:06.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-29 td/osd-backfill-space/datafile 2020-11-16T21:12:06.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-30 td/osd-backfill-space/datafile 2020-11-16T21:12:06.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-31 td/osd-backfill-space/datafile 2020-11-16T21:12:06.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-32 td/osd-backfill-space/datafile 2020-11-16T21:12:06.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-33 td/osd-backfill-space/datafile 2020-11-16T21:12:06.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-34 td/osd-backfill-space/datafile 2020-11-16T21:12:06.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-35 td/osd-backfill-space/datafile 2020-11-16T21:12:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-36 td/osd-backfill-space/datafile 2020-11-16T21:12:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-37 td/osd-backfill-space/datafile 2020-11-16T21:12:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-38 td/osd-backfill-space/datafile 2020-11-16T21:12:06.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-39 td/osd-backfill-space/datafile 2020-11-16T21:12:06.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-40 td/osd-backfill-space/datafile 2020-11-16T21:12:06.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-41 td/osd-backfill-space/datafile 2020-11-16T21:12:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-42 td/osd-backfill-space/datafile 2020-11-16T21:12:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-43 td/osd-backfill-space/datafile 2020-11-16T21:12:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-44 td/osd-backfill-space/datafile 2020-11-16T21:12:06.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-45 td/osd-backfill-space/datafile 2020-11-16T21:12:06.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-46 td/osd-backfill-space/datafile 2020-11-16T21:12:06.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-47 td/osd-backfill-space/datafile 2020-11-16T21:12:06.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-48 td/osd-backfill-space/datafile 2020-11-16T21:12:06.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-49 td/osd-backfill-space/datafile 2020-11-16T21:12:06.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-50 td/osd-backfill-space/datafile 2020-11-16T21:12:06.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-51 td/osd-backfill-space/datafile 2020-11-16T21:12:06.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-52 td/osd-backfill-space/datafile 2020-11-16T21:12:06.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-53 td/osd-backfill-space/datafile 2020-11-16T21:12:06.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-54 td/osd-backfill-space/datafile 2020-11-16T21:12:06.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-55 td/osd-backfill-space/datafile 2020-11-16T21:12:06.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-56 td/osd-backfill-space/datafile 2020-11-16T21:12:06.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-57 td/osd-backfill-space/datafile 2020-11-16T21:12:06.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-58 td/osd-backfill-space/datafile 2020-11-16T21:12:06.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-59 td/osd-backfill-space/datafile 2020-11-16T21:12:06.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-60 td/osd-backfill-space/datafile 2020-11-16T21:12:06.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-61 td/osd-backfill-space/datafile 2020-11-16T21:12:06.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-62 td/osd-backfill-space/datafile 2020-11-16T21:12:06.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-63 td/osd-backfill-space/datafile 2020-11-16T21:12:06.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-64 td/osd-backfill-space/datafile 2020-11-16T21:12:06.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-65 td/osd-backfill-space/datafile 2020-11-16T21:12:06.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-66 td/osd-backfill-space/datafile 2020-11-16T21:12:06.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-67 td/osd-backfill-space/datafile 2020-11-16T21:12:06.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-68 td/osd-backfill-space/datafile 2020-11-16T21:12:06.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-69 td/osd-backfill-space/datafile 2020-11-16T21:12:06.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-70 td/osd-backfill-space/datafile 2020-11-16T21:12:06.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-71 td/osd-backfill-space/datafile 2020-11-16T21:12:06.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-72 td/osd-backfill-space/datafile 2020-11-16T21:12:06.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-73 td/osd-backfill-space/datafile 2020-11-16T21:12:06.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-74 td/osd-backfill-space/datafile 2020-11-16T21:12:06.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-75 td/osd-backfill-space/datafile 2020-11-16T21:12:06.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-76 td/osd-backfill-space/datafile 2020-11-16T21:12:06.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-77 td/osd-backfill-space/datafile 2020-11-16T21:12:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-78 td/osd-backfill-space/datafile 2020-11-16T21:12:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-79 td/osd-backfill-space/datafile 2020-11-16T21:12:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-80 td/osd-backfill-space/datafile 2020-11-16T21:12:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-81 td/osd-backfill-space/datafile 2020-11-16T21:12:06.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-82 td/osd-backfill-space/datafile 2020-11-16T21:12:06.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-83 td/osd-backfill-space/datafile 2020-11-16T21:12:06.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-84 td/osd-backfill-space/datafile 2020-11-16T21:12:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-85 td/osd-backfill-space/datafile 2020-11-16T21:12:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-86 td/osd-backfill-space/datafile 2020-11-16T21:12:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-87 td/osd-backfill-space/datafile 2020-11-16T21:12:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-88 td/osd-backfill-space/datafile 2020-11-16T21:12:06.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-89 td/osd-backfill-space/datafile 2020-11-16T21:12:06.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-90 td/osd-backfill-space/datafile 2020-11-16T21:12:06.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-91 td/osd-backfill-space/datafile 2020-11-16T21:12:06.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-92 td/osd-backfill-space/datafile 2020-11-16T21:12:06.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-93 td/osd-backfill-space/datafile 2020-11-16T21:12:06.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-94 td/osd-backfill-space/datafile 2020-11-16T21:12:06.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-95 td/osd-backfill-space/datafile 2020-11-16T21:12:06.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-96 td/osd-backfill-space/datafile 2020-11-16T21:12:06.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-97 td/osd-backfill-space/datafile 2020-11-16T21:12:06.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-98 td/osd-backfill-space/datafile 2020-11-16T21:12:06.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-99 td/osd-backfill-space/datafile 2020-11-16T21:12:06.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-100 td/osd-backfill-space/datafile 2020-11-16T21:12:06.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-101 td/osd-backfill-space/datafile 2020-11-16T21:12:06.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-102 td/osd-backfill-space/datafile 2020-11-16T21:12:06.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-103 td/osd-backfill-space/datafile 2020-11-16T21:12:06.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-104 td/osd-backfill-space/datafile 2020-11-16T21:12:06.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-105 td/osd-backfill-space/datafile 2020-11-16T21:12:06.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-106 td/osd-backfill-space/datafile 2020-11-16T21:12:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-107 td/osd-backfill-space/datafile 2020-11-16T21:12:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-108 td/osd-backfill-space/datafile 2020-11-16T21:12:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-109 td/osd-backfill-space/datafile 2020-11-16T21:12:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-110 td/osd-backfill-space/datafile 2020-11-16T21:12:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-111 td/osd-backfill-space/datafile 2020-11-16T21:12:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-112 td/osd-backfill-space/datafile 2020-11-16T21:12:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-113 td/osd-backfill-space/datafile 2020-11-16T21:12:06.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-114 td/osd-backfill-space/datafile 2020-11-16T21:12:06.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-115 td/osd-backfill-space/datafile 2020-11-16T21:12:06.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-116 td/osd-backfill-space/datafile 2020-11-16T21:12:06.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-117 td/osd-backfill-space/datafile 2020-11-16T21:12:06.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-118 td/osd-backfill-space/datafile 2020-11-16T21:12:06.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-119 td/osd-backfill-space/datafile 2020-11-16T21:12:06.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-120 td/osd-backfill-space/datafile 2020-11-16T21:12:06.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-121 td/osd-backfill-space/datafile 2020-11-16T21:12:06.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-122 td/osd-backfill-space/datafile 2020-11-16T21:12:06.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-123 td/osd-backfill-space/datafile 2020-11-16T21:12:06.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-124 td/osd-backfill-space/datafile 2020-11-16T21:12:06.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-125 td/osd-backfill-space/datafile 2020-11-16T21:12:06.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-126 td/osd-backfill-space/datafile 2020-11-16T21:12:06.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-127 td/osd-backfill-space/datafile 2020-11-16T21:12:06.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-128 td/osd-backfill-space/datafile 2020-11-16T21:12:06.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-129 td/osd-backfill-space/datafile 2020-11-16T21:12:06.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-130 td/osd-backfill-space/datafile 2020-11-16T21:12:06.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-131 td/osd-backfill-space/datafile 2020-11-16T21:12:06.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-132 td/osd-backfill-space/datafile 2020-11-16T21:12:06.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-133 td/osd-backfill-space/datafile 2020-11-16T21:12:06.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-134 td/osd-backfill-space/datafile 2020-11-16T21:12:06.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-135 td/osd-backfill-space/datafile 2020-11-16T21:12:06.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-136 td/osd-backfill-space/datafile 2020-11-16T21:12:06.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-137 td/osd-backfill-space/datafile 2020-11-16T21:12:06.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-138 td/osd-backfill-space/datafile 2020-11-16T21:12:06.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-139 td/osd-backfill-space/datafile 2020-11-16T21:12:06.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-140 td/osd-backfill-space/datafile 2020-11-16T21:12:06.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-141 td/osd-backfill-space/datafile 2020-11-16T21:12:06.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-142 td/osd-backfill-space/datafile 2020-11-16T21:12:06.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-143 td/osd-backfill-space/datafile 2020-11-16T21:12:06.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-144 td/osd-backfill-space/datafile 2020-11-16T21:12:06.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-145 td/osd-backfill-space/datafile 2020-11-16T21:12:06.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-146 td/osd-backfill-space/datafile 2020-11-16T21:12:06.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-147 td/osd-backfill-space/datafile 2020-11-16T21:12:06.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-148 td/osd-backfill-space/datafile 2020-11-16T21:12:06.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-149 td/osd-backfill-space/datafile 2020-11-16T21:12:06.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-150 td/osd-backfill-space/datafile 2020-11-16T21:12:06.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-151 td/osd-backfill-space/datafile 2020-11-16T21:12:06.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-152 td/osd-backfill-space/datafile 2020-11-16T21:12:06.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-153 td/osd-backfill-space/datafile 2020-11-16T21:12:06.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-154 td/osd-backfill-space/datafile 2020-11-16T21:12:06.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-155 td/osd-backfill-space/datafile 2020-11-16T21:12:06.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-156 td/osd-backfill-space/datafile 2020-11-16T21:12:06.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-157 td/osd-backfill-space/datafile 2020-11-16T21:12:06.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-158 td/osd-backfill-space/datafile 2020-11-16T21:12:06.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-159 td/osd-backfill-space/datafile 2020-11-16T21:12:06.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-160 td/osd-backfill-space/datafile 2020-11-16T21:12:06.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-161 td/osd-backfill-space/datafile 2020-11-16T21:12:06.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-162 td/osd-backfill-space/datafile 2020-11-16T21:12:06.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-163 td/osd-backfill-space/datafile 2020-11-16T21:12:06.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-164 td/osd-backfill-space/datafile 2020-11-16T21:12:06.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-165 td/osd-backfill-space/datafile 2020-11-16T21:12:06.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-166 td/osd-backfill-space/datafile 2020-11-16T21:12:06.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-167 td/osd-backfill-space/datafile 2020-11-16T21:12:06.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-168 td/osd-backfill-space/datafile 2020-11-16T21:12:06.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-169 td/osd-backfill-space/datafile 2020-11-16T21:12:06.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-170 td/osd-backfill-space/datafile 2020-11-16T21:12:06.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-171 td/osd-backfill-space/datafile 2020-11-16T21:12:06.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-172 td/osd-backfill-space/datafile 2020-11-16T21:12:06.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-173 td/osd-backfill-space/datafile 2020-11-16T21:12:06.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-174 td/osd-backfill-space/datafile 2020-11-16T21:12:06.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-175 td/osd-backfill-space/datafile 2020-11-16T21:12:06.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-176 td/osd-backfill-space/datafile 2020-11-16T21:12:06.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-177 td/osd-backfill-space/datafile 2020-11-16T21:12:06.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-178 td/osd-backfill-space/datafile 2020-11-16T21:12:06.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-179 td/osd-backfill-space/datafile 2020-11-16T21:12:06.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-180 td/osd-backfill-space/datafile 2020-11-16T21:12:06.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-181 td/osd-backfill-space/datafile 2020-11-16T21:12:06.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-182 td/osd-backfill-space/datafile 2020-11-16T21:12:06.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-183 td/osd-backfill-space/datafile 2020-11-16T21:12:06.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-184 td/osd-backfill-space/datafile 2020-11-16T21:12:06.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-185 td/osd-backfill-space/datafile 2020-11-16T21:12:06.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-186 td/osd-backfill-space/datafile 2020-11-16T21:12:06.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-187 td/osd-backfill-space/datafile 2020-11-16T21:12:06.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-188 td/osd-backfill-space/datafile 2020-11-16T21:12:06.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-189 td/osd-backfill-space/datafile 2020-11-16T21:12:06.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-190 td/osd-backfill-space/datafile 2020-11-16T21:12:06.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-191 td/osd-backfill-space/datafile 2020-11-16T21:12:06.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-192 td/osd-backfill-space/datafile 2020-11-16T21:12:06.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-193 td/osd-backfill-space/datafile 2020-11-16T21:12:06.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-194 td/osd-backfill-space/datafile 2020-11-16T21:12:06.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-195 td/osd-backfill-space/datafile 2020-11-16T21:12:06.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-196 td/osd-backfill-space/datafile 2020-11-16T21:12:06.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-197 td/osd-backfill-space/datafile 2020-11-16T21:12:06.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-198 td/osd-backfill-space/datafile 2020-11-16T21:12:06.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-199 td/osd-backfill-space/datafile 2020-11-16T21:12:06.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-200 td/osd-backfill-space/datafile 2020-11-16T21:12:06.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-201 td/osd-backfill-space/datafile 2020-11-16T21:12:06.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-202 td/osd-backfill-space/datafile 2020-11-16T21:12:06.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-203 td/osd-backfill-space/datafile 2020-11-16T21:12:06.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-204 td/osd-backfill-space/datafile 2020-11-16T21:12:06.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-205 td/osd-backfill-space/datafile 2020-11-16T21:12:06.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-206 td/osd-backfill-space/datafile 2020-11-16T21:12:06.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-207 td/osd-backfill-space/datafile 2020-11-16T21:12:06.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-208 td/osd-backfill-space/datafile 2020-11-16T21:12:06.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-209 td/osd-backfill-space/datafile 2020-11-16T21:12:06.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-210 td/osd-backfill-space/datafile 2020-11-16T21:12:06.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-211 td/osd-backfill-space/datafile 2020-11-16T21:12:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-212 td/osd-backfill-space/datafile 2020-11-16T21:12:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-213 td/osd-backfill-space/datafile 2020-11-16T21:12:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-214 td/osd-backfill-space/datafile 2020-11-16T21:12:06.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-215 td/osd-backfill-space/datafile 2020-11-16T21:12:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-216 td/osd-backfill-space/datafile 2020-11-16T21:12:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-217 td/osd-backfill-space/datafile 2020-11-16T21:12:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-218 td/osd-backfill-space/datafile 2020-11-16T21:12:06.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-219 td/osd-backfill-space/datafile 2020-11-16T21:12:06.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-220 td/osd-backfill-space/datafile 2020-11-16T21:12:06.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-221 td/osd-backfill-space/datafile 2020-11-16T21:12:06.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-222 td/osd-backfill-space/datafile 2020-11-16T21:12:06.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-223 td/osd-backfill-space/datafile 2020-11-16T21:12:06.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-224 td/osd-backfill-space/datafile 2020-11-16T21:12:06.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-225 td/osd-backfill-space/datafile 2020-11-16T21:12:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-226 td/osd-backfill-space/datafile 2020-11-16T21:12:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-227 td/osd-backfill-space/datafile 2020-11-16T21:12:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-228 td/osd-backfill-space/datafile 2020-11-16T21:12:06.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-229 td/osd-backfill-space/datafile 2020-11-16T21:12:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-230 td/osd-backfill-space/datafile 2020-11-16T21:12:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-231 td/osd-backfill-space/datafile 2020-11-16T21:12:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-232 td/osd-backfill-space/datafile 2020-11-16T21:12:06.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-233 td/osd-backfill-space/datafile 2020-11-16T21:12:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-234 td/osd-backfill-space/datafile 2020-11-16T21:12:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-235 td/osd-backfill-space/datafile 2020-11-16T21:12:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-236 td/osd-backfill-space/datafile 2020-11-16T21:12:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-237 td/osd-backfill-space/datafile 2020-11-16T21:12:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-238 td/osd-backfill-space/datafile 2020-11-16T21:12:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-239 td/osd-backfill-space/datafile 2020-11-16T21:12:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-240 td/osd-backfill-space/datafile 2020-11-16T21:12:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-241 td/osd-backfill-space/datafile 2020-11-16T21:12:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-242 td/osd-backfill-space/datafile 2020-11-16T21:12:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-243 td/osd-backfill-space/datafile 2020-11-16T21:12:06.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-244 td/osd-backfill-space/datafile 2020-11-16T21:12:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-245 td/osd-backfill-space/datafile 2020-11-16T21:12:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-246 td/osd-backfill-space/datafile 2020-11-16T21:12:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-247 td/osd-backfill-space/datafile 2020-11-16T21:12:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-248 td/osd-backfill-space/datafile 2020-11-16T21:12:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-249 td/osd-backfill-space/datafile 2020-11-16T21:12:06.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-250 td/osd-backfill-space/datafile 2020-11-16T21:12:06.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-251 td/osd-backfill-space/datafile 2020-11-16T21:12:06.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-252 td/osd-backfill-space/datafile 2020-11-16T21:12:06.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-253 td/osd-backfill-space/datafile 2020-11-16T21:12:06.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-254 td/osd-backfill-space/datafile 2020-11-16T21:12:06.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-255 td/osd-backfill-space/datafile 2020-11-16T21:12:06.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-256 td/osd-backfill-space/datafile 2020-11-16T21:12:06.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-257 td/osd-backfill-space/datafile 2020-11-16T21:12:06.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-258 td/osd-backfill-space/datafile 2020-11-16T21:12:06.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-259 td/osd-backfill-space/datafile 2020-11-16T21:12:06.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-260 td/osd-backfill-space/datafile 2020-11-16T21:12:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-261 td/osd-backfill-space/datafile 2020-11-16T21:12:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-262 td/osd-backfill-space/datafile 2020-11-16T21:12:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-263 td/osd-backfill-space/datafile 2020-11-16T21:12:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-264 td/osd-backfill-space/datafile 2020-11-16T21:12:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-265 td/osd-backfill-space/datafile 2020-11-16T21:12:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-266 td/osd-backfill-space/datafile 2020-11-16T21:12:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-267 td/osd-backfill-space/datafile 2020-11-16T21:12:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-268 td/osd-backfill-space/datafile 2020-11-16T21:12:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-269 td/osd-backfill-space/datafile 2020-11-16T21:12:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-270 td/osd-backfill-space/datafile 2020-11-16T21:12:06.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-271 td/osd-backfill-space/datafile 2020-11-16T21:12:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-272 td/osd-backfill-space/datafile 2020-11-16T21:12:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-273 td/osd-backfill-space/datafile 2020-11-16T21:12:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-274 td/osd-backfill-space/datafile 2020-11-16T21:12:06.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-275 td/osd-backfill-space/datafile 2020-11-16T21:12:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-276 td/osd-backfill-space/datafile 2020-11-16T21:12:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-277 td/osd-backfill-space/datafile 2020-11-16T21:12:06.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-278 td/osd-backfill-space/datafile 2020-11-16T21:12:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-279 td/osd-backfill-space/datafile 2020-11-16T21:12:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-280 td/osd-backfill-space/datafile 2020-11-16T21:12:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-281 td/osd-backfill-space/datafile 2020-11-16T21:12:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-282 td/osd-backfill-space/datafile 2020-11-16T21:12:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-283 td/osd-backfill-space/datafile 2020-11-16T21:12:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-284 td/osd-backfill-space/datafile 2020-11-16T21:12:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-285 td/osd-backfill-space/datafile 2020-11-16T21:12:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-286 td/osd-backfill-space/datafile 2020-11-16T21:12:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-287 td/osd-backfill-space/datafile 2020-11-16T21:12:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-288 td/osd-backfill-space/datafile 2020-11-16T21:12:06.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-289 td/osd-backfill-space/datafile 2020-11-16T21:12:06.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-290 td/osd-backfill-space/datafile 2020-11-16T21:12:06.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-291 td/osd-backfill-space/datafile 2020-11-16T21:12:06.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-292 td/osd-backfill-space/datafile 2020-11-16T21:12:06.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-293 td/osd-backfill-space/datafile 2020-11-16T21:12:06.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-294 td/osd-backfill-space/datafile 2020-11-16T21:12:06.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-295 td/osd-backfill-space/datafile 2020-11-16T21:12:06.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-296 td/osd-backfill-space/datafile 2020-11-16T21:12:06.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-297 td/osd-backfill-space/datafile 2020-11-16T21:12:06.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-298 td/osd-backfill-space/datafile 2020-11-16T21:12:06.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-299 td/osd-backfill-space/datafile 2020-11-16T21:12:06.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-300 td/osd-backfill-space/datafile 2020-11-16T21:12:06.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-301 td/osd-backfill-space/datafile 2020-11-16T21:12:06.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-302 td/osd-backfill-space/datafile 2020-11-16T21:12:06.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-303 td/osd-backfill-space/datafile 2020-11-16T21:12:06.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-304 td/osd-backfill-space/datafile 2020-11-16T21:12:06.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-305 td/osd-backfill-space/datafile 2020-11-16T21:12:06.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-306 td/osd-backfill-space/datafile 2020-11-16T21:12:06.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-307 td/osd-backfill-space/datafile 2020-11-16T21:12:06.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-308 td/osd-backfill-space/datafile 2020-11-16T21:12:06.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-309 td/osd-backfill-space/datafile 2020-11-16T21:12:06.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-310 td/osd-backfill-space/datafile 2020-11-16T21:12:06.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-311 td/osd-backfill-space/datafile 2020-11-16T21:12:06.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-312 td/osd-backfill-space/datafile 2020-11-16T21:12:06.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-313 td/osd-backfill-space/datafile 2020-11-16T21:12:06.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-314 td/osd-backfill-space/datafile 2020-11-16T21:12:06.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-315 td/osd-backfill-space/datafile 2020-11-16T21:12:06.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-316 td/osd-backfill-space/datafile 2020-11-16T21:12:06.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-317 td/osd-backfill-space/datafile 2020-11-16T21:12:06.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-318 td/osd-backfill-space/datafile 2020-11-16T21:12:06.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-319 td/osd-backfill-space/datafile 2020-11-16T21:12:06.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-320 td/osd-backfill-space/datafile 2020-11-16T21:12:06.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-321 td/osd-backfill-space/datafile 2020-11-16T21:12:06.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-322 td/osd-backfill-space/datafile 2020-11-16T21:12:06.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-323 td/osd-backfill-space/datafile 2020-11-16T21:12:06.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-324 td/osd-backfill-space/datafile 2020-11-16T21:12:06.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-325 td/osd-backfill-space/datafile 2020-11-16T21:12:06.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-326 td/osd-backfill-space/datafile 2020-11-16T21:12:06.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-327 td/osd-backfill-space/datafile 2020-11-16T21:12:06.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-328 td/osd-backfill-space/datafile 2020-11-16T21:12:06.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-329 td/osd-backfill-space/datafile 2020-11-16T21:12:06.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-330 td/osd-backfill-space/datafile 2020-11-16T21:12:06.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-331 td/osd-backfill-space/datafile 2020-11-16T21:12:06.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-332 td/osd-backfill-space/datafile 2020-11-16T21:12:06.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-333 td/osd-backfill-space/datafile 2020-11-16T21:12:06.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-334 td/osd-backfill-space/datafile 2020-11-16T21:12:06.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-335 td/osd-backfill-space/datafile 2020-11-16T21:12:06.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-336 td/osd-backfill-space/datafile 2020-11-16T21:12:06.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-337 td/osd-backfill-space/datafile 2020-11-16T21:12:06.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-338 td/osd-backfill-space/datafile 2020-11-16T21:12:06.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-339 td/osd-backfill-space/datafile 2020-11-16T21:12:06.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-340 td/osd-backfill-space/datafile 2020-11-16T21:12:06.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-341 td/osd-backfill-space/datafile 2020-11-16T21:12:06.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-342 td/osd-backfill-space/datafile 2020-11-16T21:12:06.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-343 td/osd-backfill-space/datafile 2020-11-16T21:12:06.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-344 td/osd-backfill-space/datafile 2020-11-16T21:12:06.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-345 td/osd-backfill-space/datafile 2020-11-16T21:12:06.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-346 td/osd-backfill-space/datafile 2020-11-16T21:12:06.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-347 td/osd-backfill-space/datafile 2020-11-16T21:12:06.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-348 td/osd-backfill-space/datafile 2020-11-16T21:12:06.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-349 td/osd-backfill-space/datafile 2020-11-16T21:12:06.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-350 td/osd-backfill-space/datafile 2020-11-16T21:12:06.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-351 td/osd-backfill-space/datafile 2020-11-16T21:12:06.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-352 td/osd-backfill-space/datafile 2020-11-16T21:12:06.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-353 td/osd-backfill-space/datafile 2020-11-16T21:12:06.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-354 td/osd-backfill-space/datafile 2020-11-16T21:12:06.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-355 td/osd-backfill-space/datafile 2020-11-16T21:12:06.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-356 td/osd-backfill-space/datafile 2020-11-16T21:12:06.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-357 td/osd-backfill-space/datafile 2020-11-16T21:12:06.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-358 td/osd-backfill-space/datafile 2020-11-16T21:12:06.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-359 td/osd-backfill-space/datafile 2020-11-16T21:12:06.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-360 td/osd-backfill-space/datafile 2020-11-16T21:12:06.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-361 td/osd-backfill-space/datafile 2020-11-16T21:12:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-362 td/osd-backfill-space/datafile 2020-11-16T21:12:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-363 td/osd-backfill-space/datafile 2020-11-16T21:12:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-364 td/osd-backfill-space/datafile 2020-11-16T21:12:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-365 td/osd-backfill-space/datafile 2020-11-16T21:12:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-366 td/osd-backfill-space/datafile 2020-11-16T21:12:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-367 td/osd-backfill-space/datafile 2020-11-16T21:12:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-368 td/osd-backfill-space/datafile 2020-11-16T21:12:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-369 td/osd-backfill-space/datafile 2020-11-16T21:12:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-370 td/osd-backfill-space/datafile 2020-11-16T21:12:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-371 td/osd-backfill-space/datafile 2020-11-16T21:12:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-372 td/osd-backfill-space/datafile 2020-11-16T21:12:06.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-373 td/osd-backfill-space/datafile 2020-11-16T21:12:06.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-374 td/osd-backfill-space/datafile 2020-11-16T21:12:06.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-375 td/osd-backfill-space/datafile 2020-11-16T21:12:06.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-376 td/osd-backfill-space/datafile 2020-11-16T21:12:06.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-377 td/osd-backfill-space/datafile 2020-11-16T21:12:06.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-378 td/osd-backfill-space/datafile 2020-11-16T21:12:06.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-379 td/osd-backfill-space/datafile 2020-11-16T21:12:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-380 td/osd-backfill-space/datafile 2020-11-16T21:12:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-381 td/osd-backfill-space/datafile 2020-11-16T21:12:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-382 td/osd-backfill-space/datafile 2020-11-16T21:12:06.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-383 td/osd-backfill-space/datafile 2020-11-16T21:12:06.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-384 td/osd-backfill-space/datafile 2020-11-16T21:12:06.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-385 td/osd-backfill-space/datafile 2020-11-16T21:12:06.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-386 td/osd-backfill-space/datafile 2020-11-16T21:12:06.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-387 td/osd-backfill-space/datafile 2020-11-16T21:12:06.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-388 td/osd-backfill-space/datafile 2020-11-16T21:12:06.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-389 td/osd-backfill-space/datafile 2020-11-16T21:12:06.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-390 td/osd-backfill-space/datafile 2020-11-16T21:12:06.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-391 td/osd-backfill-space/datafile 2020-11-16T21:12:06.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-392 td/osd-backfill-space/datafile 2020-11-16T21:12:06.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-393 td/osd-backfill-space/datafile 2020-11-16T21:12:06.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-394 td/osd-backfill-space/datafile 2020-11-16T21:12:06.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-395 td/osd-backfill-space/datafile 2020-11-16T21:12:06.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-396 td/osd-backfill-space/datafile 2020-11-16T21:12:06.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-397 td/osd-backfill-space/datafile 2020-11-16T21:12:06.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-398 td/osd-backfill-space/datafile 2020-11-16T21:12:06.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-399 td/osd-backfill-space/datafile 2020-11-16T21:12:06.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-400 td/osd-backfill-space/datafile 2020-11-16T21:12:06.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-401 td/osd-backfill-space/datafile 2020-11-16T21:12:06.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-402 td/osd-backfill-space/datafile 2020-11-16T21:12:06.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-403 td/osd-backfill-space/datafile 2020-11-16T21:12:06.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-404 td/osd-backfill-space/datafile 2020-11-16T21:12:06.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-405 td/osd-backfill-space/datafile 2020-11-16T21:12:06.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-406 td/osd-backfill-space/datafile 2020-11-16T21:12:06.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-407 td/osd-backfill-space/datafile 2020-11-16T21:12:06.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-408 td/osd-backfill-space/datafile 2020-11-16T21:12:06.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-409 td/osd-backfill-space/datafile 2020-11-16T21:12:06.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-410 td/osd-backfill-space/datafile 2020-11-16T21:12:06.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-411 td/osd-backfill-space/datafile 2020-11-16T21:12:06.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-412 td/osd-backfill-space/datafile 2020-11-16T21:12:06.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-413 td/osd-backfill-space/datafile 2020-11-16T21:12:06.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-414 td/osd-backfill-space/datafile 2020-11-16T21:12:06.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-415 td/osd-backfill-space/datafile 2020-11-16T21:12:06.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-416 td/osd-backfill-space/datafile 2020-11-16T21:12:06.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-417 td/osd-backfill-space/datafile 2020-11-16T21:12:06.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-418 td/osd-backfill-space/datafile 2020-11-16T21:12:06.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-419 td/osd-backfill-space/datafile 2020-11-16T21:12:06.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-420 td/osd-backfill-space/datafile 2020-11-16T21:12:06.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-421 td/osd-backfill-space/datafile 2020-11-16T21:12:06.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-422 td/osd-backfill-space/datafile 2020-11-16T21:12:06.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-423 td/osd-backfill-space/datafile 2020-11-16T21:12:06.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-424 td/osd-backfill-space/datafile 2020-11-16T21:12:06.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-425 td/osd-backfill-space/datafile 2020-11-16T21:12:06.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-426 td/osd-backfill-space/datafile 2020-11-16T21:12:06.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-427 td/osd-backfill-space/datafile 2020-11-16T21:12:06.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-428 td/osd-backfill-space/datafile 2020-11-16T21:12:06.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-429 td/osd-backfill-space/datafile 2020-11-16T21:12:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-430 td/osd-backfill-space/datafile 2020-11-16T21:12:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-431 td/osd-backfill-space/datafile 2020-11-16T21:12:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-432 td/osd-backfill-space/datafile 2020-11-16T21:12:06.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-433 td/osd-backfill-space/datafile 2020-11-16T21:12:06.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-434 td/osd-backfill-space/datafile 2020-11-16T21:12:06.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-435 td/osd-backfill-space/datafile 2020-11-16T21:12:06.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-436 td/osd-backfill-space/datafile 2020-11-16T21:12:06.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-437 td/osd-backfill-space/datafile 2020-11-16T21:12:06.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-438 td/osd-backfill-space/datafile 2020-11-16T21:12:06.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-439 td/osd-backfill-space/datafile 2020-11-16T21:12:06.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-440 td/osd-backfill-space/datafile 2020-11-16T21:12:06.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-441 td/osd-backfill-space/datafile 2020-11-16T21:12:06.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-442 td/osd-backfill-space/datafile 2020-11-16T21:12:06.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-443 td/osd-backfill-space/datafile 2020-11-16T21:12:06.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-444 td/osd-backfill-space/datafile 2020-11-16T21:12:06.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-445 td/osd-backfill-space/datafile 2020-11-16T21:12:06.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-446 td/osd-backfill-space/datafile 2020-11-16T21:12:06.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-447 td/osd-backfill-space/datafile 2020-11-16T21:12:06.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-448 td/osd-backfill-space/datafile 2020-11-16T21:12:06.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-449 td/osd-backfill-space/datafile 2020-11-16T21:12:06.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-450 td/osd-backfill-space/datafile 2020-11-16T21:12:06.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-451 td/osd-backfill-space/datafile 2020-11-16T21:12:06.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-452 td/osd-backfill-space/datafile 2020-11-16T21:12:06.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-453 td/osd-backfill-space/datafile 2020-11-16T21:12:06.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-454 td/osd-backfill-space/datafile 2020-11-16T21:12:06.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-455 td/osd-backfill-space/datafile 2020-11-16T21:12:06.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-456 td/osd-backfill-space/datafile 2020-11-16T21:12:06.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-457 td/osd-backfill-space/datafile 2020-11-16T21:12:06.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-458 td/osd-backfill-space/datafile 2020-11-16T21:12:06.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-459 td/osd-backfill-space/datafile 2020-11-16T21:12:06.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-460 td/osd-backfill-space/datafile 2020-11-16T21:12:06.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-461 td/osd-backfill-space/datafile 2020-11-16T21:12:06.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-462 td/osd-backfill-space/datafile 2020-11-16T21:12:06.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-463 td/osd-backfill-space/datafile 2020-11-16T21:12:06.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-464 td/osd-backfill-space/datafile 2020-11-16T21:12:06.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-465 td/osd-backfill-space/datafile 2020-11-16T21:12:06.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-466 td/osd-backfill-space/datafile 2020-11-16T21:12:06.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-467 td/osd-backfill-space/datafile 2020-11-16T21:12:06.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-468 td/osd-backfill-space/datafile 2020-11-16T21:12:06.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-469 td/osd-backfill-space/datafile 2020-11-16T21:12:06.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-470 td/osd-backfill-space/datafile 2020-11-16T21:12:06.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-471 td/osd-backfill-space/datafile 2020-11-16T21:12:06.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-472 td/osd-backfill-space/datafile 2020-11-16T21:12:06.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-473 td/osd-backfill-space/datafile 2020-11-16T21:12:06.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-474 td/osd-backfill-space/datafile 2020-11-16T21:12:06.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-475 td/osd-backfill-space/datafile 2020-11-16T21:12:06.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-476 td/osd-backfill-space/datafile 2020-11-16T21:12:06.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-477 td/osd-backfill-space/datafile 2020-11-16T21:12:06.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-478 td/osd-backfill-space/datafile 2020-11-16T21:12:06.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-479 td/osd-backfill-space/datafile 2020-11-16T21:12:06.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-480 td/osd-backfill-space/datafile 2020-11-16T21:12:06.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-481 td/osd-backfill-space/datafile 2020-11-16T21:12:06.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-482 td/osd-backfill-space/datafile 2020-11-16T21:12:06.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-483 td/osd-backfill-space/datafile 2020-11-16T21:12:06.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-484 td/osd-backfill-space/datafile 2020-11-16T21:12:06.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-485 td/osd-backfill-space/datafile 2020-11-16T21:12:06.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-486 td/osd-backfill-space/datafile 2020-11-16T21:12:06.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-487 td/osd-backfill-space/datafile 2020-11-16T21:12:06.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-488 td/osd-backfill-space/datafile 2020-11-16T21:12:06.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-489 td/osd-backfill-space/datafile 2020-11-16T21:12:06.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-490 td/osd-backfill-space/datafile 2020-11-16T21:12:06.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-491 td/osd-backfill-space/datafile 2020-11-16T21:12:06.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-492 td/osd-backfill-space/datafile 2020-11-16T21:12:06.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-493 td/osd-backfill-space/datafile 2020-11-16T21:12:06.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-494 td/osd-backfill-space/datafile 2020-11-16T21:12:06.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-495 td/osd-backfill-space/datafile 2020-11-16T21:12:06.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-496 td/osd-backfill-space/datafile 2020-11-16T21:12:06.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-497 td/osd-backfill-space/datafile 2020-11-16T21:12:06.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-498 td/osd-backfill-space/datafile 2020-11-16T21:12:06.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-499 td/osd-backfill-space/datafile 2020-11-16T21:12:06.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-500 td/osd-backfill-space/datafile 2020-11-16T21:12:06.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-501 td/osd-backfill-space/datafile 2020-11-16T21:12:06.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-502 td/osd-backfill-space/datafile 2020-11-16T21:12:06.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-503 td/osd-backfill-space/datafile 2020-11-16T21:12:06.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-504 td/osd-backfill-space/datafile 2020-11-16T21:12:06.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-505 td/osd-backfill-space/datafile 2020-11-16T21:12:06.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-506 td/osd-backfill-space/datafile 2020-11-16T21:12:06.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-507 td/osd-backfill-space/datafile 2020-11-16T21:12:06.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-508 td/osd-backfill-space/datafile 2020-11-16T21:12:06.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-509 td/osd-backfill-space/datafile 2020-11-16T21:12:06.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-510 td/osd-backfill-space/datafile 2020-11-16T21:12:06.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-511 td/osd-backfill-space/datafile 2020-11-16T21:12:06.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-512 td/osd-backfill-space/datafile 2020-11-16T21:12:06.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-513 td/osd-backfill-space/datafile 2020-11-16T21:12:06.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-514 td/osd-backfill-space/datafile 2020-11-16T21:12:06.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-515 td/osd-backfill-space/datafile 2020-11-16T21:12:06.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-516 td/osd-backfill-space/datafile 2020-11-16T21:12:06.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-517 td/osd-backfill-space/datafile 2020-11-16T21:12:06.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-518 td/osd-backfill-space/datafile 2020-11-16T21:12:06.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-519 td/osd-backfill-space/datafile 2020-11-16T21:12:06.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-520 td/osd-backfill-space/datafile 2020-11-16T21:12:06.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-521 td/osd-backfill-space/datafile 2020-11-16T21:12:06.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-522 td/osd-backfill-space/datafile 2020-11-16T21:12:06.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-523 td/osd-backfill-space/datafile 2020-11-16T21:12:06.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-524 td/osd-backfill-space/datafile 2020-11-16T21:12:06.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-525 td/osd-backfill-space/datafile 2020-11-16T21:12:06.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-526 td/osd-backfill-space/datafile 2020-11-16T21:12:06.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-527 td/osd-backfill-space/datafile 2020-11-16T21:12:06.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-528 td/osd-backfill-space/datafile 2020-11-16T21:12:06.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-529 td/osd-backfill-space/datafile 2020-11-16T21:12:06.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-530 td/osd-backfill-space/datafile 2020-11-16T21:12:06.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-531 td/osd-backfill-space/datafile 2020-11-16T21:12:06.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-532 td/osd-backfill-space/datafile 2020-11-16T21:12:06.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-533 td/osd-backfill-space/datafile 2020-11-16T21:12:06.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-534 td/osd-backfill-space/datafile 2020-11-16T21:12:06.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-535 td/osd-backfill-space/datafile 2020-11-16T21:12:06.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-536 td/osd-backfill-space/datafile 2020-11-16T21:12:06.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-537 td/osd-backfill-space/datafile 2020-11-16T21:12:06.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-538 td/osd-backfill-space/datafile 2020-11-16T21:12:06.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-539 td/osd-backfill-space/datafile 2020-11-16T21:12:06.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-540 td/osd-backfill-space/datafile 2020-11-16T21:12:06.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-541 td/osd-backfill-space/datafile 2020-11-16T21:12:06.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-542 td/osd-backfill-space/datafile 2020-11-16T21:12:06.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-543 td/osd-backfill-space/datafile 2020-11-16T21:12:06.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-544 td/osd-backfill-space/datafile 2020-11-16T21:12:06.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-545 td/osd-backfill-space/datafile 2020-11-16T21:12:06.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-546 td/osd-backfill-space/datafile 2020-11-16T21:12:06.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-547 td/osd-backfill-space/datafile 2020-11-16T21:12:06.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-548 td/osd-backfill-space/datafile 2020-11-16T21:12:06.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-549 td/osd-backfill-space/datafile 2020-11-16T21:12:06.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-550 td/osd-backfill-space/datafile 2020-11-16T21:12:06.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-551 td/osd-backfill-space/datafile 2020-11-16T21:12:06.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-552 td/osd-backfill-space/datafile 2020-11-16T21:12:06.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-553 td/osd-backfill-space/datafile 2020-11-16T21:12:06.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-554 td/osd-backfill-space/datafile 2020-11-16T21:12:06.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-555 td/osd-backfill-space/datafile 2020-11-16T21:12:06.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-556 td/osd-backfill-space/datafile 2020-11-16T21:12:06.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-557 td/osd-backfill-space/datafile 2020-11-16T21:12:06.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-558 td/osd-backfill-space/datafile 2020-11-16T21:12:06.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-559 td/osd-backfill-space/datafile 2020-11-16T21:12:06.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-560 td/osd-backfill-space/datafile 2020-11-16T21:12:06.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-561 td/osd-backfill-space/datafile 2020-11-16T21:12:06.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-562 td/osd-backfill-space/datafile 2020-11-16T21:12:06.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-563 td/osd-backfill-space/datafile 2020-11-16T21:12:06.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-564 td/osd-backfill-space/datafile 2020-11-16T21:12:06.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-565 td/osd-backfill-space/datafile 2020-11-16T21:12:06.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-566 td/osd-backfill-space/datafile 2020-11-16T21:12:06.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-567 td/osd-backfill-space/datafile 2020-11-16T21:12:06.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-568 td/osd-backfill-space/datafile 2020-11-16T21:12:06.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-569 td/osd-backfill-space/datafile 2020-11-16T21:12:06.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-570 td/osd-backfill-space/datafile 2020-11-16T21:12:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-571 td/osd-backfill-space/datafile 2020-11-16T21:12:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-572 td/osd-backfill-space/datafile 2020-11-16T21:12:06.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-573 td/osd-backfill-space/datafile 2020-11-16T21:12:06.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-574 td/osd-backfill-space/datafile 2020-11-16T21:12:06.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-575 td/osd-backfill-space/datafile 2020-11-16T21:12:06.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-576 td/osd-backfill-space/datafile 2020-11-16T21:12:06.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-577 td/osd-backfill-space/datafile 2020-11-16T21:12:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-578 td/osd-backfill-space/datafile 2020-11-16T21:12:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-579 td/osd-backfill-space/datafile 2020-11-16T21:12:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-580 td/osd-backfill-space/datafile 2020-11-16T21:12:06.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-581 td/osd-backfill-space/datafile 2020-11-16T21:12:06.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-582 td/osd-backfill-space/datafile 2020-11-16T21:12:06.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-583 td/osd-backfill-space/datafile 2020-11-16T21:12:06.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-584 td/osd-backfill-space/datafile 2020-11-16T21:12:06.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-585 td/osd-backfill-space/datafile 2020-11-16T21:12:06.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-586 td/osd-backfill-space/datafile 2020-11-16T21:12:06.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-587 td/osd-backfill-space/datafile 2020-11-16T21:12:06.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-588 td/osd-backfill-space/datafile 2020-11-16T21:12:06.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-589 td/osd-backfill-space/datafile 2020-11-16T21:12:06.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-590 td/osd-backfill-space/datafile 2020-11-16T21:12:06.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-591 td/osd-backfill-space/datafile 2020-11-16T21:12:06.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-592 td/osd-backfill-space/datafile 2020-11-16T21:12:06.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-593 td/osd-backfill-space/datafile 2020-11-16T21:12:06.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-594 td/osd-backfill-space/datafile 2020-11-16T21:12:06.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-595 td/osd-backfill-space/datafile 2020-11-16T21:12:06.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-596 td/osd-backfill-space/datafile 2020-11-16T21:12:06.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-597 td/osd-backfill-space/datafile 2020-11-16T21:12:06.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-598 td/osd-backfill-space/datafile 2020-11-16T21:12:06.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-599 td/osd-backfill-space/datafile 2020-11-16T21:12:06.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:06.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-600 td/osd-backfill-space/datafile 2020-11-16T21:12:06.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:487: TEST_backfill_multi_partial: ceph pg dump pgs 2020-11-16T21:12:06.995 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:12:06.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:489: TEST_backfill_multi_partial: _objectstore_tool_nowait td/osd-backfill-space 2 --op export --pgid 2.0 --file td/osd-backfill-space/export.out 2020-11-16T21:12:06.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/osd-backfill-space 2020-11-16T21:12:06.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-11-16T21:12:06.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=2 2020-11-16T21:12:06.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-11-16T21:12:06.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/osd-backfill-space TERM osd.2 2020-11-16T21:12:06.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:12:06.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:12:06.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:12:06.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:12:06.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:12:06.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:12:06.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-backfill-space 2 --op export --pgid 2.0 --file td/osd-backfill-space/export.out 2020-11-16T21:12:06.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-space 2020-11-16T21:12:06.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T21:12:06.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=2 2020-11-16T21:12:06.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T21:12:06.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-space/2 2020-11-16T21:12:06.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-space/2 --op export --pgid 2.0 --file td/osd-backfill-space/export.out 2020-11-16T21:12:06.998 INFO:tasks.workunit.client.0.smithi189.stderr:Exporting 2.0 info 2.0( v 40'600 (40'500,40'600] local-lis/les=38/39 n=600 ec=26/26 lis/c=38/38 les/c/f=39/39/0 sis=38) 2020-11-16T21:12:06.999 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:00f101c6:::obj-1-121:head# 2020-11-16T21:12:06.999 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0168051e:::obj-1-571:head# 2020-11-16T21:12:06.999 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:01b67738:::obj-1-182:head# 2020-11-16T21:12:06.999 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:040dbc59:::obj-1-23:head# 2020-11-16T21:12:06.999 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0415ce6e:::obj-1-20:head# 2020-11-16T21:12:06.999 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:058b28d7:::obj-1-572:head# 2020-11-16T21:12:07.000 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:059d2e5c:::obj-1-87:head# 2020-11-16T21:12:07.000 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:05a823ca:::obj-1-52:head# 2020-11-16T21:12:07.000 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:05b8b7fe:::obj-1-119:head# 2020-11-16T21:12:07.000 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:05c9492b:::obj-1-78:head# 2020-11-16T21:12:07.000 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:05e91e96:::obj-1-393:head# 2020-11-16T21:12:07.001 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:07bdb780:::obj-1-204:head# 2020-11-16T21:12:07.001 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:07cfbd49:::obj-1-318:head# 2020-11-16T21:12:07.001 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:084a829f:::obj-1-112:head# 2020-11-16T21:12:07.001 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:08cedd34:::obj-1-438:head# 2020-11-16T21:12:07.001 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0941af69:::obj-1-117:head# 2020-11-16T21:12:07.002 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:09607265:::obj-1-429:head# 2020-11-16T21:12:07.002 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0be86f7e:::obj-1-568:head# 2020-11-16T21:12:07.002 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0ca5819d:::obj-1-327:head# 2020-11-16T21:12:07.002 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0cd02051:::obj-1-208:head# 2020-11-16T21:12:07.002 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0d0264b5:::obj-1-171:head# 2020-11-16T21:12:07.003 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0d195060:::obj-1-420:head# 2020-11-16T21:12:07.003 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0d362b19:::obj-1-219:head# 2020-11-16T21:12:07.003 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0dea6f4d:::obj-1-61:head# 2020-11-16T21:12:07.003 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0ecd8390:::obj-1-132:head# 2020-11-16T21:12:07.003 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0f49e2fa:::obj-1-200:head# 2020-11-16T21:12:07.004 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0f7bb95f:::obj-1-268:head# 2020-11-16T21:12:07.004 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0fabf252:::obj-1-441:head# 2020-11-16T21:12:07.004 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:0fb1697a:::obj-1-443:head# 2020-11-16T21:12:07.004 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:105d3efe:::obj-1-490:head# 2020-11-16T21:12:07.004 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:10f52c01:::obj-1-449:head# 2020-11-16T21:12:07.004 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:11965425:::obj-1-335:head# 2020-11-16T21:12:07.005 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:11c9f5f8:::obj-1-565:head# 2020-11-16T21:12:07.005 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:11cfcfb6:::obj-1-588:head# 2020-11-16T21:12:07.005 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:11d46f0a:::obj-1-387:head# 2020-11-16T21:12:07.005 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:11ebb0d6:::obj-1-214:head# 2020-11-16T21:12:07.005 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1295e402:::obj-1-178:head# 2020-11-16T21:12:07.006 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:12c996c1:::obj-1-517:head# 2020-11-16T21:12:07.006 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:133dec15:::obj-1-475:head# 2020-11-16T21:12:07.006 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1475c981:::obj-1-505:head# 2020-11-16T21:12:07.006 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:148e92a4:::obj-1-509:head# 2020-11-16T21:12:07.006 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1490b121:::obj-1-172:head# 2020-11-16T21:12:07.007 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:14ce082c:::obj-1-99:head# 2020-11-16T21:12:07.007 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1552ad99:::obj-1-41:head# 2020-11-16T21:12:07.007 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:15544270:::obj-1-271:head# 2020-11-16T21:12:07.007 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1589d957:::obj-1-86:head# 2020-11-16T21:12:07.007 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:16db3947:::obj-1-477:head# 2020-11-16T21:12:07.007 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:16dc98bd:::obj-1-375:head# 2020-11-16T21:12:07.008 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:17e54db3:::obj-1-196:head# 2020-11-16T21:12:07.008 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:17f2ad3f:::obj-1-10:head# 2020-11-16T21:12:07.008 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:18075e41:::obj-1-55:head# 2020-11-16T21:12:07.008 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:18086253:::obj-1-71:head# 2020-11-16T21:12:07.008 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:191cf8d8:::obj-1-372:head# 2020-11-16T21:12:07.009 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:19311f70:::obj-1-454:head# 2020-11-16T21:12:07.009 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:198318e4:::obj-1-464:head# 2020-11-16T21:12:07.009 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1999d806:::obj-1-163:head# 2020-11-16T21:12:07.009 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1a92cab7:::obj-1-412:head# 2020-11-16T21:12:07.009 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1b16c0f6:::obj-1-210:head# 2020-11-16T21:12:07.009 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1b53ed46:::obj-1-577:head# 2020-11-16T21:12:07.010 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1b5c5a8d:::obj-1-556:head# 2020-11-16T21:12:07.010 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1c12c05e:::obj-1-79:head# 2020-11-16T21:12:07.010 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1c571ba5:::obj-1-262:head# 2020-11-16T21:12:07.010 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1c5f42b7:::obj-1-92:head# 2020-11-16T21:12:07.010 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1c6ad881:::obj-1-312:head# 2020-11-16T21:12:07.011 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1c89877b:::obj-1-142:head# 2020-11-16T21:12:07.011 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1e4168ca:::obj-1-282:head# 2020-11-16T21:12:07.011 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1e424445:::obj-1-484:head# 2020-11-16T21:12:07.011 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1e467959:::obj-1-22:head# 2020-11-16T21:12:07.011 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1eaf7e7c:::obj-1-392:head# 2020-11-16T21:12:07.012 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1f06c86d:::obj-1-307:head# 2020-11-16T21:12:07.012 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1f2ee1f8:::obj-1-306:head# 2020-11-16T21:12:07.012 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:1f4ef040:::obj-1-349:head# 2020-11-16T21:12:07.012 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2039481f:::obj-1-545:head# 2020-11-16T21:12:07.012 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:22233df0:::obj-1-543:head# 2020-11-16T21:12:07.013 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2225184e:::obj-1-266:head# 2020-11-16T21:12:07.013 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:234766bb:::obj-1-281:head# 2020-11-16T21:12:07.013 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2372e022:::obj-1-205:head# 2020-11-16T21:12:07.013 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2379547b:::obj-1-113:head# 2020-11-16T21:12:07.013 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:23ae14bf:::obj-1-531:head# 2020-11-16T21:12:07.013 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:23bc5f90:::obj-1-452:head# 2020-11-16T21:12:07.014 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:23fbef3d:::obj-1-82:head# 2020-11-16T21:12:07.014 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:24106c2b:::obj-1-232:head# 2020-11-16T21:12:07.014 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2529b4e4:::obj-1-75:head# 2020-11-16T21:12:07.014 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:25a24ae7:::obj-1-459:head# 2020-11-16T21:12:07.014 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:26627aa9:::obj-1-7:head# 2020-11-16T21:12:07.015 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:26b77221:::obj-1-227:head# 2020-11-16T21:12:07.015 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:27450826:::obj-1-344:head# 2020-11-16T21:12:07.015 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:278905af:::obj-1-592:head# 2020-11-16T21:12:07.015 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:27ad0d2d:::obj-1-136:head# 2020-11-16T21:12:07.015 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:28a8c213:::obj-1-118:head# 2020-11-16T21:12:07.015 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:28c0c009:::obj-1-74:head# 2020-11-16T21:12:07.016 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2a214708:::obj-1-260:head# 2020-11-16T21:12:07.016 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2a67e195:::obj-1-474:head# 2020-11-16T21:12:07.016 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2aa852ff:::obj-1-5:head# 2020-11-16T21:12:07.016 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2ba1fb52:::obj-1-563:head# 2020-11-16T21:12:07.016 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2bae8cb9:::obj-1-320:head# 2020-11-16T21:12:07.016 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2c569d02:::obj-1-139:head# 2020-11-16T21:12:07.017 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2c71d691:::obj-1-201:head# 2020-11-16T21:12:07.017 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2c88e92c:::obj-1-389:head# 2020-11-16T21:12:07.017 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2d659109:::obj-1-580:head# 2020-11-16T21:12:07.017 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2d74ea60:::obj-1-230:head# 2020-11-16T21:12:07.018 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2de96edf:::obj-1-233:head# 2020-11-16T21:12:07.018 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2e43da40:::obj-1-382:head# 2020-11-16T21:12:07.018 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2e530c9f:::obj-1-358:head# 2020-11-16T21:12:07.018 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:2ed02239:::obj-1-81:head# 2020-11-16T21:12:07.018 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:30264a0c:::obj-1-373:head# 2020-11-16T21:12:07.019 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:302ca5e7:::obj-1-206:head# 2020-11-16T21:12:07.019 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:309878e8:::obj-1-508:head# 2020-11-16T21:12:07.019 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:309a0ac3:::obj-1-188:head# 2020-11-16T21:12:07.019 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:30f030e9:::obj-1-53:head# 2020-11-16T21:12:07.019 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:31242a73:::obj-1-528:head# 2020-11-16T21:12:07.020 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:31463825:::obj-1-34:head# 2020-11-16T21:12:07.020 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3166dc86:::obj-1-355:head# 2020-11-16T21:12:07.020 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:31baa2fa:::obj-1-252:head# 2020-11-16T21:12:07.020 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:31d8935e:::obj-1-193:head# 2020-11-16T21:12:07.020 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:32670b14:::obj-1-305:head# 2020-11-16T21:12:07.021 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:33da02fa:::obj-1-114:head# 2020-11-16T21:12:07.021 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:34373133:::obj-1-446:head# 2020-11-16T21:12:07.021 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:347b7083:::obj-1-89:head# 2020-11-16T21:12:07.021 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:349125c1:::obj-1-148:head# 2020-11-16T21:12:07.021 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:34b80eee:::obj-1-138:head# 2020-11-16T21:12:07.022 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:34db48f8:::obj-1-49:head# 2020-11-16T21:12:07.022 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3669ff0c:::obj-1-313:head# 2020-11-16T21:12:07.022 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:36c1b610:::obj-1-249:head# 2020-11-16T21:12:07.022 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:374c250b:::obj-1-324:head# 2020-11-16T21:12:07.022 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3761a149:::obj-1-342:head# 2020-11-16T21:12:07.023 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:379ff95a:::obj-1-341:head# 2020-11-16T21:12:07.023 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:37dc15a6:::obj-1-26:head# 2020-11-16T21:12:07.023 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:38130e33:::obj-1-552:head# 2020-11-16T21:12:07.023 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:39c44880:::obj-1-538:head# 2020-11-16T21:12:07.023 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3a08dec9:::obj-1-368:head# 2020-11-16T21:12:07.024 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3a2b46a6:::obj-1-407:head# 2020-11-16T21:12:07.024 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3a7dfe0a:::obj-1-285:head# 2020-11-16T21:12:07.024 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3aebb144:::obj-1-248:head# 2020-11-16T21:12:07.024 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3b08b36f:::obj-1-310:head# 2020-11-16T21:12:07.024 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3b28ffa3:::obj-1-295:head# 2020-11-16T21:12:07.024 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3ba79bd1:::obj-1-444:head# 2020-11-16T21:12:07.025 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3c05df8d:::obj-1-319:head# 2020-11-16T21:12:07.025 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3c57e6a9:::obj-1-436:head# 2020-11-16T21:12:07.025 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3c73d8ee:::obj-1-298:head# 2020-11-16T21:12:07.025 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3cb6ae2a:::obj-1-400:head# 2020-11-16T21:12:07.025 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3cb71491:::obj-1-328:head# 2020-11-16T21:12:07.026 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3cb842fe:::obj-1-541:head# 2020-11-16T21:12:07.026 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3cf6835e:::obj-1-496:head# 2020-11-16T21:12:07.026 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3cf78507:::obj-1-573:head# 2020-11-16T21:12:07.026 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3d0dab84:::obj-1-554:head# 2020-11-16T21:12:07.026 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3d1a5f9d:::obj-1-323:head# 2020-11-16T21:12:07.027 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3da5f3a2:::obj-1-414:head# 2020-11-16T21:12:07.027 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3dbb65f0:::obj-1-440:head# 2020-11-16T21:12:07.027 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3dc94618:::obj-1-223:head# 2020-11-16T21:12:07.027 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3df3d5ab:::obj-1-25:head# 2020-11-16T21:12:07.027 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3e167988:::obj-1-336:head# 2020-11-16T21:12:07.028 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3e4ef3bd:::obj-1-104:head# 2020-11-16T21:12:07.028 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3e9e1efd:::obj-1-357:head# 2020-11-16T21:12:07.028 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3ebbe3a7:::obj-1-42:head# 2020-11-16T21:12:07.028 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3ec80373:::obj-1-403:head# 2020-11-16T21:12:07.028 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3ed086e4:::obj-1-13:head# 2020-11-16T21:12:07.028 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:3fe8e157:::obj-1-164:head# 2020-11-16T21:12:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:401ddfd4:::obj-1-379:head# 2020-11-16T21:12:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4092f6cd:::obj-1-430:head# 2020-11-16T21:12:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:414bd7cc:::obj-1-562:head# 2020-11-16T21:12:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:422bd9d0:::obj-1-287:head# 2020-11-16T21:12:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4255002d:::obj-1-578:head# 2020-11-16T21:12:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4270725c:::obj-1-109:head# 2020-11-16T21:12:07.030 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:42836669:::obj-1-31:head# 2020-11-16T21:12:07.030 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:428d93ef:::obj-1-32:head# 2020-11-16T21:12:07.030 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4327ef5b:::obj-1-453:head# 2020-11-16T21:12:07.030 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:440eb07b:::obj-1-236:head# 2020-11-16T21:12:07.030 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:442a028a:::obj-1-365:head# 2020-11-16T21:12:07.031 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:449ca54a:::obj-1-161:head# 2020-11-16T21:12:07.031 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:45128c3b:::obj-1-343:head# 2020-11-16T21:12:07.031 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:45801bf6:::obj-1-264:head# 2020-11-16T21:12:07.031 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:45896f61:::obj-1-48:head# 2020-11-16T21:12:07.031 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:459b7f1c:::obj-1-190:head# 2020-11-16T21:12:07.032 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:466af96a:::obj-1-486:head# 2020-11-16T21:12:07.032 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:47387ce4:::obj-1-575:head# 2020-11-16T21:12:07.032 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:47c94cc8:::obj-1-564:head# 2020-11-16T21:12:07.032 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:47d7f8ed:::obj-1-544:head# 2020-11-16T21:12:07.032 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:47fc0ec3:::obj-1-558:head# 2020-11-16T21:12:07.033 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:48ba42d8:::obj-1-156:head# 2020-11-16T21:12:07.033 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:494e269e:::obj-1-600:head# 2020-11-16T21:12:07.033 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:49646095:::obj-1-73:head# 2020-11-16T21:12:07.033 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:49eb65ab:::obj-1-374:head# 2020-11-16T21:12:07.033 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4a0075a4:::obj-1-432:head# 2020-11-16T21:12:07.033 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4af1ffc2:::obj-1-72:head# 2020-11-16T21:12:07.034 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4bc1c46f:::obj-1-346:head# 2020-11-16T21:12:07.034 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4bd55e68:::obj-1-93:head# 2020-11-16T21:12:07.034 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4c6d2760:::obj-1-198:head# 2020-11-16T21:12:07.034 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4cb9be3b:::obj-1-451:head# 2020-11-16T21:12:07.035 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4d260a64:::obj-1-514:head# 2020-11-16T21:12:07.035 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4d7d6284:::obj-1-583:head# 2020-11-16T21:12:07.035 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4e79ac02:::obj-1-437:head# 2020-11-16T21:12:07.035 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4e980099:::obj-1-507:head# 2020-11-16T21:12:07.035 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4ea8f99c:::obj-1-211:head# 2020-11-16T21:12:07.035 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4eabdc2a:::obj-1-590:head# 2020-11-16T21:12:07.036 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4f97fd73:::obj-1-243:head# 2020-11-16T21:12:07.036 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4fe00224:::obj-1-229:head# 2020-11-16T21:12:07.036 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4fead9ab:::obj-1-476:head# 2020-11-16T21:12:07.036 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:4ffddfd7:::obj-1-462:head# 2020-11-16T21:12:07.036 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5085c04d:::obj-1-100:head# 2020-11-16T21:12:07.036 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:50a8b465:::obj-1-574:head# 2020-11-16T21:12:07.037 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:518a151f:::obj-1-189:head# 2020-11-16T21:12:07.037 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:519afc95:::obj-1-314:head# 2020-11-16T21:12:07.037 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:52199781:::obj-1-202:head# 2020-11-16T21:12:07.037 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:52313062:::obj-1-337:head# 2020-11-16T21:12:07.037 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:526a5196:::obj-1-225:head# 2020-11-16T21:12:07.038 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:53128d32:::obj-1-483:head# 2020-11-16T21:12:07.038 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:53c2bb21:::obj-1-532:head# 2020-11-16T21:12:07.038 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:53fa4f3c:::obj-1-423:head# 2020-11-16T21:12:07.038 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:541c1d30:::obj-1-143:head# 2020-11-16T21:12:07.038 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:548d20c3:::obj-1-352:head# 2020-11-16T21:12:07.038 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:54e4c6f9:::obj-1-101:head# 2020-11-16T21:12:07.039 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:55016e06:::obj-1-276:head# 2020-11-16T21:12:07.039 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5510c784:::obj-1-491:head# 2020-11-16T21:12:07.039 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:551275ef:::obj-1-239:head# 2020-11-16T21:12:07.039 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:556df001:::obj-1-46:head# 2020-11-16T21:12:07.039 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:558fb5b1:::obj-1-536:head# 2020-11-16T21:12:07.040 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:56390d98:::obj-1-64:head# 2020-11-16T21:12:07.040 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:564f962a:::obj-1-125:head# 2020-11-16T21:12:07.040 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:56936e94:::obj-1-251:head# 2020-11-16T21:12:07.040 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:57585d3d:::obj-1-294:head# 2020-11-16T21:12:07.040 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5764cf7b:::obj-1-479:head# 2020-11-16T21:12:07.041 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5782341a:::obj-1-218:head# 2020-11-16T21:12:07.041 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:57d07a1a:::obj-1-317:head# 2020-11-16T21:12:07.041 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:58700205:::obj-1-176:head# 2020-11-16T21:12:07.041 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:58aee718:::obj-1-434:head# 2020-11-16T21:12:07.041 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:594c9d52:::obj-1-458:head# 2020-11-16T21:12:07.041 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5a70b555:::obj-1-279:head# 2020-11-16T21:12:07.042 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5b02568b:::obj-1-586:head# 2020-11-16T21:12:07.042 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5b69cdcf:::obj-1-391:head# 2020-11-16T21:12:07.042 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5cf151b5:::obj-1-510:head# 2020-11-16T21:12:07.042 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5d7fd07f:::obj-1-152:head# 2020-11-16T21:12:07.042 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5de288d7:::obj-1-235:head# 2020-11-16T21:12:07.043 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5ec497cd:::obj-1-44:head# 2020-11-16T21:12:07.043 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5fb3a09f:::obj-1-518:head# 2020-11-16T21:12:07.043 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:5ff3c2ef:::obj-1-470:head# 2020-11-16T21:12:07.043 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:601efcb1:::obj-1-549:head# 2020-11-16T21:12:07.043 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:60831338:::obj-1-263:head# 2020-11-16T21:12:07.044 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6092777b:::obj-1-145:head# 2020-11-16T21:12:07.044 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:60b17206:::obj-1-411:head# 2020-11-16T21:12:07.044 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:610c957c:::obj-1-278:head# 2020-11-16T21:12:07.044 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:613b5ef4:::obj-1-504:head# 2020-11-16T21:12:07.044 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:61ed7d60:::obj-1-274:head# 2020-11-16T21:12:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:621b6751:::obj-1-212:head# 2020-11-16T21:12:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:62b0aba9:::obj-1-253:head# 2020-11-16T21:12:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:64017240:::obj-1-102:head# 2020-11-16T21:12:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:645f4f49:::obj-1-332:head# 2020-11-16T21:12:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:65db82b9:::obj-1-334:head# 2020-11-16T21:12:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:667422e5:::obj-1-465:head# 2020-11-16T21:12:07.046 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:66c50bc2:::obj-1-322:head# 2020-11-16T21:12:07.046 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:681a4577:::obj-1-511:head# 2020-11-16T21:12:07.046 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6874e007:::obj-1-398:head# 2020-11-16T21:12:07.046 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:693746c1:::obj-1-582:head# 2020-11-16T21:12:07.046 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:69638b0c:::obj-1-8:head# 2020-11-16T21:12:07.047 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6967de6e:::obj-1-331:head# 2020-11-16T21:12:07.047 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:69da97aa:::obj-1-340:head# 2020-11-16T21:12:07.047 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6b3a50a6:::obj-1-144:head# 2020-11-16T21:12:07.047 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6be4ef26:::obj-1-579:head# 2020-11-16T21:12:07.047 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6d735549:::obj-1-599:head# 2020-11-16T21:12:07.047 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6e22c25e:::obj-1-9:head# 2020-11-16T21:12:07.048 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6e2ced4e:::obj-1-370:head# 2020-11-16T21:12:07.048 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6e961169:::obj-1-428:head# 2020-11-16T21:12:07.048 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6efcbbee:::obj-1-168:head# 2020-11-16T21:12:07.048 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6f20741c:::obj-1-33:head# 2020-11-16T21:12:07.048 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6fe4c885:::obj-1-273:head# 2020-11-16T21:12:07.048 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:6fffdea2:::obj-1-425:head# 2020-11-16T21:12:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:70716852:::obj-1-593:head# 2020-11-16T21:12:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:70919606:::obj-1-515:head# 2020-11-16T21:12:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:70c39d76:::obj-1-467:head# 2020-11-16T21:12:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:70f57ad3:::obj-1-361:head# 2020-11-16T21:12:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7177c7b5:::obj-1-503:head# 2020-11-16T21:12:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:719b69bd:::obj-1-497:head# 2020-11-16T21:12:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:724f9347:::obj-1-6:head# 2020-11-16T21:12:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:726d9f6b:::obj-1-111:head# 2020-11-16T21:12:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:72a91baf:::obj-1-439:head# 2020-11-16T21:12:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:72aa43ee:::obj-1-534:head# 2020-11-16T21:12:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:72c40e11:::obj-1-288:head# 2020-11-16T21:12:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:74196d45:::obj-1-246:head# 2020-11-16T21:12:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:743f42b1:::obj-1-326:head# 2020-11-16T21:12:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7523b538:::obj-1-494:head# 2020-11-16T21:12:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:75310ed6:::obj-1-364:head# 2020-11-16T21:12:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7568e0bc:::obj-1-585:head# 2020-11-16T21:12:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:760219a3:::obj-1-353:head# 2020-11-16T21:12:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:763338c9:::obj-1-555:head# 2020-11-16T21:12:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:77bc2d65:::obj-1-386:head# 2020-11-16T21:12:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:77ed0c7a:::obj-1-492:head# 2020-11-16T21:12:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:785a4052:::obj-1-456:head# 2020-11-16T21:12:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:796799c3:::obj-1-195:head# 2020-11-16T21:12:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:79caf0c1:::obj-1-445:head# 2020-11-16T21:12:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7af5b3e4:::obj-1-217:head# 2020-11-16T21:12:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7b3a9063:::obj-1-169:head# 2020-11-16T21:12:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7b509a92:::obj-1-561:head# 2020-11-16T21:12:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7bc646b1:::obj-1-415:head# 2020-11-16T21:12:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7c2f9188:::obj-1-482:head# 2020-11-16T21:12:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7d0da936:::obj-1-11:head# 2020-11-16T21:12:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7d25ca2d:::obj-1-242:head# 2020-11-16T21:12:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7d64f81e:::obj-1-473:head# 2020-11-16T21:12:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7ef2ea1b:::obj-1-417:head# 2020-11-16T21:12:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7f52b238:::obj-1-308:head# 2020-11-16T21:12:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:7f7a313c:::obj-1-551:head# 2020-11-16T21:12:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:806827a7:::obj-1-472:head# 2020-11-16T21:12:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8112c3ab:::obj-1-45:head# 2020-11-16T21:12:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:81365b55:::obj-1-315:head# 2020-11-16T21:12:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:818ffdc3:::obj-1-345:head# 2020-11-16T21:12:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:81babab8:::obj-1-535:head# 2020-11-16T21:12:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8288f652:::obj-1-347:head# 2020-11-16T21:12:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8312eadd:::obj-1-14:head# 2020-11-16T21:12:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:83b89bc7:::obj-1-137:head# 2020-11-16T21:12:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:84a12321:::obj-1-134:head# 2020-11-16T21:12:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:84b26d62:::obj-1-408:head# 2020-11-16T21:12:07.057 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:84bbb26e:::obj-1-43:head# 2020-11-16T21:12:07.057 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:86fc0014:::obj-1-222:head# 2020-11-16T21:12:07.057 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:87488bf4:::obj-1-174:head# 2020-11-16T21:12:07.057 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8754389a:::obj-1-65:head# 2020-11-16T21:12:07.057 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:87a6ad67:::obj-1-366:head# 2020-11-16T21:12:07.058 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:87d32b83:::obj-1-513:head# 2020-11-16T21:12:07.058 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8813a346:::obj-1-540:head# 2020-11-16T21:12:07.058 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:88146951:::obj-1-216:head# 2020-11-16T21:12:07.058 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8820800d:::obj-1-83:head# 2020-11-16T21:12:07.058 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:88319745:::obj-1-98:head# 2020-11-16T21:12:07.058 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:88c969b8:::obj-1-54:head# 2020-11-16T21:12:07.059 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:895a2d53:::obj-1-150:head# 2020-11-16T21:12:07.059 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8ae006a0:::obj-1-241:head# 2020-11-16T21:12:07.059 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8bd44e64:::obj-1-300:head# 2020-11-16T21:12:07.059 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8beb257f:::obj-1-3:head# 2020-11-16T21:12:07.059 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8cd8c743:::obj-1-527:head# 2020-11-16T21:12:07.060 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8daa99c7:::obj-1-237:head# 2020-11-16T21:12:07.060 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8f5bccbf:::obj-1-59:head# 2020-11-16T21:12:07.060 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8f6d9945:::obj-1-151:head# 2020-11-16T21:12:07.060 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8f7c568c:::obj-1-261:head# 2020-11-16T21:12:07.060 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:8fbc59a9:::obj-1-589:head# 2020-11-16T21:12:07.061 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:90225d94:::obj-1-194:head# 2020-11-16T21:12:07.061 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:904c50d3:::obj-1-63:head# 2020-11-16T21:12:07.061 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:90e1bd79:::obj-1-450:head# 2020-11-16T21:12:07.061 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9274b06d:::obj-1-460:head# 2020-11-16T21:12:07.062 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:92af41f5:::obj-1-409:head# 2020-11-16T21:12:07.062 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:935a08d6:::obj-1-192:head# 2020-11-16T21:12:07.062 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:935f8a13:::obj-1-24:head# 2020-11-16T21:12:07.062 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:95230148:::obj-1-280:head# 2020-11-16T21:12:07.062 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:95342eb0:::obj-1-390:head# 2020-11-16T21:12:07.063 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9659d09c:::obj-1-292:head# 2020-11-16T21:12:07.063 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:970e8cb4:::obj-1-18:head# 2020-11-16T21:12:07.063 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:97d21cf1:::obj-1-80:head# 2020-11-16T21:12:07.063 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:982bdb12:::obj-1-302:head# 2020-11-16T21:12:07.063 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:985dfd75:::obj-1-91:head# 2020-11-16T21:12:07.064 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:986d245a:::obj-1-154:head# 2020-11-16T21:12:07.064 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:98a03e99:::obj-1-529:head# 2020-11-16T21:12:07.064 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:98b7103e:::obj-1-275:head# 2020-11-16T21:12:07.064 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:990f0ecf:::obj-1-427:head# 2020-11-16T21:12:07.064 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:996c1f26:::obj-1-96:head# 2020-11-16T21:12:07.065 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:99bd2018:::obj-1-17:head# 2020-11-16T21:12:07.065 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9a7434c9:::obj-1-469:head# 2020-11-16T21:12:07.065 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9a8e36cc:::obj-1-598:head# 2020-11-16T21:12:07.065 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9abcc369:::obj-1-442:head# 2020-11-16T21:12:07.065 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9b0ea06e:::obj-1-231:head# 2020-11-16T21:12:07.065 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9b34e80d:::obj-1-378:head# 2020-11-16T21:12:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9b55a2a9:::obj-1-60:head# 2020-11-16T21:12:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9be3f2da:::obj-1-597:head# 2020-11-16T21:12:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9bf62c88:::obj-1-447:head# 2020-11-16T21:12:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9c31857b:::obj-1-272:head# 2020-11-16T21:12:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9c533e91:::obj-1-19:head# 2020-11-16T21:12:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9ca70688:::obj-1-570:head# 2020-11-16T21:12:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9ccce337:::obj-1-147:head# 2020-11-16T21:12:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9d4b2dfc:::obj-1-267:head# 2020-11-16T21:12:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9d4df969:::obj-1-286:head# 2020-11-16T21:12:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9d714651:::obj-1-495:head# 2020-11-16T21:12:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9da65518:::obj-1-498:head# 2020-11-16T21:12:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9e94f27b:::obj-1-209:head# 2020-11-16T21:12:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9f39e910:::obj-1-354:head# 2020-11-16T21:12:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9f8f9ef4:::obj-1-226:head# 2020-11-16T21:12:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:9fcd3f36:::obj-1-47:head# 2020-11-16T21:12:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a0326d7d:::obj-1-338:head# 2020-11-16T21:12:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a05e7475:::obj-1-62:head# 2020-11-16T21:12:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a0955052:::obj-1-533:head# 2020-11-16T21:12:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a0aa6848:::obj-1-38:head# 2020-11-16T21:12:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a0fdc285:::obj-1-133:head# 2020-11-16T21:12:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a10290b8:::obj-1-245:head# 2020-11-16T21:12:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a118a8a8:::obj-1-500:head# 2020-11-16T21:12:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a11a4079:::obj-1-199:head# 2020-11-16T21:12:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a1b3ec36:::obj-1-15:head# 2020-11-16T21:12:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a1f5079b:::obj-1-468:head# 2020-11-16T21:12:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a1ff04b5:::obj-1-167:head# 2020-11-16T21:12:07.076 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a21cc437:::obj-1-128:head# 2020-11-16T21:12:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a2966c77:::obj-1-140:head# 2020-11-16T21:12:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a321e537:::obj-1-184:head# 2020-11-16T21:12:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a33418d8:::obj-1-488:head# 2020-11-16T21:12:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a38b1c1a:::obj-1-584:head# 2020-11-16T21:12:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a3f83221:::obj-1-158:head# 2020-11-16T21:12:07.078 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a46e33eb:::obj-1-106:head# 2020-11-16T21:12:07.078 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a491460d:::obj-1-421:head# 2020-11-16T21:12:07.078 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a4dd813b:::obj-1-530:head# 2020-11-16T21:12:07.078 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a4e0d96d:::obj-1-351:head# 2020-11-16T21:12:07.078 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a52657b4:::obj-1-557:head# 2020-11-16T21:12:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a5b3bca6:::obj-1-277:head# 2020-11-16T21:12:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a5f5a9fb:::obj-1-170:head# 2020-11-16T21:12:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a6aeed57:::obj-1-141:head# 2020-11-16T21:12:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a6dda70f:::obj-1-471:head# 2020-11-16T21:12:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a71186c0:::obj-1-406:head# 2020-11-16T21:12:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a764ad8c:::obj-1-221:head# 2020-11-16T21:12:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a7b63473:::obj-1-131:head# 2020-11-16T21:12:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a88042fa:::obj-1-296:head# 2020-11-16T21:12:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:a8805fe0:::obj-1-213:head# 2020-11-16T21:12:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:aa4f12b1:::obj-1-270:head# 2020-11-16T21:12:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ab27f701:::obj-1-399:head# 2020-11-16T21:12:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ab8027c1:::obj-1-360:head# 2020-11-16T21:12:07.081 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ac3ef8ab:::obj-1-371:head# 2020-11-16T21:12:07.081 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ac4db76d:::obj-1-405:head# 2020-11-16T21:12:07.081 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ad53cb29:::obj-1-256:head# 2020-11-16T21:12:07.081 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ad704c16:::obj-1-485:head# 2020-11-16T21:12:07.081 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:adec4920:::obj-1-107:head# 2020-11-16T21:12:07.082 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ae3a94b3:::obj-1-95:head# 2020-11-16T21:12:07.082 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ae92616e:::obj-1-448:head# 2020-11-16T21:12:07.082 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ae975168:::obj-1-426:head# 2020-11-16T21:12:07.082 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:aee8fd62:::obj-1-27:head# 2020-11-16T21:12:07.082 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:af38aa87:::obj-1-50:head# 2020-11-16T21:12:07.083 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b0327eb3:::obj-1-380:head# 2020-11-16T21:12:07.083 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b03cd166:::obj-1-203:head# 2020-11-16T21:12:07.083 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b06f06e0:::obj-1-126:head# 2020-11-16T21:12:07.083 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b0972538:::obj-1-506:head# 2020-11-16T21:12:07.083 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b0cfb86a:::obj-1-359:head# 2020-11-16T21:12:07.084 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b10adb1c:::obj-1-333:head# 2020-11-16T21:12:07.084 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b20b9004:::obj-1-401:head# 2020-11-16T21:12:07.084 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b2113757:::obj-1-4:head# 2020-11-16T21:12:07.084 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b22441e9:::obj-1-29:head# 2020-11-16T21:12:07.084 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b2472384:::obj-1-207:head# 2020-11-16T21:12:07.084 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b2604227:::obj-1-502:head# 2020-11-16T21:12:07.085 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b3277f89:::obj-1-175:head# 2020-11-16T21:12:07.085 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b3d09e0d:::obj-1-258:head# 2020-11-16T21:12:07.085 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b3e9969a:::obj-1-499:head# 2020-11-16T21:12:07.085 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b4892e7d:::obj-1-155:head# 2020-11-16T21:12:07.085 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b4b76084:::obj-1-581:head# 2020-11-16T21:12:07.085 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b6e47ae5:::obj-1-215:head# 2020-11-16T21:12:07.086 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b75af5ff:::obj-1-228:head# 2020-11-16T21:12:07.086 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b8e2e20f:::obj-1-220:head# 2020-11-16T21:12:07.086 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:b8e80c4e:::obj-1-576:head# 2020-11-16T21:12:07.086 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ba502aee:::obj-1-255:head# 2020-11-16T21:12:07.086 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ba9c3e6e:::obj-1-57:head# 2020-11-16T21:12:07.086 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bb068062:::obj-1-461:head# 2020-11-16T21:12:07.087 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bb16c115:::obj-1-157:head# 2020-11-16T21:12:07.087 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bb447914:::obj-1-481:head# 2020-11-16T21:12:07.087 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bb6c8e0f:::obj-1-348:head# 2020-11-16T21:12:07.087 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bb77d6d0:::obj-1-301:head# 2020-11-16T21:12:07.087 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bc51c5f6:::obj-1-422:head# 2020-11-16T21:12:07.087 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bc838d5c:::obj-1-35:head# 2020-11-16T21:12:07.088 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bceae5eb:::obj-1-56:head# 2020-11-16T21:12:07.088 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bea17ef6:::obj-1-84:head# 2020-11-16T21:12:07.088 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:beb88be5:::obj-1-396:head# 2020-11-16T21:12:07.088 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:beed7b18:::obj-1-569:head# 2020-11-16T21:12:07.088 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bf0bff49:::obj-1-385:head# 2020-11-16T21:12:07.089 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bf545d5a:::obj-1-69:head# 2020-11-16T21:12:07.089 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bf560a7f:::obj-1-40:head# 2020-11-16T21:12:07.089 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:bff6999c:::obj-1-240:head# 2020-11-16T21:12:07.089 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c0a33d38:::obj-1-321:head# 2020-11-16T21:12:07.089 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c1e4c9b4:::obj-1-299:head# 2020-11-16T21:12:07.089 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c1ec515a:::obj-1-16:head# 2020-11-16T21:12:07.090 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c22de3c8:::obj-1-110:head# 2020-11-16T21:12:07.090 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c231cd7c:::obj-1-58:head# 2020-11-16T21:12:07.090 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c2ee6ace:::obj-1-186:head# 2020-11-16T21:12:07.090 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c32bfef2:::obj-1-181:head# 2020-11-16T21:12:07.090 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c575a298:::obj-1-466:head# 2020-11-16T21:12:07.090 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c58c3f80:::obj-1-493:head# 2020-11-16T21:12:07.091 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c6442cf2:::obj-1-455:head# 2020-11-16T21:12:07.091 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c6619413:::obj-1-37:head# 2020-11-16T21:12:07.091 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c6c2f8e5:::obj-1-519:head# 2020-11-16T21:12:07.091 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c6f811a2:::obj-1-180:head# 2020-11-16T21:12:07.091 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c72af975:::obj-1-21:head# 2020-11-16T21:12:07.091 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c73eb350:::obj-1-356:head# 2020-11-16T21:12:07.092 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c7a683da:::obj-1-410:head# 2020-11-16T21:12:07.092 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c7eb9bcc:::obj-1-553:head# 2020-11-16T21:12:07.092 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c886df99:::obj-1-269:head# 2020-11-16T21:12:07.092 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c932e3ec:::obj-1-135:head# 2020-11-16T21:12:07.092 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c9b6507a:::obj-1-520:head# 2020-11-16T21:12:07.092 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:c9e3b064:::obj-1-244:head# 2020-11-16T21:12:07.093 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:cb705d66:::obj-1-377:head# 2020-11-16T21:12:07.093 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:cc80e25c:::obj-1-115:head# 2020-11-16T21:12:07.093 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:cd1ec88e:::obj-1-297:head# 2020-11-16T21:12:07.093 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:cd7e6936:::obj-1-234:head# 2020-11-16T21:12:07.093 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ce32b69c:::obj-1-30:head# 2020-11-16T21:12:07.093 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:cf2b123c:::obj-1-293:head# 2020-11-16T21:12:07.094 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:cf2e80a9:::obj-1-542:head# 2020-11-16T21:12:07.094 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:cf51da6a:::obj-1-284:head# 2020-11-16T21:12:07.094 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:cf590f42:::obj-1-381:head# 2020-11-16T21:12:07.094 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:cf68a2b1:::obj-1-68:head# 2020-11-16T21:12:07.094 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:cf6aefb6:::obj-1-28:head# 2020-11-16T21:12:07.094 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:cfd8e6af:::obj-1-105:head# 2020-11-16T21:12:07.095 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d0640bad:::obj-1-254:head# 2020-11-16T21:12:07.095 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d101cda5:::obj-1-259:head# 2020-11-16T21:12:07.095 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d1be9386:::obj-1-416:head# 2020-11-16T21:12:07.095 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d243f036:::obj-1-265:head# 2020-11-16T21:12:07.095 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d2808c34:::obj-1-159:head# 2020-11-16T21:12:07.095 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d3415824:::obj-1-39:head# 2020-11-16T21:12:07.096 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d424f07a:::obj-1-547:head# 2020-11-16T21:12:07.096 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d44139f2:::obj-1-289:head# 2020-11-16T21:12:07.096 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d46015ab:::obj-1-546:head# 2020-11-16T21:12:07.096 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d4e503db:::obj-1-525:head# 2020-11-16T21:12:07.096 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d4e97d18:::obj-1-85:head# 2020-11-16T21:12:07.096 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d511492e:::obj-1-388:head# 2020-11-16T21:12:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d526a7fa:::obj-1-12:head# 2020-11-16T21:12:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d5667b59:::obj-1-369:head# 2020-11-16T21:12:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d57f2261:::obj-1-419:head# 2020-11-16T21:12:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d57f281b:::obj-1-250:head# 2020-11-16T21:12:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d6e05894:::obj-1-166:head# 2020-11-16T21:12:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d7689481:::obj-1-424:head# 2020-11-16T21:12:07.098 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d7d27743:::obj-1-129:head# 2020-11-16T21:12:07.098 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d7e4d3e9:::obj-1-127:head# 2020-11-16T21:12:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d7f1422a:::obj-1-594:head# 2020-11-16T21:12:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d8469fd2:::obj-1-153:head# 2020-11-16T21:12:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d84efd9e:::obj-1-435:head# 2020-11-16T21:12:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d883004b:::obj-1-162:head# 2020-11-16T21:12:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d8b60854:::obj-1-257:head# 2020-11-16T21:12:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d91aaae6:::obj-1-1:head# 2020-11-16T21:12:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d98695ca:::obj-1-524:head# 2020-11-16T21:12:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:d9ae97c8:::obj-1-177:head# 2020-11-16T21:12:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:da1713bd:::obj-1-309:head# 2020-11-16T21:12:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:da37824c:::obj-1-2:head# 2020-11-16T21:12:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:dab221b0:::obj-1-526:head# 2020-11-16T21:12:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:db27d885:::obj-1-363:head# 2020-11-16T21:12:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:dc001d76:::obj-1-329:head# 2020-11-16T21:12:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:dc4f4119:::obj-1-122:head# 2020-11-16T21:12:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:dcc36eb7:::obj-1-283:head# 2020-11-16T21:12:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:df7da06c:::obj-1-124:head# 2020-11-16T21:12:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:df7fcacb:::obj-1-330:head# 2020-11-16T21:12:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e05cbe31:::obj-1-413:head# 2020-11-16T21:12:11.213 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e1060a54:::obj-1-384:head# 2020-11-16T21:12:11.213 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e15b839b:::obj-1-316:head# 2020-11-16T21:12:11.213 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e161b6e5:::obj-1-103:head# 2020-11-16T21:12:11.213 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e1d1d79c:::obj-1-512:head# 2020-11-16T21:12:11.213 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e224b482:::obj-1-501:head# 2020-11-16T21:12:11.213 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e22d6f76:::obj-1-70:head# 2020-11-16T21:12:11.213 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e2bd801c:::obj-1-523:head# 2020-11-16T21:12:11.214 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e3d9f395:::obj-1-397:head# 2020-11-16T21:12:11.214 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e474ea25:::obj-1-457:head# 2020-11-16T21:12:11.214 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e49e1ff3:::obj-1-130:head# 2020-11-16T21:12:11.214 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e56c67a0:::obj-1-191:head# 2020-11-16T21:12:11.214 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e576e921:::obj-1-376:head# 2020-11-16T21:12:11.214 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e619fd26:::obj-1-395:head# 2020-11-16T21:12:11.214 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e625bdc5:::obj-1-522:head# 2020-11-16T21:12:11.214 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e68a0fa6:::obj-1-394:head# 2020-11-16T21:12:11.215 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e6cf260f:::obj-1-77:head# 2020-11-16T21:12:11.215 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e6d6255a:::obj-1-108:head# 2020-11-16T21:12:11.215 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e7a35d2f:::obj-1-116:head# 2020-11-16T21:12:11.215 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e7b07de5:::obj-1-238:head# 2020-11-16T21:12:11.215 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e8037aba:::obj-1-404:head# 2020-11-16T21:12:11.215 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e8e8b34f:::obj-1-291:head# 2020-11-16T21:12:11.215 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:e987cd7a:::obj-1-123:head# 2020-11-16T21:12:11.216 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ea534db1:::obj-1-311:head# 2020-11-16T21:12:11.216 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:eabcc188:::obj-1-325:head# 2020-11-16T21:12:11.216 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:eb086660:::obj-1-595:head# 2020-11-16T21:12:11.216 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:eb2235d3:::obj-1-596:head# 2020-11-16T21:12:11.216 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:eb35ae40:::obj-1-516:head# 2020-11-16T21:12:11.216 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:eb360a13:::obj-1-224:head# 2020-11-16T21:12:11.216 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:eb3a3ea1:::obj-1-120:head# 2020-11-16T21:12:11.216 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ebca9f6e:::obj-1-290:head# 2020-11-16T21:12:11.217 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ec81db71:::obj-1-478:head# 2020-11-16T21:12:11.217 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:edeafa4c:::obj-1-362:head# 2020-11-16T21:12:11.217 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:eeb5261b:::obj-1-539:head# 2020-11-16T21:12:11.217 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:eec00eee:::obj-1-567:head# 2020-11-16T21:12:11.217 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ef075e10:::obj-1-463:head# 2020-11-16T21:12:11.217 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ef8e37d9:::obj-1-94:head# 2020-11-16T21:12:11.217 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:efad7f67:::obj-1-197:head# 2020-11-16T21:12:11.217 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:efe09ea7:::obj-1-537:head# 2020-11-16T21:12:11.218 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f0445019:::obj-1-480:head# 2020-11-16T21:12:11.218 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f08a432c:::obj-1-489:head# 2020-11-16T21:12:11.218 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f0bd9006:::obj-1-487:head# 2020-11-16T21:12:11.218 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f11a115d:::obj-1-591:head# 2020-11-16T21:12:11.218 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f232e0af:::obj-1-146:head# 2020-11-16T21:12:11.218 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f258db54:::obj-1-521:head# 2020-11-16T21:12:11.218 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f29cc204:::obj-1-566:head# 2020-11-16T21:12:11.219 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f2ae2451:::obj-1-66:head# 2020-11-16T21:12:11.219 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f318810a:::obj-1-160:head# 2020-11-16T21:12:11.219 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f3c35a0d:::obj-1-67:head# 2020-11-16T21:12:11.219 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f3ce9c90:::obj-1-88:head# 2020-11-16T21:12:11.219 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f3e3e81f:::obj-1-304:head# 2020-11-16T21:12:11.220 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f3f65b0a:::obj-1-367:head# 2020-11-16T21:12:11.220 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f4ea014b:::obj-1-149:head# 2020-11-16T21:12:11.220 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f52d115b:::obj-1-383:head# 2020-11-16T21:12:11.220 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f590311e:::obj-1-76:head# 2020-11-16T21:12:11.220 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f5a37734:::obj-1-51:head# 2020-11-16T21:12:11.220 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f5e2e77c:::obj-1-179:head# 2020-11-16T21:12:11.220 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f5e381bc:::obj-1-173:head# 2020-11-16T21:12:11.220 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f6c0ae0b:::obj-1-247:head# 2020-11-16T21:12:11.221 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f6db8a82:::obj-1-548:head# 2020-11-16T21:12:11.221 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f6e2dc46:::obj-1-36:head# 2020-11-16T21:12:11.221 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f76ee569:::obj-1-587:head# 2020-11-16T21:12:11.221 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f773b9ed:::obj-1-550:head# 2020-11-16T21:12:11.221 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f7e1adad:::obj-1-350:head# 2020-11-16T21:12:11.221 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f7eebf7a:::obj-1-559:head# 2020-11-16T21:12:11.221 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f877f025:::obj-1-187:head# 2020-11-16T21:12:11.222 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f992f9e9:::obj-1-402:head# 2020-11-16T21:12:11.222 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:f9fe960a:::obj-1-418:head# 2020-11-16T21:12:11.222 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:fa9b882f:::obj-1-165:head# 2020-11-16T21:12:11.222 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:fbd6461c:::obj-1-339:head# 2020-11-16T21:12:11.222 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:fbeab28d:::obj-1-431:head# 2020-11-16T21:12:11.223 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:fdea4138:::obj-1-185:head# 2020-11-16T21:12:11.223 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:fdfa7c9b:::obj-1-183:head# 2020-11-16T21:12:11.223 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:fe1ba57e:::obj-1-90:head# 2020-11-16T21:12:11.223 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:feb7adea:::obj-1-560:head# 2020-11-16T21:12:11.223 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ffab306e:::obj-1-97:head# 2020-11-16T21:12:11.223 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ffdfd397:::obj-1-433:head# 2020-11-16T21:12:11.223 INFO:tasks.workunit.client.0.smithi189.stderr:Read #2:ffe8cf04:::obj-1-303:head# 2020-11-16T21:12:11.223 INFO:tasks.workunit.client.0.smithi189.stderr:Export successful 2020-11-16T21:12:11.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/osd-backfill-space 2 2020-11-16T21:12:11.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:11.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T21:12:11.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-11-16T21:12:11.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T21:12:11.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/2 2020-11-16T21:12:11.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:11.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:11.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:11.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:11.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2020-11-16T21:12:11.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2020-11-16T21:12:11.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:11.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T21:12:11.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:11.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T21:12:11.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:11.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:11.226 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:11.226 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:11.226 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:11.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:11.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:11.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:11.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:11.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:11.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:11.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:11.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:11.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T21:12:11.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T21:12:11.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/2 2020-11-16T21:12:11.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-11-16T21:12:11.228 INFO:tasks.workunit.client.0.smithi189.stderr:start osd.2 2020-11-16T21:12:11.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:11.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/2/whoami 2020-11-16T21:12:11.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-11-16T21:12:11.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T21:12:11.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T21:12:11.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T21:12:11.229 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:44:45.730+0000 7f3c5564dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:11.229 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:44:45.730+0000 7f3c5564dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:11.229 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:44:45.734+0000 7f3c5564dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:11.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-11-16T21:12:11.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:11.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:12:11.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:11.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:11.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:11.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:11.230 INFO:tasks.workunit.client.0.smithi189.stderr:0 2020-11-16T21:12:11.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:12:11.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:11.231 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:44:46.278+0000 7f3c5564dd80 -1 Falling back to public interface 2020-11-16T21:12:11.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:11.231 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:44:47.314+0000 7f3c5564dd80 -1 osd.2 40 log_to_monitors {default=true} 2020-11-16T21:12:11.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:11.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:11.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:11.232 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:12:11.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:11.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:12:11.232 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 up in weight 1 up_from 43 up_thru 43 down_at 41 last_clean_interval [15,40) [v2:127.0.0.1:6818/199659,v1:127.0.0.1:6819/199659] [v2:127.0.0.1:6820/199659,v1:127.0.0.1:6821/199659] exists,up 30e48a20-7415-4187-bdb6-fc1f9fb3e8fb 2020-11-16T21:12:11.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:11.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:11.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:11.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:490: TEST_backfill_multi_partial: kill_daemon td/osd-backfill-space/osd.1.pid TERM 2020-11-16T21:12:11.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/osd-backfill-space/osd.1.pid 2020-11-16T21:12:11.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=186933 2020-11-16T21:12:11.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2020-11-16T21:12:11.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2020-11-16T21:12:11.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2020-11-16T21:12:11.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2020-11-16T21:12:11.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 186933 2020-11-16T21:12:11.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2020-11-16T21:12:11.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2020-11-16T21:12:11.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2020-11-16T21:12:11.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2020-11-16T21:12:11.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 186933 2020-11-16T21:12:11.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2020-11-16T21:12:11.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2020-11-16T21:12:11.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2020-11-16T21:12:11.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:491: TEST_backfill_multi_partial: _objectstore_tool_nodown td/osd-backfill-space 1 --force --op remove --pgid 2.0 2020-11-16T21:12:11.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-space 2020-11-16T21:12:11.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T21:12:11.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-11-16T21:12:11.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T21:12:11.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-space/1 2020-11-16T21:12:11.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-space/1 --force --op remove --pgid 2.0 2020-11-16T21:12:11.236 INFO:tasks.workunit.client.0.smithi189.stderr:PG '2.0' not found 2020-11-16T21:12:11.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: return 1 2020-11-16T21:12:11.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:492: TEST_backfill_multi_partial: _objectstore_tool_nodown td/osd-backfill-space 1 --op import --pgid 2.0 --file td/osd-backfill-space/export.out 2020-11-16T21:12:11.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-space 2020-11-16T21:12:11.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T21:12:11.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-11-16T21:12:11.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T21:12:11.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-space/1 2020-11-16T21:12:11.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-space/1 --op import --pgid 2.0 --file td/osd-backfill-space/export.out 2020-11-16T21:12:11.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:493: TEST_backfill_multi_partial: _objectstore_tool_nodown td/osd-backfill-space 1 --op import --pgid 1.0 --file td/osd-backfill-space/fillexport.out 2020-11-16T21:12:11.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-space 2020-11-16T21:12:11.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-11-16T21:12:11.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-11-16T21:12:11.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-11-16T21:12:11.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-space/1 2020-11-16T21:12:11.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-space/1 --op import --pgid 1.0 --file td/osd-backfill-space/fillexport.out 2020-11-16T21:12:11.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:494: TEST_backfill_multi_partial: ceph pg dump pgs 2020-11-16T21:12:11.239 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:12:11.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:495: TEST_backfill_multi_partial: sleep 20 2020-11-16T21:12:11.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:496: TEST_backfill_multi_partial: ceph pg dump pgs 2020-11-16T21:12:11.239 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:12:11.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:499: TEST_backfill_multi_partial: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=2611 count=1 2020-11-16T21:12:11.240 INFO:tasks.workunit.client.0.smithi189.stderr:1+0 records in 2020-11-16T21:12:11.240 INFO:tasks.workunit.client.0.smithi189.stderr:1+0 records out 2020-11-16T21:12:11.240 INFO:tasks.workunit.client.0.smithi189.stderr:2611 bytes (2.6 kB, 2.5 KiB) copied, 0.000268315 s, 9.7 MB/s 2020-11-16T21:12:11.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: seq 1 600 2020-11-16T21:12:11.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-1 td/osd-backfill-space/datafile 2020-11-16T21:12:11.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-1 td/osd-backfill-space/datafile 2020-11-16T21:12:11.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-2 td/osd-backfill-space/datafile 2020-11-16T21:12:11.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-2 td/osd-backfill-space/datafile 2020-11-16T21:12:11.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-3 td/osd-backfill-space/datafile 2020-11-16T21:12:11.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-3 td/osd-backfill-space/datafile 2020-11-16T21:12:11.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-4 td/osd-backfill-space/datafile 2020-11-16T21:12:11.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-4 td/osd-backfill-space/datafile 2020-11-16T21:12:11.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-5 td/osd-backfill-space/datafile 2020-11-16T21:12:11.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-5 td/osd-backfill-space/datafile 2020-11-16T21:12:11.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-6 td/osd-backfill-space/datafile 2020-11-16T21:12:11.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-6 td/osd-backfill-space/datafile 2020-11-16T21:12:11.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-7 td/osd-backfill-space/datafile 2020-11-16T21:12:11.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-7 td/osd-backfill-space/datafile 2020-11-16T21:12:11.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-8 td/osd-backfill-space/datafile 2020-11-16T21:12:11.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-8 td/osd-backfill-space/datafile 2020-11-16T21:12:11.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-9 td/osd-backfill-space/datafile 2020-11-16T21:12:11.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-9 td/osd-backfill-space/datafile 2020-11-16T21:12:11.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-10 td/osd-backfill-space/datafile 2020-11-16T21:12:11.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-10 td/osd-backfill-space/datafile 2020-11-16T21:12:11.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-11 td/osd-backfill-space/datafile 2020-11-16T21:12:11.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-11 td/osd-backfill-space/datafile 2020-11-16T21:12:11.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-12 td/osd-backfill-space/datafile 2020-11-16T21:12:11.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-12 td/osd-backfill-space/datafile 2020-11-16T21:12:11.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-13 td/osd-backfill-space/datafile 2020-11-16T21:12:11.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-13 td/osd-backfill-space/datafile 2020-11-16T21:12:11.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-14 td/osd-backfill-space/datafile 2020-11-16T21:12:11.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-14 td/osd-backfill-space/datafile 2020-11-16T21:12:11.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-15 td/osd-backfill-space/datafile 2020-11-16T21:12:11.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-15 td/osd-backfill-space/datafile 2020-11-16T21:12:11.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-16 td/osd-backfill-space/datafile 2020-11-16T21:12:11.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-16 td/osd-backfill-space/datafile 2020-11-16T21:12:11.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-17 td/osd-backfill-space/datafile 2020-11-16T21:12:11.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-17 td/osd-backfill-space/datafile 2020-11-16T21:12:11.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-18 td/osd-backfill-space/datafile 2020-11-16T21:12:11.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-18 td/osd-backfill-space/datafile 2020-11-16T21:12:11.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-19 td/osd-backfill-space/datafile 2020-11-16T21:12:11.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-19 td/osd-backfill-space/datafile 2020-11-16T21:12:11.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-20 td/osd-backfill-space/datafile 2020-11-16T21:12:11.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-20 td/osd-backfill-space/datafile 2020-11-16T21:12:11.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-21 td/osd-backfill-space/datafile 2020-11-16T21:12:11.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-21 td/osd-backfill-space/datafile 2020-11-16T21:12:11.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-22 td/osd-backfill-space/datafile 2020-11-16T21:12:11.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-22 td/osd-backfill-space/datafile 2020-11-16T21:12:11.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-23 td/osd-backfill-space/datafile 2020-11-16T21:12:11.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-23 td/osd-backfill-space/datafile 2020-11-16T21:12:11.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-24 td/osd-backfill-space/datafile 2020-11-16T21:12:11.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-24 td/osd-backfill-space/datafile 2020-11-16T21:12:11.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-25 td/osd-backfill-space/datafile 2020-11-16T21:12:11.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-25 td/osd-backfill-space/datafile 2020-11-16T21:12:11.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-26 td/osd-backfill-space/datafile 2020-11-16T21:12:11.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-26 td/osd-backfill-space/datafile 2020-11-16T21:12:11.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-27 td/osd-backfill-space/datafile 2020-11-16T21:12:11.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-27 td/osd-backfill-space/datafile 2020-11-16T21:12:11.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.263 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-28 td/osd-backfill-space/datafile 2020-11-16T21:12:11.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-28 td/osd-backfill-space/datafile 2020-11-16T21:12:11.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-29 td/osd-backfill-space/datafile 2020-11-16T21:12:11.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-29 td/osd-backfill-space/datafile 2020-11-16T21:12:11.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-30 td/osd-backfill-space/datafile 2020-11-16T21:12:11.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-30 td/osd-backfill-space/datafile 2020-11-16T21:12:11.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-31 td/osd-backfill-space/datafile 2020-11-16T21:12:11.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-31 td/osd-backfill-space/datafile 2020-11-16T21:12:11.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-32 td/osd-backfill-space/datafile 2020-11-16T21:12:11.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-32 td/osd-backfill-space/datafile 2020-11-16T21:12:11.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-33 td/osd-backfill-space/datafile 2020-11-16T21:12:11.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-33 td/osd-backfill-space/datafile 2020-11-16T21:12:11.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-34 td/osd-backfill-space/datafile 2020-11-16T21:12:11.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-34 td/osd-backfill-space/datafile 2020-11-16T21:12:11.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-35 td/osd-backfill-space/datafile 2020-11-16T21:12:11.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-35 td/osd-backfill-space/datafile 2020-11-16T21:12:11.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-36 td/osd-backfill-space/datafile 2020-11-16T21:12:11.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-36 td/osd-backfill-space/datafile 2020-11-16T21:12:11.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-37 td/osd-backfill-space/datafile 2020-11-16T21:12:11.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-37 td/osd-backfill-space/datafile 2020-11-16T21:12:11.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-38 td/osd-backfill-space/datafile 2020-11-16T21:12:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-38 td/osd-backfill-space/datafile 2020-11-16T21:12:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-39 td/osd-backfill-space/datafile 2020-11-16T21:12:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-39 td/osd-backfill-space/datafile 2020-11-16T21:12:11.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-40 td/osd-backfill-space/datafile 2020-11-16T21:12:11.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-40 td/osd-backfill-space/datafile 2020-11-16T21:12:11.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-41 td/osd-backfill-space/datafile 2020-11-16T21:12:11.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-41 td/osd-backfill-space/datafile 2020-11-16T21:12:11.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-42 td/osd-backfill-space/datafile 2020-11-16T21:12:11.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-42 td/osd-backfill-space/datafile 2020-11-16T21:12:11.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-43 td/osd-backfill-space/datafile 2020-11-16T21:12:11.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-43 td/osd-backfill-space/datafile 2020-11-16T21:12:11.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-44 td/osd-backfill-space/datafile 2020-11-16T21:12:11.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-44 td/osd-backfill-space/datafile 2020-11-16T21:12:11.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-45 td/osd-backfill-space/datafile 2020-11-16T21:12:11.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-45 td/osd-backfill-space/datafile 2020-11-16T21:12:11.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-46 td/osd-backfill-space/datafile 2020-11-16T21:12:11.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-46 td/osd-backfill-space/datafile 2020-11-16T21:12:11.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-47 td/osd-backfill-space/datafile 2020-11-16T21:12:11.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-47 td/osd-backfill-space/datafile 2020-11-16T21:12:11.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-48 td/osd-backfill-space/datafile 2020-11-16T21:12:11.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-48 td/osd-backfill-space/datafile 2020-11-16T21:12:11.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-49 td/osd-backfill-space/datafile 2020-11-16T21:12:11.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-49 td/osd-backfill-space/datafile 2020-11-16T21:12:11.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-50 td/osd-backfill-space/datafile 2020-11-16T21:12:11.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-50 td/osd-backfill-space/datafile 2020-11-16T21:12:11.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-51 td/osd-backfill-space/datafile 2020-11-16T21:12:11.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-51 td/osd-backfill-space/datafile 2020-11-16T21:12:11.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-52 td/osd-backfill-space/datafile 2020-11-16T21:12:11.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-52 td/osd-backfill-space/datafile 2020-11-16T21:12:11.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-53 td/osd-backfill-space/datafile 2020-11-16T21:12:11.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-53 td/osd-backfill-space/datafile 2020-11-16T21:12:11.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-54 td/osd-backfill-space/datafile 2020-11-16T21:12:11.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-54 td/osd-backfill-space/datafile 2020-11-16T21:12:11.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-55 td/osd-backfill-space/datafile 2020-11-16T21:12:11.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-55 td/osd-backfill-space/datafile 2020-11-16T21:12:11.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-56 td/osd-backfill-space/datafile 2020-11-16T21:12:11.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-56 td/osd-backfill-space/datafile 2020-11-16T21:12:11.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-57 td/osd-backfill-space/datafile 2020-11-16T21:12:11.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-57 td/osd-backfill-space/datafile 2020-11-16T21:12:11.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-58 td/osd-backfill-space/datafile 2020-11-16T21:12:11.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-58 td/osd-backfill-space/datafile 2020-11-16T21:12:11.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-59 td/osd-backfill-space/datafile 2020-11-16T21:12:11.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-59 td/osd-backfill-space/datafile 2020-11-16T21:12:11.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-60 td/osd-backfill-space/datafile 2020-11-16T21:12:11.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-60 td/osd-backfill-space/datafile 2020-11-16T21:12:11.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.290 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-61 td/osd-backfill-space/datafile 2020-11-16T21:12:11.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-61 td/osd-backfill-space/datafile 2020-11-16T21:12:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-62 td/osd-backfill-space/datafile 2020-11-16T21:12:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-62 td/osd-backfill-space/datafile 2020-11-16T21:12:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-63 td/osd-backfill-space/datafile 2020-11-16T21:12:11.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-63 td/osd-backfill-space/datafile 2020-11-16T21:12:11.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-64 td/osd-backfill-space/datafile 2020-11-16T21:12:11.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-64 td/osd-backfill-space/datafile 2020-11-16T21:12:11.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-65 td/osd-backfill-space/datafile 2020-11-16T21:12:11.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-65 td/osd-backfill-space/datafile 2020-11-16T21:12:11.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-66 td/osd-backfill-space/datafile 2020-11-16T21:12:11.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-66 td/osd-backfill-space/datafile 2020-11-16T21:12:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-67 td/osd-backfill-space/datafile 2020-11-16T21:12:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-67 td/osd-backfill-space/datafile 2020-11-16T21:12:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-68 td/osd-backfill-space/datafile 2020-11-16T21:12:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-68 td/osd-backfill-space/datafile 2020-11-16T21:12:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-69 td/osd-backfill-space/datafile 2020-11-16T21:12:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-69 td/osd-backfill-space/datafile 2020-11-16T21:12:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-70 td/osd-backfill-space/datafile 2020-11-16T21:12:11.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-70 td/osd-backfill-space/datafile 2020-11-16T21:12:11.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-71 td/osd-backfill-space/datafile 2020-11-16T21:12:11.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-71 td/osd-backfill-space/datafile 2020-11-16T21:12:11.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-72 td/osd-backfill-space/datafile 2020-11-16T21:12:11.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-72 td/osd-backfill-space/datafile 2020-11-16T21:12:11.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-73 td/osd-backfill-space/datafile 2020-11-16T21:12:11.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-73 td/osd-backfill-space/datafile 2020-11-16T21:12:11.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-74 td/osd-backfill-space/datafile 2020-11-16T21:12:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-74 td/osd-backfill-space/datafile 2020-11-16T21:12:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.301 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-75 td/osd-backfill-space/datafile 2020-11-16T21:12:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-75 td/osd-backfill-space/datafile 2020-11-16T21:12:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-76 td/osd-backfill-space/datafile 2020-11-16T21:12:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-76 td/osd-backfill-space/datafile 2020-11-16T21:12:11.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-77 td/osd-backfill-space/datafile 2020-11-16T21:12:11.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-77 td/osd-backfill-space/datafile 2020-11-16T21:12:11.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-78 td/osd-backfill-space/datafile 2020-11-16T21:12:11.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-78 td/osd-backfill-space/datafile 2020-11-16T21:12:11.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-79 td/osd-backfill-space/datafile 2020-11-16T21:12:11.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-79 td/osd-backfill-space/datafile 2020-11-16T21:12:11.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-80 td/osd-backfill-space/datafile 2020-11-16T21:12:11.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-80 td/osd-backfill-space/datafile 2020-11-16T21:12:11.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-81 td/osd-backfill-space/datafile 2020-11-16T21:12:11.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-81 td/osd-backfill-space/datafile 2020-11-16T21:12:11.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.307 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-82 td/osd-backfill-space/datafile 2020-11-16T21:12:11.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-82 td/osd-backfill-space/datafile 2020-11-16T21:12:11.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.308 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-83 td/osd-backfill-space/datafile 2020-11-16T21:12:11.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-83 td/osd-backfill-space/datafile 2020-11-16T21:12:11.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.308 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-84 td/osd-backfill-space/datafile 2020-11-16T21:12:11.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-84 td/osd-backfill-space/datafile 2020-11-16T21:12:11.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-85 td/osd-backfill-space/datafile 2020-11-16T21:12:11.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-85 td/osd-backfill-space/datafile 2020-11-16T21:12:11.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-86 td/osd-backfill-space/datafile 2020-11-16T21:12:11.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-86 td/osd-backfill-space/datafile 2020-11-16T21:12:11.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-87 td/osd-backfill-space/datafile 2020-11-16T21:12:11.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-87 td/osd-backfill-space/datafile 2020-11-16T21:12:11.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-88 td/osd-backfill-space/datafile 2020-11-16T21:12:11.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-88 td/osd-backfill-space/datafile 2020-11-16T21:12:11.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-89 td/osd-backfill-space/datafile 2020-11-16T21:12:11.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-89 td/osd-backfill-space/datafile 2020-11-16T21:12:11.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-90 td/osd-backfill-space/datafile 2020-11-16T21:12:11.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-90 td/osd-backfill-space/datafile 2020-11-16T21:12:11.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-91 td/osd-backfill-space/datafile 2020-11-16T21:12:11.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-91 td/osd-backfill-space/datafile 2020-11-16T21:12:11.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-92 td/osd-backfill-space/datafile 2020-11-16T21:12:11.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-92 td/osd-backfill-space/datafile 2020-11-16T21:12:11.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-93 td/osd-backfill-space/datafile 2020-11-16T21:12:11.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-93 td/osd-backfill-space/datafile 2020-11-16T21:12:11.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.317 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-94 td/osd-backfill-space/datafile 2020-11-16T21:12:11.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-94 td/osd-backfill-space/datafile 2020-11-16T21:12:11.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-95 td/osd-backfill-space/datafile 2020-11-16T21:12:11.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-95 td/osd-backfill-space/datafile 2020-11-16T21:12:11.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-96 td/osd-backfill-space/datafile 2020-11-16T21:12:11.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-96 td/osd-backfill-space/datafile 2020-11-16T21:12:11.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-97 td/osd-backfill-space/datafile 2020-11-16T21:12:11.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-97 td/osd-backfill-space/datafile 2020-11-16T21:12:11.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-98 td/osd-backfill-space/datafile 2020-11-16T21:12:11.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-98 td/osd-backfill-space/datafile 2020-11-16T21:12:11.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-99 td/osd-backfill-space/datafile 2020-11-16T21:12:11.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-99 td/osd-backfill-space/datafile 2020-11-16T21:12:11.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-100 td/osd-backfill-space/datafile 2020-11-16T21:12:11.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-100 td/osd-backfill-space/datafile 2020-11-16T21:12:11.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-101 td/osd-backfill-space/datafile 2020-11-16T21:12:11.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-101 td/osd-backfill-space/datafile 2020-11-16T21:12:11.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-102 td/osd-backfill-space/datafile 2020-11-16T21:12:11.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-102 td/osd-backfill-space/datafile 2020-11-16T21:12:11.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-103 td/osd-backfill-space/datafile 2020-11-16T21:12:11.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-103 td/osd-backfill-space/datafile 2020-11-16T21:12:11.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-104 td/osd-backfill-space/datafile 2020-11-16T21:12:11.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-104 td/osd-backfill-space/datafile 2020-11-16T21:12:11.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-105 td/osd-backfill-space/datafile 2020-11-16T21:12:11.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-105 td/osd-backfill-space/datafile 2020-11-16T21:12:11.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-106 td/osd-backfill-space/datafile 2020-11-16T21:12:11.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-106 td/osd-backfill-space/datafile 2020-11-16T21:12:11.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-107 td/osd-backfill-space/datafile 2020-11-16T21:12:11.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-107 td/osd-backfill-space/datafile 2020-11-16T21:12:11.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-108 td/osd-backfill-space/datafile 2020-11-16T21:12:11.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-108 td/osd-backfill-space/datafile 2020-11-16T21:12:11.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-109 td/osd-backfill-space/datafile 2020-11-16T21:12:11.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-109 td/osd-backfill-space/datafile 2020-11-16T21:12:11.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-110 td/osd-backfill-space/datafile 2020-11-16T21:12:11.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-110 td/osd-backfill-space/datafile 2020-11-16T21:12:11.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-111 td/osd-backfill-space/datafile 2020-11-16T21:12:11.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-111 td/osd-backfill-space/datafile 2020-11-16T21:12:11.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-112 td/osd-backfill-space/datafile 2020-11-16T21:12:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-112 td/osd-backfill-space/datafile 2020-11-16T21:12:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-113 td/osd-backfill-space/datafile 2020-11-16T21:12:11.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-113 td/osd-backfill-space/datafile 2020-11-16T21:12:11.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-114 td/osd-backfill-space/datafile 2020-11-16T21:12:11.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-114 td/osd-backfill-space/datafile 2020-11-16T21:12:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-115 td/osd-backfill-space/datafile 2020-11-16T21:12:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-115 td/osd-backfill-space/datafile 2020-11-16T21:12:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-116 td/osd-backfill-space/datafile 2020-11-16T21:12:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-116 td/osd-backfill-space/datafile 2020-11-16T21:12:11.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-117 td/osd-backfill-space/datafile 2020-11-16T21:12:11.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-117 td/osd-backfill-space/datafile 2020-11-16T21:12:11.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-118 td/osd-backfill-space/datafile 2020-11-16T21:12:11.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-118 td/osd-backfill-space/datafile 2020-11-16T21:12:11.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-119 td/osd-backfill-space/datafile 2020-11-16T21:12:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-119 td/osd-backfill-space/datafile 2020-11-16T21:12:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-120 td/osd-backfill-space/datafile 2020-11-16T21:12:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-120 td/osd-backfill-space/datafile 2020-11-16T21:12:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-121 td/osd-backfill-space/datafile 2020-11-16T21:12:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-121 td/osd-backfill-space/datafile 2020-11-16T21:12:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-122 td/osd-backfill-space/datafile 2020-11-16T21:12:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-122 td/osd-backfill-space/datafile 2020-11-16T21:12:11.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-123 td/osd-backfill-space/datafile 2020-11-16T21:12:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-123 td/osd-backfill-space/datafile 2020-11-16T21:12:11.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-124 td/osd-backfill-space/datafile 2020-11-16T21:12:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-124 td/osd-backfill-space/datafile 2020-11-16T21:12:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-125 td/osd-backfill-space/datafile 2020-11-16T21:12:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-125 td/osd-backfill-space/datafile 2020-11-16T21:12:11.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-126 td/osd-backfill-space/datafile 2020-11-16T21:12:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-126 td/osd-backfill-space/datafile 2020-11-16T21:12:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-127 td/osd-backfill-space/datafile 2020-11-16T21:12:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-127 td/osd-backfill-space/datafile 2020-11-16T21:12:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-128 td/osd-backfill-space/datafile 2020-11-16T21:12:11.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-128 td/osd-backfill-space/datafile 2020-11-16T21:12:11.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-129 td/osd-backfill-space/datafile 2020-11-16T21:12:11.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-129 td/osd-backfill-space/datafile 2020-11-16T21:12:11.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-130 td/osd-backfill-space/datafile 2020-11-16T21:12:11.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-130 td/osd-backfill-space/datafile 2020-11-16T21:12:11.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-131 td/osd-backfill-space/datafile 2020-11-16T21:12:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-131 td/osd-backfill-space/datafile 2020-11-16T21:12:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-132 td/osd-backfill-space/datafile 2020-11-16T21:12:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-132 td/osd-backfill-space/datafile 2020-11-16T21:12:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-133 td/osd-backfill-space/datafile 2020-11-16T21:12:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-133 td/osd-backfill-space/datafile 2020-11-16T21:12:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-134 td/osd-backfill-space/datafile 2020-11-16T21:12:11.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-134 td/osd-backfill-space/datafile 2020-11-16T21:12:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-135 td/osd-backfill-space/datafile 2020-11-16T21:12:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-135 td/osd-backfill-space/datafile 2020-11-16T21:12:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-136 td/osd-backfill-space/datafile 2020-11-16T21:12:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-136 td/osd-backfill-space/datafile 2020-11-16T21:12:11.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-137 td/osd-backfill-space/datafile 2020-11-16T21:12:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-137 td/osd-backfill-space/datafile 2020-11-16T21:12:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-138 td/osd-backfill-space/datafile 2020-11-16T21:12:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-138 td/osd-backfill-space/datafile 2020-11-16T21:12:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-139 td/osd-backfill-space/datafile 2020-11-16T21:12:11.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-139 td/osd-backfill-space/datafile 2020-11-16T21:12:11.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-140 td/osd-backfill-space/datafile 2020-11-16T21:12:11.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-140 td/osd-backfill-space/datafile 2020-11-16T21:12:11.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-141 td/osd-backfill-space/datafile 2020-11-16T21:12:11.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-141 td/osd-backfill-space/datafile 2020-11-16T21:12:11.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-142 td/osd-backfill-space/datafile 2020-11-16T21:12:11.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-142 td/osd-backfill-space/datafile 2020-11-16T21:12:11.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-143 td/osd-backfill-space/datafile 2020-11-16T21:12:11.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-143 td/osd-backfill-space/datafile 2020-11-16T21:12:11.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-144 td/osd-backfill-space/datafile 2020-11-16T21:12:11.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-144 td/osd-backfill-space/datafile 2020-11-16T21:12:11.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-145 td/osd-backfill-space/datafile 2020-11-16T21:12:11.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-145 td/osd-backfill-space/datafile 2020-11-16T21:12:11.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-146 td/osd-backfill-space/datafile 2020-11-16T21:12:11.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-146 td/osd-backfill-space/datafile 2020-11-16T21:12:11.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.361 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-147 td/osd-backfill-space/datafile 2020-11-16T21:12:11.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-147 td/osd-backfill-space/datafile 2020-11-16T21:12:11.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-148 td/osd-backfill-space/datafile 2020-11-16T21:12:11.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-148 td/osd-backfill-space/datafile 2020-11-16T21:12:11.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-149 td/osd-backfill-space/datafile 2020-11-16T21:12:11.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-149 td/osd-backfill-space/datafile 2020-11-16T21:12:11.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-150 td/osd-backfill-space/datafile 2020-11-16T21:12:11.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-150 td/osd-backfill-space/datafile 2020-11-16T21:12:11.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-151 td/osd-backfill-space/datafile 2020-11-16T21:12:11.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-151 td/osd-backfill-space/datafile 2020-11-16T21:12:11.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-152 td/osd-backfill-space/datafile 2020-11-16T21:12:11.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-152 td/osd-backfill-space/datafile 2020-11-16T21:12:11.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-153 td/osd-backfill-space/datafile 2020-11-16T21:12:11.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-153 td/osd-backfill-space/datafile 2020-11-16T21:12:11.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-154 td/osd-backfill-space/datafile 2020-11-16T21:12:11.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-154 td/osd-backfill-space/datafile 2020-11-16T21:12:11.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-155 td/osd-backfill-space/datafile 2020-11-16T21:12:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-155 td/osd-backfill-space/datafile 2020-11-16T21:12:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-156 td/osd-backfill-space/datafile 2020-11-16T21:12:11.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-156 td/osd-backfill-space/datafile 2020-11-16T21:12:11.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-157 td/osd-backfill-space/datafile 2020-11-16T21:12:11.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-157 td/osd-backfill-space/datafile 2020-11-16T21:12:11.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-158 td/osd-backfill-space/datafile 2020-11-16T21:12:11.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-158 td/osd-backfill-space/datafile 2020-11-16T21:12:11.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-159 td/osd-backfill-space/datafile 2020-11-16T21:12:11.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-159 td/osd-backfill-space/datafile 2020-11-16T21:12:11.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-160 td/osd-backfill-space/datafile 2020-11-16T21:12:11.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-160 td/osd-backfill-space/datafile 2020-11-16T21:12:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-161 td/osd-backfill-space/datafile 2020-11-16T21:12:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-161 td/osd-backfill-space/datafile 2020-11-16T21:12:11.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-162 td/osd-backfill-space/datafile 2020-11-16T21:12:11.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-162 td/osd-backfill-space/datafile 2020-11-16T21:12:11.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-163 td/osd-backfill-space/datafile 2020-11-16T21:12:11.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-163 td/osd-backfill-space/datafile 2020-11-16T21:12:11.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-164 td/osd-backfill-space/datafile 2020-11-16T21:12:11.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-164 td/osd-backfill-space/datafile 2020-11-16T21:12:11.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-165 td/osd-backfill-space/datafile 2020-11-16T21:12:11.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-165 td/osd-backfill-space/datafile 2020-11-16T21:12:11.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-166 td/osd-backfill-space/datafile 2020-11-16T21:12:11.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-166 td/osd-backfill-space/datafile 2020-11-16T21:12:11.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-167 td/osd-backfill-space/datafile 2020-11-16T21:12:11.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-167 td/osd-backfill-space/datafile 2020-11-16T21:12:11.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-168 td/osd-backfill-space/datafile 2020-11-16T21:12:11.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-168 td/osd-backfill-space/datafile 2020-11-16T21:12:11.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-169 td/osd-backfill-space/datafile 2020-11-16T21:12:11.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-169 td/osd-backfill-space/datafile 2020-11-16T21:12:11.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-170 td/osd-backfill-space/datafile 2020-11-16T21:12:11.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-170 td/osd-backfill-space/datafile 2020-11-16T21:12:11.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-171 td/osd-backfill-space/datafile 2020-11-16T21:12:11.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-171 td/osd-backfill-space/datafile 2020-11-16T21:12:11.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-172 td/osd-backfill-space/datafile 2020-11-16T21:12:11.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-172 td/osd-backfill-space/datafile 2020-11-16T21:12:11.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-173 td/osd-backfill-space/datafile 2020-11-16T21:12:11.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-173 td/osd-backfill-space/datafile 2020-11-16T21:12:11.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-174 td/osd-backfill-space/datafile 2020-11-16T21:12:11.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-174 td/osd-backfill-space/datafile 2020-11-16T21:12:11.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-175 td/osd-backfill-space/datafile 2020-11-16T21:12:11.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-175 td/osd-backfill-space/datafile 2020-11-16T21:12:11.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-176 td/osd-backfill-space/datafile 2020-11-16T21:12:11.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-176 td/osd-backfill-space/datafile 2020-11-16T21:12:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-177 td/osd-backfill-space/datafile 2020-11-16T21:12:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-177 td/osd-backfill-space/datafile 2020-11-16T21:12:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-178 td/osd-backfill-space/datafile 2020-11-16T21:12:11.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-178 td/osd-backfill-space/datafile 2020-11-16T21:12:11.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-179 td/osd-backfill-space/datafile 2020-11-16T21:12:11.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-179 td/osd-backfill-space/datafile 2020-11-16T21:12:11.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-180 td/osd-backfill-space/datafile 2020-11-16T21:12:11.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-180 td/osd-backfill-space/datafile 2020-11-16T21:12:11.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-181 td/osd-backfill-space/datafile 2020-11-16T21:12:11.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-181 td/osd-backfill-space/datafile 2020-11-16T21:12:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-182 td/osd-backfill-space/datafile 2020-11-16T21:12:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-182 td/osd-backfill-space/datafile 2020-11-16T21:12:11.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-183 td/osd-backfill-space/datafile 2020-11-16T21:12:11.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-183 td/osd-backfill-space/datafile 2020-11-16T21:12:11.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-184 td/osd-backfill-space/datafile 2020-11-16T21:12:11.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-184 td/osd-backfill-space/datafile 2020-11-16T21:12:11.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-185 td/osd-backfill-space/datafile 2020-11-16T21:12:11.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-185 td/osd-backfill-space/datafile 2020-11-16T21:12:11.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-186 td/osd-backfill-space/datafile 2020-11-16T21:12:11.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-186 td/osd-backfill-space/datafile 2020-11-16T21:12:11.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-187 td/osd-backfill-space/datafile 2020-11-16T21:12:11.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-187 td/osd-backfill-space/datafile 2020-11-16T21:12:11.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-188 td/osd-backfill-space/datafile 2020-11-16T21:12:11.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-188 td/osd-backfill-space/datafile 2020-11-16T21:12:11.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-189 td/osd-backfill-space/datafile 2020-11-16T21:12:11.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-189 td/osd-backfill-space/datafile 2020-11-16T21:12:11.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-190 td/osd-backfill-space/datafile 2020-11-16T21:12:11.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-190 td/osd-backfill-space/datafile 2020-11-16T21:12:11.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-191 td/osd-backfill-space/datafile 2020-11-16T21:12:11.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-191 td/osd-backfill-space/datafile 2020-11-16T21:12:11.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-192 td/osd-backfill-space/datafile 2020-11-16T21:12:11.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-192 td/osd-backfill-space/datafile 2020-11-16T21:12:11.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-193 td/osd-backfill-space/datafile 2020-11-16T21:12:11.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-193 td/osd-backfill-space/datafile 2020-11-16T21:12:11.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-194 td/osd-backfill-space/datafile 2020-11-16T21:12:11.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-194 td/osd-backfill-space/datafile 2020-11-16T21:12:11.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-195 td/osd-backfill-space/datafile 2020-11-16T21:12:11.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-195 td/osd-backfill-space/datafile 2020-11-16T21:12:11.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-196 td/osd-backfill-space/datafile 2020-11-16T21:12:11.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-196 td/osd-backfill-space/datafile 2020-11-16T21:12:11.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-197 td/osd-backfill-space/datafile 2020-11-16T21:12:11.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-197 td/osd-backfill-space/datafile 2020-11-16T21:12:11.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-198 td/osd-backfill-space/datafile 2020-11-16T21:12:11.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-198 td/osd-backfill-space/datafile 2020-11-16T21:12:11.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-199 td/osd-backfill-space/datafile 2020-11-16T21:12:11.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-199 td/osd-backfill-space/datafile 2020-11-16T21:12:11.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-200 td/osd-backfill-space/datafile 2020-11-16T21:12:11.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-200 td/osd-backfill-space/datafile 2020-11-16T21:12:11.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-201 td/osd-backfill-space/datafile 2020-11-16T21:12:11.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-201 td/osd-backfill-space/datafile 2020-11-16T21:12:11.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-202 td/osd-backfill-space/datafile 2020-11-16T21:12:11.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-202 td/osd-backfill-space/datafile 2020-11-16T21:12:11.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-203 td/osd-backfill-space/datafile 2020-11-16T21:12:11.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-203 td/osd-backfill-space/datafile 2020-11-16T21:12:11.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-204 td/osd-backfill-space/datafile 2020-11-16T21:12:11.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-204 td/osd-backfill-space/datafile 2020-11-16T21:12:11.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-205 td/osd-backfill-space/datafile 2020-11-16T21:12:11.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-205 td/osd-backfill-space/datafile 2020-11-16T21:12:11.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-206 td/osd-backfill-space/datafile 2020-11-16T21:12:11.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-206 td/osd-backfill-space/datafile 2020-11-16T21:12:11.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-207 td/osd-backfill-space/datafile 2020-11-16T21:12:11.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-207 td/osd-backfill-space/datafile 2020-11-16T21:12:11.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-208 td/osd-backfill-space/datafile 2020-11-16T21:12:11.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-208 td/osd-backfill-space/datafile 2020-11-16T21:12:11.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-209 td/osd-backfill-space/datafile 2020-11-16T21:12:11.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-209 td/osd-backfill-space/datafile 2020-11-16T21:12:11.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-210 td/osd-backfill-space/datafile 2020-11-16T21:12:11.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-210 td/osd-backfill-space/datafile 2020-11-16T21:12:11.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-211 td/osd-backfill-space/datafile 2020-11-16T21:12:11.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-211 td/osd-backfill-space/datafile 2020-11-16T21:12:11.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-212 td/osd-backfill-space/datafile 2020-11-16T21:12:11.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-212 td/osd-backfill-space/datafile 2020-11-16T21:12:11.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-213 td/osd-backfill-space/datafile 2020-11-16T21:12:11.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-213 td/osd-backfill-space/datafile 2020-11-16T21:12:11.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-214 td/osd-backfill-space/datafile 2020-11-16T21:12:11.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-214 td/osd-backfill-space/datafile 2020-11-16T21:12:11.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-215 td/osd-backfill-space/datafile 2020-11-16T21:12:11.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-215 td/osd-backfill-space/datafile 2020-11-16T21:12:11.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-216 td/osd-backfill-space/datafile 2020-11-16T21:12:11.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-216 td/osd-backfill-space/datafile 2020-11-16T21:12:11.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-217 td/osd-backfill-space/datafile 2020-11-16T21:12:11.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-217 td/osd-backfill-space/datafile 2020-11-16T21:12:11.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-218 td/osd-backfill-space/datafile 2020-11-16T21:12:11.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-218 td/osd-backfill-space/datafile 2020-11-16T21:12:11.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-219 td/osd-backfill-space/datafile 2020-11-16T21:12:11.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-219 td/osd-backfill-space/datafile 2020-11-16T21:12:11.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-220 td/osd-backfill-space/datafile 2020-11-16T21:12:11.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-220 td/osd-backfill-space/datafile 2020-11-16T21:12:11.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-221 td/osd-backfill-space/datafile 2020-11-16T21:12:11.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-221 td/osd-backfill-space/datafile 2020-11-16T21:12:11.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-222 td/osd-backfill-space/datafile 2020-11-16T21:12:11.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-222 td/osd-backfill-space/datafile 2020-11-16T21:12:11.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-223 td/osd-backfill-space/datafile 2020-11-16T21:12:11.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-223 td/osd-backfill-space/datafile 2020-11-16T21:12:11.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-224 td/osd-backfill-space/datafile 2020-11-16T21:12:11.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-224 td/osd-backfill-space/datafile 2020-11-16T21:12:11.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-225 td/osd-backfill-space/datafile 2020-11-16T21:12:11.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-225 td/osd-backfill-space/datafile 2020-11-16T21:12:11.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-226 td/osd-backfill-space/datafile 2020-11-16T21:12:11.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-226 td/osd-backfill-space/datafile 2020-11-16T21:12:11.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-227 td/osd-backfill-space/datafile 2020-11-16T21:12:11.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-227 td/osd-backfill-space/datafile 2020-11-16T21:12:11.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-228 td/osd-backfill-space/datafile 2020-11-16T21:12:11.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-228 td/osd-backfill-space/datafile 2020-11-16T21:12:11.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-229 td/osd-backfill-space/datafile 2020-11-16T21:12:11.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-229 td/osd-backfill-space/datafile 2020-11-16T21:12:11.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-230 td/osd-backfill-space/datafile 2020-11-16T21:12:11.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-230 td/osd-backfill-space/datafile 2020-11-16T21:12:11.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-231 td/osd-backfill-space/datafile 2020-11-16T21:12:11.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-231 td/osd-backfill-space/datafile 2020-11-16T21:12:11.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-232 td/osd-backfill-space/datafile 2020-11-16T21:12:11.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-232 td/osd-backfill-space/datafile 2020-11-16T21:12:11.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-233 td/osd-backfill-space/datafile 2020-11-16T21:12:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-233 td/osd-backfill-space/datafile 2020-11-16T21:12:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-234 td/osd-backfill-space/datafile 2020-11-16T21:12:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-234 td/osd-backfill-space/datafile 2020-11-16T21:12:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-235 td/osd-backfill-space/datafile 2020-11-16T21:12:11.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-235 td/osd-backfill-space/datafile 2020-11-16T21:12:11.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-236 td/osd-backfill-space/datafile 2020-11-16T21:12:11.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-236 td/osd-backfill-space/datafile 2020-11-16T21:12:11.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-237 td/osd-backfill-space/datafile 2020-11-16T21:12:11.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-237 td/osd-backfill-space/datafile 2020-11-16T21:12:11.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-238 td/osd-backfill-space/datafile 2020-11-16T21:12:11.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-238 td/osd-backfill-space/datafile 2020-11-16T21:12:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-239 td/osd-backfill-space/datafile 2020-11-16T21:12:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-239 td/osd-backfill-space/datafile 2020-11-16T21:12:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-240 td/osd-backfill-space/datafile 2020-11-16T21:12:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-240 td/osd-backfill-space/datafile 2020-11-16T21:12:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-241 td/osd-backfill-space/datafile 2020-11-16T21:12:11.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-241 td/osd-backfill-space/datafile 2020-11-16T21:12:11.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-242 td/osd-backfill-space/datafile 2020-11-16T21:12:11.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-242 td/osd-backfill-space/datafile 2020-11-16T21:12:11.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-243 td/osd-backfill-space/datafile 2020-11-16T21:12:11.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-243 td/osd-backfill-space/datafile 2020-11-16T21:12:11.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-244 td/osd-backfill-space/datafile 2020-11-16T21:12:11.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-244 td/osd-backfill-space/datafile 2020-11-16T21:12:11.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-245 td/osd-backfill-space/datafile 2020-11-16T21:12:11.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-245 td/osd-backfill-space/datafile 2020-11-16T21:12:11.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-246 td/osd-backfill-space/datafile 2020-11-16T21:12:11.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-246 td/osd-backfill-space/datafile 2020-11-16T21:12:11.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-247 td/osd-backfill-space/datafile 2020-11-16T21:12:11.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-247 td/osd-backfill-space/datafile 2020-11-16T21:12:11.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-248 td/osd-backfill-space/datafile 2020-11-16T21:12:11.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-248 td/osd-backfill-space/datafile 2020-11-16T21:12:11.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-249 td/osd-backfill-space/datafile 2020-11-16T21:12:11.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-249 td/osd-backfill-space/datafile 2020-11-16T21:12:11.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-250 td/osd-backfill-space/datafile 2020-11-16T21:12:11.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-250 td/osd-backfill-space/datafile 2020-11-16T21:12:11.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-251 td/osd-backfill-space/datafile 2020-11-16T21:12:11.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-251 td/osd-backfill-space/datafile 2020-11-16T21:12:11.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-252 td/osd-backfill-space/datafile 2020-11-16T21:12:11.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-252 td/osd-backfill-space/datafile 2020-11-16T21:12:11.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-253 td/osd-backfill-space/datafile 2020-11-16T21:12:11.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-253 td/osd-backfill-space/datafile 2020-11-16T21:12:11.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-254 td/osd-backfill-space/datafile 2020-11-16T21:12:11.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-254 td/osd-backfill-space/datafile 2020-11-16T21:12:11.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-255 td/osd-backfill-space/datafile 2020-11-16T21:12:11.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-255 td/osd-backfill-space/datafile 2020-11-16T21:12:11.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-256 td/osd-backfill-space/datafile 2020-11-16T21:12:11.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-256 td/osd-backfill-space/datafile 2020-11-16T21:12:11.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-257 td/osd-backfill-space/datafile 2020-11-16T21:12:11.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-257 td/osd-backfill-space/datafile 2020-11-16T21:12:11.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-258 td/osd-backfill-space/datafile 2020-11-16T21:12:11.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-258 td/osd-backfill-space/datafile 2020-11-16T21:12:11.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-259 td/osd-backfill-space/datafile 2020-11-16T21:12:11.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-259 td/osd-backfill-space/datafile 2020-11-16T21:12:11.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-260 td/osd-backfill-space/datafile 2020-11-16T21:12:11.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-260 td/osd-backfill-space/datafile 2020-11-16T21:12:11.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-261 td/osd-backfill-space/datafile 2020-11-16T21:12:11.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-261 td/osd-backfill-space/datafile 2020-11-16T21:12:11.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-262 td/osd-backfill-space/datafile 2020-11-16T21:12:11.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-262 td/osd-backfill-space/datafile 2020-11-16T21:12:11.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-263 td/osd-backfill-space/datafile 2020-11-16T21:12:11.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-263 td/osd-backfill-space/datafile 2020-11-16T21:12:11.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-264 td/osd-backfill-space/datafile 2020-11-16T21:12:11.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-264 td/osd-backfill-space/datafile 2020-11-16T21:12:11.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-265 td/osd-backfill-space/datafile 2020-11-16T21:12:11.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-265 td/osd-backfill-space/datafile 2020-11-16T21:12:11.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-266 td/osd-backfill-space/datafile 2020-11-16T21:12:11.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-266 td/osd-backfill-space/datafile 2020-11-16T21:12:11.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-267 td/osd-backfill-space/datafile 2020-11-16T21:12:11.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-267 td/osd-backfill-space/datafile 2020-11-16T21:12:11.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-268 td/osd-backfill-space/datafile 2020-11-16T21:12:11.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-268 td/osd-backfill-space/datafile 2020-11-16T21:12:11.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-269 td/osd-backfill-space/datafile 2020-11-16T21:12:11.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-269 td/osd-backfill-space/datafile 2020-11-16T21:12:11.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-270 td/osd-backfill-space/datafile 2020-11-16T21:12:11.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-270 td/osd-backfill-space/datafile 2020-11-16T21:12:11.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-271 td/osd-backfill-space/datafile 2020-11-16T21:12:11.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-271 td/osd-backfill-space/datafile 2020-11-16T21:12:11.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-272 td/osd-backfill-space/datafile 2020-11-16T21:12:11.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-272 td/osd-backfill-space/datafile 2020-11-16T21:12:11.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-273 td/osd-backfill-space/datafile 2020-11-16T21:12:11.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-273 td/osd-backfill-space/datafile 2020-11-16T21:12:11.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-274 td/osd-backfill-space/datafile 2020-11-16T21:12:11.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-274 td/osd-backfill-space/datafile 2020-11-16T21:12:11.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-275 td/osd-backfill-space/datafile 2020-11-16T21:12:11.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-275 td/osd-backfill-space/datafile 2020-11-16T21:12:11.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-276 td/osd-backfill-space/datafile 2020-11-16T21:12:11.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-276 td/osd-backfill-space/datafile 2020-11-16T21:12:11.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-277 td/osd-backfill-space/datafile 2020-11-16T21:12:11.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-277 td/osd-backfill-space/datafile 2020-11-16T21:12:11.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-278 td/osd-backfill-space/datafile 2020-11-16T21:12:11.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-278 td/osd-backfill-space/datafile 2020-11-16T21:12:11.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-279 td/osd-backfill-space/datafile 2020-11-16T21:12:11.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-279 td/osd-backfill-space/datafile 2020-11-16T21:12:11.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-280 td/osd-backfill-space/datafile 2020-11-16T21:12:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-280 td/osd-backfill-space/datafile 2020-11-16T21:12:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-281 td/osd-backfill-space/datafile 2020-11-16T21:12:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-281 td/osd-backfill-space/datafile 2020-11-16T21:12:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-282 td/osd-backfill-space/datafile 2020-11-16T21:12:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-282 td/osd-backfill-space/datafile 2020-11-16T21:12:11.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-283 td/osd-backfill-space/datafile 2020-11-16T21:12:11.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-283 td/osd-backfill-space/datafile 2020-11-16T21:12:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-284 td/osd-backfill-space/datafile 2020-11-16T21:12:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-284 td/osd-backfill-space/datafile 2020-11-16T21:12:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-285 td/osd-backfill-space/datafile 2020-11-16T21:12:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-285 td/osd-backfill-space/datafile 2020-11-16T21:12:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-286 td/osd-backfill-space/datafile 2020-11-16T21:12:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-286 td/osd-backfill-space/datafile 2020-11-16T21:12:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-287 td/osd-backfill-space/datafile 2020-11-16T21:12:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-287 td/osd-backfill-space/datafile 2020-11-16T21:12:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-288 td/osd-backfill-space/datafile 2020-11-16T21:12:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-288 td/osd-backfill-space/datafile 2020-11-16T21:12:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-289 td/osd-backfill-space/datafile 2020-11-16T21:12:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-289 td/osd-backfill-space/datafile 2020-11-16T21:12:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-290 td/osd-backfill-space/datafile 2020-11-16T21:12:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-290 td/osd-backfill-space/datafile 2020-11-16T21:12:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-291 td/osd-backfill-space/datafile 2020-11-16T21:12:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-291 td/osd-backfill-space/datafile 2020-11-16T21:12:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-292 td/osd-backfill-space/datafile 2020-11-16T21:12:11.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-292 td/osd-backfill-space/datafile 2020-11-16T21:12:11.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-293 td/osd-backfill-space/datafile 2020-11-16T21:12:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-293 td/osd-backfill-space/datafile 2020-11-16T21:12:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-294 td/osd-backfill-space/datafile 2020-11-16T21:12:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-294 td/osd-backfill-space/datafile 2020-11-16T21:12:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-295 td/osd-backfill-space/datafile 2020-11-16T21:12:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-295 td/osd-backfill-space/datafile 2020-11-16T21:12:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-296 td/osd-backfill-space/datafile 2020-11-16T21:12:11.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-296 td/osd-backfill-space/datafile 2020-11-16T21:12:11.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-297 td/osd-backfill-space/datafile 2020-11-16T21:12:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-297 td/osd-backfill-space/datafile 2020-11-16T21:12:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-298 td/osd-backfill-space/datafile 2020-11-16T21:12:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-298 td/osd-backfill-space/datafile 2020-11-16T21:12:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-299 td/osd-backfill-space/datafile 2020-11-16T21:12:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-299 td/osd-backfill-space/datafile 2020-11-16T21:12:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-300 td/osd-backfill-space/datafile 2020-11-16T21:12:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-300 td/osd-backfill-space/datafile 2020-11-16T21:12:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-301 td/osd-backfill-space/datafile 2020-11-16T21:12:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-301 td/osd-backfill-space/datafile 2020-11-16T21:12:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-302 td/osd-backfill-space/datafile 2020-11-16T21:12:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-302 td/osd-backfill-space/datafile 2020-11-16T21:12:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-303 td/osd-backfill-space/datafile 2020-11-16T21:12:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-303 td/osd-backfill-space/datafile 2020-11-16T21:12:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-304 td/osd-backfill-space/datafile 2020-11-16T21:12:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-304 td/osd-backfill-space/datafile 2020-11-16T21:12:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-305 td/osd-backfill-space/datafile 2020-11-16T21:12:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-305 td/osd-backfill-space/datafile 2020-11-16T21:12:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-306 td/osd-backfill-space/datafile 2020-11-16T21:12:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-306 td/osd-backfill-space/datafile 2020-11-16T21:12:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-307 td/osd-backfill-space/datafile 2020-11-16T21:12:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-307 td/osd-backfill-space/datafile 2020-11-16T21:12:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-308 td/osd-backfill-space/datafile 2020-11-16T21:12:11.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-308 td/osd-backfill-space/datafile 2020-11-16T21:12:11.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-309 td/osd-backfill-space/datafile 2020-11-16T21:12:11.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-309 td/osd-backfill-space/datafile 2020-11-16T21:12:11.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-310 td/osd-backfill-space/datafile 2020-11-16T21:12:11.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-310 td/osd-backfill-space/datafile 2020-11-16T21:12:11.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-311 td/osd-backfill-space/datafile 2020-11-16T21:12:11.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-311 td/osd-backfill-space/datafile 2020-11-16T21:12:11.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-312 td/osd-backfill-space/datafile 2020-11-16T21:12:11.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-312 td/osd-backfill-space/datafile 2020-11-16T21:12:11.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-313 td/osd-backfill-space/datafile 2020-11-16T21:12:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-313 td/osd-backfill-space/datafile 2020-11-16T21:12:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-314 td/osd-backfill-space/datafile 2020-11-16T21:12:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-314 td/osd-backfill-space/datafile 2020-11-16T21:12:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-315 td/osd-backfill-space/datafile 2020-11-16T21:12:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-315 td/osd-backfill-space/datafile 2020-11-16T21:12:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-316 td/osd-backfill-space/datafile 2020-11-16T21:12:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-316 td/osd-backfill-space/datafile 2020-11-16T21:12:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-317 td/osd-backfill-space/datafile 2020-11-16T21:12:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-317 td/osd-backfill-space/datafile 2020-11-16T21:12:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-318 td/osd-backfill-space/datafile 2020-11-16T21:12:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-318 td/osd-backfill-space/datafile 2020-11-16T21:12:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-319 td/osd-backfill-space/datafile 2020-11-16T21:12:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-319 td/osd-backfill-space/datafile 2020-11-16T21:12:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-320 td/osd-backfill-space/datafile 2020-11-16T21:12:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-320 td/osd-backfill-space/datafile 2020-11-16T21:12:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-321 td/osd-backfill-space/datafile 2020-11-16T21:12:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-321 td/osd-backfill-space/datafile 2020-11-16T21:12:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-322 td/osd-backfill-space/datafile 2020-11-16T21:12:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-322 td/osd-backfill-space/datafile 2020-11-16T21:12:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-323 td/osd-backfill-space/datafile 2020-11-16T21:12:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-323 td/osd-backfill-space/datafile 2020-11-16T21:12:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-324 td/osd-backfill-space/datafile 2020-11-16T21:12:11.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-324 td/osd-backfill-space/datafile 2020-11-16T21:12:11.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-325 td/osd-backfill-space/datafile 2020-11-16T21:12:11.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-325 td/osd-backfill-space/datafile 2020-11-16T21:12:11.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-326 td/osd-backfill-space/datafile 2020-11-16T21:12:11.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-326 td/osd-backfill-space/datafile 2020-11-16T21:12:11.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-327 td/osd-backfill-space/datafile 2020-11-16T21:12:11.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-327 td/osd-backfill-space/datafile 2020-11-16T21:12:11.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-328 td/osd-backfill-space/datafile 2020-11-16T21:12:11.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-328 td/osd-backfill-space/datafile 2020-11-16T21:12:11.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-329 td/osd-backfill-space/datafile 2020-11-16T21:12:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-329 td/osd-backfill-space/datafile 2020-11-16T21:12:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-330 td/osd-backfill-space/datafile 2020-11-16T21:12:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-330 td/osd-backfill-space/datafile 2020-11-16T21:12:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-331 td/osd-backfill-space/datafile 2020-11-16T21:12:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-331 td/osd-backfill-space/datafile 2020-11-16T21:12:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-332 td/osd-backfill-space/datafile 2020-11-16T21:12:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-332 td/osd-backfill-space/datafile 2020-11-16T21:12:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-333 td/osd-backfill-space/datafile 2020-11-16T21:12:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-333 td/osd-backfill-space/datafile 2020-11-16T21:12:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-334 td/osd-backfill-space/datafile 2020-11-16T21:12:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-334 td/osd-backfill-space/datafile 2020-11-16T21:12:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-335 td/osd-backfill-space/datafile 2020-11-16T21:12:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-335 td/osd-backfill-space/datafile 2020-11-16T21:12:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-336 td/osd-backfill-space/datafile 2020-11-16T21:12:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-336 td/osd-backfill-space/datafile 2020-11-16T21:12:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-337 td/osd-backfill-space/datafile 2020-11-16T21:12:11.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-337 td/osd-backfill-space/datafile 2020-11-16T21:12:11.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-338 td/osd-backfill-space/datafile 2020-11-16T21:12:11.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-338 td/osd-backfill-space/datafile 2020-11-16T21:12:11.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-339 td/osd-backfill-space/datafile 2020-11-16T21:12:11.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-339 td/osd-backfill-space/datafile 2020-11-16T21:12:11.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-340 td/osd-backfill-space/datafile 2020-11-16T21:12:11.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-340 td/osd-backfill-space/datafile 2020-11-16T21:12:11.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-341 td/osd-backfill-space/datafile 2020-11-16T21:12:11.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-341 td/osd-backfill-space/datafile 2020-11-16T21:12:11.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-342 td/osd-backfill-space/datafile 2020-11-16T21:12:11.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-342 td/osd-backfill-space/datafile 2020-11-16T21:12:11.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-343 td/osd-backfill-space/datafile 2020-11-16T21:12:11.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-343 td/osd-backfill-space/datafile 2020-11-16T21:12:11.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-344 td/osd-backfill-space/datafile 2020-11-16T21:12:11.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-344 td/osd-backfill-space/datafile 2020-11-16T21:12:11.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-345 td/osd-backfill-space/datafile 2020-11-16T21:12:11.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-345 td/osd-backfill-space/datafile 2020-11-16T21:12:11.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-346 td/osd-backfill-space/datafile 2020-11-16T21:12:11.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-346 td/osd-backfill-space/datafile 2020-11-16T21:12:11.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-347 td/osd-backfill-space/datafile 2020-11-16T21:12:11.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-347 td/osd-backfill-space/datafile 2020-11-16T21:12:11.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-348 td/osd-backfill-space/datafile 2020-11-16T21:12:11.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-348 td/osd-backfill-space/datafile 2020-11-16T21:12:11.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-349 td/osd-backfill-space/datafile 2020-11-16T21:12:11.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-349 td/osd-backfill-space/datafile 2020-11-16T21:12:11.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-350 td/osd-backfill-space/datafile 2020-11-16T21:12:11.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-350 td/osd-backfill-space/datafile 2020-11-16T21:12:11.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-351 td/osd-backfill-space/datafile 2020-11-16T21:12:11.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-351 td/osd-backfill-space/datafile 2020-11-16T21:12:11.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-352 td/osd-backfill-space/datafile 2020-11-16T21:12:11.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-352 td/osd-backfill-space/datafile 2020-11-16T21:12:11.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-353 td/osd-backfill-space/datafile 2020-11-16T21:12:11.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-353 td/osd-backfill-space/datafile 2020-11-16T21:12:11.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-354 td/osd-backfill-space/datafile 2020-11-16T21:12:11.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-354 td/osd-backfill-space/datafile 2020-11-16T21:12:11.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-355 td/osd-backfill-space/datafile 2020-11-16T21:12:11.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-355 td/osd-backfill-space/datafile 2020-11-16T21:12:11.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-356 td/osd-backfill-space/datafile 2020-11-16T21:12:11.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-356 td/osd-backfill-space/datafile 2020-11-16T21:12:11.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-357 td/osd-backfill-space/datafile 2020-11-16T21:12:11.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-357 td/osd-backfill-space/datafile 2020-11-16T21:12:11.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-358 td/osd-backfill-space/datafile 2020-11-16T21:12:11.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-358 td/osd-backfill-space/datafile 2020-11-16T21:12:11.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-359 td/osd-backfill-space/datafile 2020-11-16T21:12:11.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-359 td/osd-backfill-space/datafile 2020-11-16T21:12:11.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-360 td/osd-backfill-space/datafile 2020-11-16T21:12:11.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-360 td/osd-backfill-space/datafile 2020-11-16T21:12:11.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-361 td/osd-backfill-space/datafile 2020-11-16T21:12:11.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-361 td/osd-backfill-space/datafile 2020-11-16T21:12:11.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-362 td/osd-backfill-space/datafile 2020-11-16T21:12:11.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-362 td/osd-backfill-space/datafile 2020-11-16T21:12:11.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-363 td/osd-backfill-space/datafile 2020-11-16T21:12:11.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-363 td/osd-backfill-space/datafile 2020-11-16T21:12:11.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-364 td/osd-backfill-space/datafile 2020-11-16T21:12:11.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-364 td/osd-backfill-space/datafile 2020-11-16T21:12:11.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-365 td/osd-backfill-space/datafile 2020-11-16T21:12:11.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-365 td/osd-backfill-space/datafile 2020-11-16T21:12:11.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-366 td/osd-backfill-space/datafile 2020-11-16T21:12:11.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-366 td/osd-backfill-space/datafile 2020-11-16T21:12:11.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-367 td/osd-backfill-space/datafile 2020-11-16T21:12:11.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-367 td/osd-backfill-space/datafile 2020-11-16T21:12:11.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-368 td/osd-backfill-space/datafile 2020-11-16T21:12:11.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-368 td/osd-backfill-space/datafile 2020-11-16T21:12:11.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-369 td/osd-backfill-space/datafile 2020-11-16T21:12:11.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-369 td/osd-backfill-space/datafile 2020-11-16T21:12:11.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-370 td/osd-backfill-space/datafile 2020-11-16T21:12:11.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-370 td/osd-backfill-space/datafile 2020-11-16T21:12:11.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-371 td/osd-backfill-space/datafile 2020-11-16T21:12:11.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-371 td/osd-backfill-space/datafile 2020-11-16T21:12:11.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-372 td/osd-backfill-space/datafile 2020-11-16T21:12:11.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-372 td/osd-backfill-space/datafile 2020-11-16T21:12:11.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-373 td/osd-backfill-space/datafile 2020-11-16T21:12:11.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-373 td/osd-backfill-space/datafile 2020-11-16T21:12:11.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-374 td/osd-backfill-space/datafile 2020-11-16T21:12:11.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-374 td/osd-backfill-space/datafile 2020-11-16T21:12:11.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-375 td/osd-backfill-space/datafile 2020-11-16T21:12:11.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-375 td/osd-backfill-space/datafile 2020-11-16T21:12:11.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-376 td/osd-backfill-space/datafile 2020-11-16T21:12:11.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-376 td/osd-backfill-space/datafile 2020-11-16T21:12:11.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-377 td/osd-backfill-space/datafile 2020-11-16T21:12:11.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-377 td/osd-backfill-space/datafile 2020-11-16T21:12:11.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-378 td/osd-backfill-space/datafile 2020-11-16T21:12:11.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-378 td/osd-backfill-space/datafile 2020-11-16T21:12:11.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-379 td/osd-backfill-space/datafile 2020-11-16T21:12:11.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-379 td/osd-backfill-space/datafile 2020-11-16T21:12:11.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-380 td/osd-backfill-space/datafile 2020-11-16T21:12:11.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-380 td/osd-backfill-space/datafile 2020-11-16T21:12:11.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-381 td/osd-backfill-space/datafile 2020-11-16T21:12:11.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-381 td/osd-backfill-space/datafile 2020-11-16T21:12:11.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-382 td/osd-backfill-space/datafile 2020-11-16T21:12:11.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-382 td/osd-backfill-space/datafile 2020-11-16T21:12:11.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-383 td/osd-backfill-space/datafile 2020-11-16T21:12:11.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-383 td/osd-backfill-space/datafile 2020-11-16T21:12:11.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-384 td/osd-backfill-space/datafile 2020-11-16T21:12:11.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-384 td/osd-backfill-space/datafile 2020-11-16T21:12:11.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-385 td/osd-backfill-space/datafile 2020-11-16T21:12:11.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-385 td/osd-backfill-space/datafile 2020-11-16T21:12:11.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-386 td/osd-backfill-space/datafile 2020-11-16T21:12:11.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-386 td/osd-backfill-space/datafile 2020-11-16T21:12:11.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-387 td/osd-backfill-space/datafile 2020-11-16T21:12:11.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-387 td/osd-backfill-space/datafile 2020-11-16T21:12:11.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-388 td/osd-backfill-space/datafile 2020-11-16T21:12:11.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-388 td/osd-backfill-space/datafile 2020-11-16T21:12:11.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-389 td/osd-backfill-space/datafile 2020-11-16T21:12:11.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-389 td/osd-backfill-space/datafile 2020-11-16T21:12:11.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-390 td/osd-backfill-space/datafile 2020-11-16T21:12:11.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-390 td/osd-backfill-space/datafile 2020-11-16T21:12:11.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-391 td/osd-backfill-space/datafile 2020-11-16T21:12:11.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-391 td/osd-backfill-space/datafile 2020-11-16T21:12:11.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-392 td/osd-backfill-space/datafile 2020-11-16T21:12:11.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-392 td/osd-backfill-space/datafile 2020-11-16T21:12:11.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-393 td/osd-backfill-space/datafile 2020-11-16T21:12:11.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-393 td/osd-backfill-space/datafile 2020-11-16T21:12:11.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-394 td/osd-backfill-space/datafile 2020-11-16T21:12:11.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-394 td/osd-backfill-space/datafile 2020-11-16T21:12:11.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-395 td/osd-backfill-space/datafile 2020-11-16T21:12:11.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-395 td/osd-backfill-space/datafile 2020-11-16T21:12:11.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-396 td/osd-backfill-space/datafile 2020-11-16T21:12:11.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-396 td/osd-backfill-space/datafile 2020-11-16T21:12:11.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-397 td/osd-backfill-space/datafile 2020-11-16T21:12:11.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-397 td/osd-backfill-space/datafile 2020-11-16T21:12:11.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-398 td/osd-backfill-space/datafile 2020-11-16T21:12:11.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-398 td/osd-backfill-space/datafile 2020-11-16T21:12:11.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-399 td/osd-backfill-space/datafile 2020-11-16T21:12:11.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-399 td/osd-backfill-space/datafile 2020-11-16T21:12:11.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-400 td/osd-backfill-space/datafile 2020-11-16T21:12:11.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-400 td/osd-backfill-space/datafile 2020-11-16T21:12:11.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-401 td/osd-backfill-space/datafile 2020-11-16T21:12:11.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-401 td/osd-backfill-space/datafile 2020-11-16T21:12:11.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-402 td/osd-backfill-space/datafile 2020-11-16T21:12:11.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-402 td/osd-backfill-space/datafile 2020-11-16T21:12:11.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-403 td/osd-backfill-space/datafile 2020-11-16T21:12:11.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-403 td/osd-backfill-space/datafile 2020-11-16T21:12:11.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-404 td/osd-backfill-space/datafile 2020-11-16T21:12:11.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-404 td/osd-backfill-space/datafile 2020-11-16T21:12:11.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-405 td/osd-backfill-space/datafile 2020-11-16T21:12:11.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-405 td/osd-backfill-space/datafile 2020-11-16T21:12:11.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-406 td/osd-backfill-space/datafile 2020-11-16T21:12:11.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-406 td/osd-backfill-space/datafile 2020-11-16T21:12:11.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-407 td/osd-backfill-space/datafile 2020-11-16T21:12:11.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-407 td/osd-backfill-space/datafile 2020-11-16T21:12:11.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-408 td/osd-backfill-space/datafile 2020-11-16T21:12:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-408 td/osd-backfill-space/datafile 2020-11-16T21:12:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-409 td/osd-backfill-space/datafile 2020-11-16T21:12:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-409 td/osd-backfill-space/datafile 2020-11-16T21:12:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-410 td/osd-backfill-space/datafile 2020-11-16T21:12:11.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-410 td/osd-backfill-space/datafile 2020-11-16T21:12:11.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-411 td/osd-backfill-space/datafile 2020-11-16T21:12:11.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-411 td/osd-backfill-space/datafile 2020-11-16T21:12:11.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-412 td/osd-backfill-space/datafile 2020-11-16T21:12:11.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-412 td/osd-backfill-space/datafile 2020-11-16T21:12:11.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-413 td/osd-backfill-space/datafile 2020-11-16T21:12:11.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-413 td/osd-backfill-space/datafile 2020-11-16T21:12:11.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.588 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-414 td/osd-backfill-space/datafile 2020-11-16T21:12:11.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-414 td/osd-backfill-space/datafile 2020-11-16T21:12:11.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.590 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-415 td/osd-backfill-space/datafile 2020-11-16T21:12:11.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-415 td/osd-backfill-space/datafile 2020-11-16T21:12:11.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-416 td/osd-backfill-space/datafile 2020-11-16T21:12:11.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-416 td/osd-backfill-space/datafile 2020-11-16T21:12:11.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-417 td/osd-backfill-space/datafile 2020-11-16T21:12:11.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-417 td/osd-backfill-space/datafile 2020-11-16T21:12:11.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-418 td/osd-backfill-space/datafile 2020-11-16T21:12:11.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-418 td/osd-backfill-space/datafile 2020-11-16T21:12:11.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-419 td/osd-backfill-space/datafile 2020-11-16T21:12:11.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-419 td/osd-backfill-space/datafile 2020-11-16T21:12:11.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-420 td/osd-backfill-space/datafile 2020-11-16T21:12:11.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-420 td/osd-backfill-space/datafile 2020-11-16T21:12:11.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-421 td/osd-backfill-space/datafile 2020-11-16T21:12:11.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-421 td/osd-backfill-space/datafile 2020-11-16T21:12:11.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-422 td/osd-backfill-space/datafile 2020-11-16T21:12:11.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-422 td/osd-backfill-space/datafile 2020-11-16T21:12:11.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-423 td/osd-backfill-space/datafile 2020-11-16T21:12:11.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-423 td/osd-backfill-space/datafile 2020-11-16T21:12:11.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-424 td/osd-backfill-space/datafile 2020-11-16T21:12:11.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-424 td/osd-backfill-space/datafile 2020-11-16T21:12:11.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-425 td/osd-backfill-space/datafile 2020-11-16T21:12:11.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-425 td/osd-backfill-space/datafile 2020-11-16T21:12:11.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-426 td/osd-backfill-space/datafile 2020-11-16T21:12:11.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-426 td/osd-backfill-space/datafile 2020-11-16T21:12:11.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-427 td/osd-backfill-space/datafile 2020-11-16T21:12:11.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-427 td/osd-backfill-space/datafile 2020-11-16T21:12:11.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-428 td/osd-backfill-space/datafile 2020-11-16T21:12:11.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-428 td/osd-backfill-space/datafile 2020-11-16T21:12:11.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.602 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-429 td/osd-backfill-space/datafile 2020-11-16T21:12:11.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-429 td/osd-backfill-space/datafile 2020-11-16T21:12:11.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-430 td/osd-backfill-space/datafile 2020-11-16T21:12:11.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-430 td/osd-backfill-space/datafile 2020-11-16T21:12:11.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-431 td/osd-backfill-space/datafile 2020-11-16T21:12:11.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-431 td/osd-backfill-space/datafile 2020-11-16T21:12:11.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-432 td/osd-backfill-space/datafile 2020-11-16T21:12:11.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-432 td/osd-backfill-space/datafile 2020-11-16T21:12:11.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-433 td/osd-backfill-space/datafile 2020-11-16T21:12:11.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-433 td/osd-backfill-space/datafile 2020-11-16T21:12:11.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-434 td/osd-backfill-space/datafile 2020-11-16T21:12:11.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-434 td/osd-backfill-space/datafile 2020-11-16T21:12:11.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-435 td/osd-backfill-space/datafile 2020-11-16T21:12:11.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-435 td/osd-backfill-space/datafile 2020-11-16T21:12:11.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-436 td/osd-backfill-space/datafile 2020-11-16T21:12:11.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-436 td/osd-backfill-space/datafile 2020-11-16T21:12:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-437 td/osd-backfill-space/datafile 2020-11-16T21:12:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-437 td/osd-backfill-space/datafile 2020-11-16T21:12:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-438 td/osd-backfill-space/datafile 2020-11-16T21:12:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-438 td/osd-backfill-space/datafile 2020-11-16T21:12:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-439 td/osd-backfill-space/datafile 2020-11-16T21:12:11.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-439 td/osd-backfill-space/datafile 2020-11-16T21:12:11.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-440 td/osd-backfill-space/datafile 2020-11-16T21:12:11.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-440 td/osd-backfill-space/datafile 2020-11-16T21:12:11.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-441 td/osd-backfill-space/datafile 2020-11-16T21:12:11.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-441 td/osd-backfill-space/datafile 2020-11-16T21:12:11.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-442 td/osd-backfill-space/datafile 2020-11-16T21:12:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-442 td/osd-backfill-space/datafile 2020-11-16T21:12:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-443 td/osd-backfill-space/datafile 2020-11-16T21:12:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-443 td/osd-backfill-space/datafile 2020-11-16T21:12:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-444 td/osd-backfill-space/datafile 2020-11-16T21:12:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-444 td/osd-backfill-space/datafile 2020-11-16T21:12:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-445 td/osd-backfill-space/datafile 2020-11-16T21:12:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-445 td/osd-backfill-space/datafile 2020-11-16T21:12:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-446 td/osd-backfill-space/datafile 2020-11-16T21:12:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-446 td/osd-backfill-space/datafile 2020-11-16T21:12:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-447 td/osd-backfill-space/datafile 2020-11-16T21:12:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-447 td/osd-backfill-space/datafile 2020-11-16T21:12:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-448 td/osd-backfill-space/datafile 2020-11-16T21:12:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-448 td/osd-backfill-space/datafile 2020-11-16T21:12:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-449 td/osd-backfill-space/datafile 2020-11-16T21:12:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-449 td/osd-backfill-space/datafile 2020-11-16T21:12:11.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-450 td/osd-backfill-space/datafile 2020-11-16T21:12:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-450 td/osd-backfill-space/datafile 2020-11-16T21:12:11.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-451 td/osd-backfill-space/datafile 2020-11-16T21:12:11.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-451 td/osd-backfill-space/datafile 2020-11-16T21:12:11.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-452 td/osd-backfill-space/datafile 2020-11-16T21:12:11.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-452 td/osd-backfill-space/datafile 2020-11-16T21:12:11.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-453 td/osd-backfill-space/datafile 2020-11-16T21:12:11.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-453 td/osd-backfill-space/datafile 2020-11-16T21:12:11.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-454 td/osd-backfill-space/datafile 2020-11-16T21:12:11.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-454 td/osd-backfill-space/datafile 2020-11-16T21:12:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-455 td/osd-backfill-space/datafile 2020-11-16T21:12:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-455 td/osd-backfill-space/datafile 2020-11-16T21:12:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-456 td/osd-backfill-space/datafile 2020-11-16T21:12:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-456 td/osd-backfill-space/datafile 2020-11-16T21:12:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-457 td/osd-backfill-space/datafile 2020-11-16T21:12:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-457 td/osd-backfill-space/datafile 2020-11-16T21:12:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-458 td/osd-backfill-space/datafile 2020-11-16T21:12:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-458 td/osd-backfill-space/datafile 2020-11-16T21:12:11.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-459 td/osd-backfill-space/datafile 2020-11-16T21:12:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-459 td/osd-backfill-space/datafile 2020-11-16T21:12:11.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-460 td/osd-backfill-space/datafile 2020-11-16T21:12:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-460 td/osd-backfill-space/datafile 2020-11-16T21:12:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-461 td/osd-backfill-space/datafile 2020-11-16T21:12:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-461 td/osd-backfill-space/datafile 2020-11-16T21:12:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-462 td/osd-backfill-space/datafile 2020-11-16T21:12:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-462 td/osd-backfill-space/datafile 2020-11-16T21:12:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-463 td/osd-backfill-space/datafile 2020-11-16T21:12:11.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-463 td/osd-backfill-space/datafile 2020-11-16T21:12:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-464 td/osd-backfill-space/datafile 2020-11-16T21:12:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-464 td/osd-backfill-space/datafile 2020-11-16T21:12:11.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-465 td/osd-backfill-space/datafile 2020-11-16T21:12:11.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-465 td/osd-backfill-space/datafile 2020-11-16T21:12:11.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-466 td/osd-backfill-space/datafile 2020-11-16T21:12:11.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-466 td/osd-backfill-space/datafile 2020-11-16T21:12:11.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-467 td/osd-backfill-space/datafile 2020-11-16T21:12:11.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-467 td/osd-backfill-space/datafile 2020-11-16T21:12:11.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-468 td/osd-backfill-space/datafile 2020-11-16T21:12:11.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-468 td/osd-backfill-space/datafile 2020-11-16T21:12:11.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-469 td/osd-backfill-space/datafile 2020-11-16T21:12:11.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-469 td/osd-backfill-space/datafile 2020-11-16T21:12:11.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-470 td/osd-backfill-space/datafile 2020-11-16T21:12:11.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-470 td/osd-backfill-space/datafile 2020-11-16T21:12:11.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-471 td/osd-backfill-space/datafile 2020-11-16T21:12:11.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-471 td/osd-backfill-space/datafile 2020-11-16T21:12:11.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-472 td/osd-backfill-space/datafile 2020-11-16T21:12:11.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-472 td/osd-backfill-space/datafile 2020-11-16T21:12:11.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-473 td/osd-backfill-space/datafile 2020-11-16T21:12:11.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-473 td/osd-backfill-space/datafile 2020-11-16T21:12:11.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-474 td/osd-backfill-space/datafile 2020-11-16T21:12:11.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-474 td/osd-backfill-space/datafile 2020-11-16T21:12:11.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-475 td/osd-backfill-space/datafile 2020-11-16T21:12:11.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-475 td/osd-backfill-space/datafile 2020-11-16T21:12:11.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-476 td/osd-backfill-space/datafile 2020-11-16T21:12:11.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-476 td/osd-backfill-space/datafile 2020-11-16T21:12:11.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-477 td/osd-backfill-space/datafile 2020-11-16T21:12:11.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-477 td/osd-backfill-space/datafile 2020-11-16T21:12:11.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-478 td/osd-backfill-space/datafile 2020-11-16T21:12:11.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-478 td/osd-backfill-space/datafile 2020-11-16T21:12:11.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-479 td/osd-backfill-space/datafile 2020-11-16T21:12:11.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-479 td/osd-backfill-space/datafile 2020-11-16T21:12:11.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-480 td/osd-backfill-space/datafile 2020-11-16T21:12:11.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-480 td/osd-backfill-space/datafile 2020-11-16T21:12:11.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-481 td/osd-backfill-space/datafile 2020-11-16T21:12:11.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-481 td/osd-backfill-space/datafile 2020-11-16T21:12:11.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-482 td/osd-backfill-space/datafile 2020-11-16T21:12:11.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-482 td/osd-backfill-space/datafile 2020-11-16T21:12:11.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-483 td/osd-backfill-space/datafile 2020-11-16T21:12:11.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-483 td/osd-backfill-space/datafile 2020-11-16T21:12:11.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-484 td/osd-backfill-space/datafile 2020-11-16T21:12:11.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-484 td/osd-backfill-space/datafile 2020-11-16T21:12:11.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-485 td/osd-backfill-space/datafile 2020-11-16T21:12:11.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-485 td/osd-backfill-space/datafile 2020-11-16T21:12:11.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-486 td/osd-backfill-space/datafile 2020-11-16T21:12:11.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-486 td/osd-backfill-space/datafile 2020-11-16T21:12:11.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-487 td/osd-backfill-space/datafile 2020-11-16T21:12:11.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-487 td/osd-backfill-space/datafile 2020-11-16T21:12:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-488 td/osd-backfill-space/datafile 2020-11-16T21:12:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-488 td/osd-backfill-space/datafile 2020-11-16T21:12:11.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-489 td/osd-backfill-space/datafile 2020-11-16T21:12:11.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-489 td/osd-backfill-space/datafile 2020-11-16T21:12:11.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-490 td/osd-backfill-space/datafile 2020-11-16T21:12:11.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-490 td/osd-backfill-space/datafile 2020-11-16T21:12:11.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-491 td/osd-backfill-space/datafile 2020-11-16T21:12:11.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-491 td/osd-backfill-space/datafile 2020-11-16T21:12:11.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-492 td/osd-backfill-space/datafile 2020-11-16T21:12:11.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-492 td/osd-backfill-space/datafile 2020-11-16T21:12:11.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-493 td/osd-backfill-space/datafile 2020-11-16T21:12:11.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-493 td/osd-backfill-space/datafile 2020-11-16T21:12:11.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-494 td/osd-backfill-space/datafile 2020-11-16T21:12:11.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-494 td/osd-backfill-space/datafile 2020-11-16T21:12:11.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-495 td/osd-backfill-space/datafile 2020-11-16T21:12:11.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-495 td/osd-backfill-space/datafile 2020-11-16T21:12:11.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-496 td/osd-backfill-space/datafile 2020-11-16T21:12:11.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-496 td/osd-backfill-space/datafile 2020-11-16T21:12:11.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-497 td/osd-backfill-space/datafile 2020-11-16T21:12:11.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-497 td/osd-backfill-space/datafile 2020-11-16T21:12:11.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-498 td/osd-backfill-space/datafile 2020-11-16T21:12:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-498 td/osd-backfill-space/datafile 2020-11-16T21:12:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-499 td/osd-backfill-space/datafile 2020-11-16T21:12:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-499 td/osd-backfill-space/datafile 2020-11-16T21:12:11.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-500 td/osd-backfill-space/datafile 2020-11-16T21:12:11.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-500 td/osd-backfill-space/datafile 2020-11-16T21:12:11.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-501 td/osd-backfill-space/datafile 2020-11-16T21:12:11.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-501 td/osd-backfill-space/datafile 2020-11-16T21:12:11.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-502 td/osd-backfill-space/datafile 2020-11-16T21:12:11.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-502 td/osd-backfill-space/datafile 2020-11-16T21:12:11.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-503 td/osd-backfill-space/datafile 2020-11-16T21:12:11.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-503 td/osd-backfill-space/datafile 2020-11-16T21:12:11.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-504 td/osd-backfill-space/datafile 2020-11-16T21:12:11.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-504 td/osd-backfill-space/datafile 2020-11-16T21:12:11.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-505 td/osd-backfill-space/datafile 2020-11-16T21:12:11.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-505 td/osd-backfill-space/datafile 2020-11-16T21:12:11.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-506 td/osd-backfill-space/datafile 2020-11-16T21:12:11.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-506 td/osd-backfill-space/datafile 2020-11-16T21:12:11.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-507 td/osd-backfill-space/datafile 2020-11-16T21:12:11.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-507 td/osd-backfill-space/datafile 2020-11-16T21:12:11.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-508 td/osd-backfill-space/datafile 2020-11-16T21:12:11.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-508 td/osd-backfill-space/datafile 2020-11-16T21:12:11.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-509 td/osd-backfill-space/datafile 2020-11-16T21:12:11.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-509 td/osd-backfill-space/datafile 2020-11-16T21:12:11.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-510 td/osd-backfill-space/datafile 2020-11-16T21:12:11.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-510 td/osd-backfill-space/datafile 2020-11-16T21:12:11.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-511 td/osd-backfill-space/datafile 2020-11-16T21:12:11.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-511 td/osd-backfill-space/datafile 2020-11-16T21:12:11.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-512 td/osd-backfill-space/datafile 2020-11-16T21:12:11.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-512 td/osd-backfill-space/datafile 2020-11-16T21:12:11.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-513 td/osd-backfill-space/datafile 2020-11-16T21:12:11.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-513 td/osd-backfill-space/datafile 2020-11-16T21:12:11.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-514 td/osd-backfill-space/datafile 2020-11-16T21:12:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-514 td/osd-backfill-space/datafile 2020-11-16T21:12:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-515 td/osd-backfill-space/datafile 2020-11-16T21:12:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-515 td/osd-backfill-space/datafile 2020-11-16T21:12:11.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-516 td/osd-backfill-space/datafile 2020-11-16T21:12:11.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-516 td/osd-backfill-space/datafile 2020-11-16T21:12:11.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-517 td/osd-backfill-space/datafile 2020-11-16T21:12:11.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-517 td/osd-backfill-space/datafile 2020-11-16T21:12:11.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-518 td/osd-backfill-space/datafile 2020-11-16T21:12:11.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-518 td/osd-backfill-space/datafile 2020-11-16T21:12:11.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-519 td/osd-backfill-space/datafile 2020-11-16T21:12:11.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-519 td/osd-backfill-space/datafile 2020-11-16T21:12:11.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.679 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-520 td/osd-backfill-space/datafile 2020-11-16T21:12:11.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-520 td/osd-backfill-space/datafile 2020-11-16T21:12:11.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.679 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-521 td/osd-backfill-space/datafile 2020-11-16T21:12:11.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-521 td/osd-backfill-space/datafile 2020-11-16T21:12:11.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-522 td/osd-backfill-space/datafile 2020-11-16T21:12:11.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-522 td/osd-backfill-space/datafile 2020-11-16T21:12:11.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-523 td/osd-backfill-space/datafile 2020-11-16T21:12:11.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-523 td/osd-backfill-space/datafile 2020-11-16T21:12:11.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.682 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-524 td/osd-backfill-space/datafile 2020-11-16T21:12:11.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-524 td/osd-backfill-space/datafile 2020-11-16T21:12:11.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-525 td/osd-backfill-space/datafile 2020-11-16T21:12:11.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-525 td/osd-backfill-space/datafile 2020-11-16T21:12:11.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-526 td/osd-backfill-space/datafile 2020-11-16T21:12:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-526 td/osd-backfill-space/datafile 2020-11-16T21:12:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-527 td/osd-backfill-space/datafile 2020-11-16T21:12:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-527 td/osd-backfill-space/datafile 2020-11-16T21:12:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-528 td/osd-backfill-space/datafile 2020-11-16T21:12:11.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-528 td/osd-backfill-space/datafile 2020-11-16T21:12:11.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-529 td/osd-backfill-space/datafile 2020-11-16T21:12:11.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-529 td/osd-backfill-space/datafile 2020-11-16T21:12:11.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-530 td/osd-backfill-space/datafile 2020-11-16T21:12:11.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-530 td/osd-backfill-space/datafile 2020-11-16T21:12:11.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-531 td/osd-backfill-space/datafile 2020-11-16T21:12:11.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-531 td/osd-backfill-space/datafile 2020-11-16T21:12:11.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-532 td/osd-backfill-space/datafile 2020-11-16T21:12:11.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-532 td/osd-backfill-space/datafile 2020-11-16T21:12:11.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-533 td/osd-backfill-space/datafile 2020-11-16T21:12:11.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-533 td/osd-backfill-space/datafile 2020-11-16T21:12:11.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-534 td/osd-backfill-space/datafile 2020-11-16T21:12:11.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-534 td/osd-backfill-space/datafile 2020-11-16T21:12:11.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-535 td/osd-backfill-space/datafile 2020-11-16T21:12:11.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-535 td/osd-backfill-space/datafile 2020-11-16T21:12:11.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-536 td/osd-backfill-space/datafile 2020-11-16T21:12:11.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-536 td/osd-backfill-space/datafile 2020-11-16T21:12:11.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-537 td/osd-backfill-space/datafile 2020-11-16T21:12:11.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-537 td/osd-backfill-space/datafile 2020-11-16T21:12:11.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-538 td/osd-backfill-space/datafile 2020-11-16T21:12:11.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-538 td/osd-backfill-space/datafile 2020-11-16T21:12:11.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-539 td/osd-backfill-space/datafile 2020-11-16T21:12:11.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-539 td/osd-backfill-space/datafile 2020-11-16T21:12:11.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-540 td/osd-backfill-space/datafile 2020-11-16T21:12:11.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:11.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-540 td/osd-backfill-space/datafile 2020-11-16T21:12:11.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:11.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:11.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-541 td/osd-backfill-space/datafile 2020-11-16T21:12:15.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-541 td/osd-backfill-space/datafile 2020-11-16T21:12:15.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-542 td/osd-backfill-space/datafile 2020-11-16T21:12:15.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-542 td/osd-backfill-space/datafile 2020-11-16T21:12:15.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-543 td/osd-backfill-space/datafile 2020-11-16T21:12:15.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-543 td/osd-backfill-space/datafile 2020-11-16T21:12:15.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-544 td/osd-backfill-space/datafile 2020-11-16T21:12:15.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-544 td/osd-backfill-space/datafile 2020-11-16T21:12:15.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-545 td/osd-backfill-space/datafile 2020-11-16T21:12:15.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-545 td/osd-backfill-space/datafile 2020-11-16T21:12:15.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-546 td/osd-backfill-space/datafile 2020-11-16T21:12:15.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-546 td/osd-backfill-space/datafile 2020-11-16T21:12:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-547 td/osd-backfill-space/datafile 2020-11-16T21:12:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-547 td/osd-backfill-space/datafile 2020-11-16T21:12:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-548 td/osd-backfill-space/datafile 2020-11-16T21:12:15.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-548 td/osd-backfill-space/datafile 2020-11-16T21:12:15.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-549 td/osd-backfill-space/datafile 2020-11-16T21:12:15.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-549 td/osd-backfill-space/datafile 2020-11-16T21:12:15.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-550 td/osd-backfill-space/datafile 2020-11-16T21:12:15.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-550 td/osd-backfill-space/datafile 2020-11-16T21:12:15.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-551 td/osd-backfill-space/datafile 2020-11-16T21:12:15.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-551 td/osd-backfill-space/datafile 2020-11-16T21:12:15.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-552 td/osd-backfill-space/datafile 2020-11-16T21:12:15.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-552 td/osd-backfill-space/datafile 2020-11-16T21:12:15.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-553 td/osd-backfill-space/datafile 2020-11-16T21:12:15.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-553 td/osd-backfill-space/datafile 2020-11-16T21:12:15.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-554 td/osd-backfill-space/datafile 2020-11-16T21:12:15.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-554 td/osd-backfill-space/datafile 2020-11-16T21:12:15.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-555 td/osd-backfill-space/datafile 2020-11-16T21:12:15.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-555 td/osd-backfill-space/datafile 2020-11-16T21:12:15.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-556 td/osd-backfill-space/datafile 2020-11-16T21:12:15.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-556 td/osd-backfill-space/datafile 2020-11-16T21:12:15.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-557 td/osd-backfill-space/datafile 2020-11-16T21:12:15.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-557 td/osd-backfill-space/datafile 2020-11-16T21:12:15.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-558 td/osd-backfill-space/datafile 2020-11-16T21:12:15.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-558 td/osd-backfill-space/datafile 2020-11-16T21:12:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-559 td/osd-backfill-space/datafile 2020-11-16T21:12:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-559 td/osd-backfill-space/datafile 2020-11-16T21:12:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-560 td/osd-backfill-space/datafile 2020-11-16T21:12:15.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-560 td/osd-backfill-space/datafile 2020-11-16T21:12:15.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-561 td/osd-backfill-space/datafile 2020-11-16T21:12:15.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-561 td/osd-backfill-space/datafile 2020-11-16T21:12:15.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-562 td/osd-backfill-space/datafile 2020-11-16T21:12:15.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-562 td/osd-backfill-space/datafile 2020-11-16T21:12:15.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.882 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-563 td/osd-backfill-space/datafile 2020-11-16T21:12:15.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-563 td/osd-backfill-space/datafile 2020-11-16T21:12:15.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-564 td/osd-backfill-space/datafile 2020-11-16T21:12:15.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-564 td/osd-backfill-space/datafile 2020-11-16T21:12:15.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-565 td/osd-backfill-space/datafile 2020-11-16T21:12:15.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-565 td/osd-backfill-space/datafile 2020-11-16T21:12:15.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-566 td/osd-backfill-space/datafile 2020-11-16T21:12:15.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-566 td/osd-backfill-space/datafile 2020-11-16T21:12:15.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-567 td/osd-backfill-space/datafile 2020-11-16T21:12:15.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-567 td/osd-backfill-space/datafile 2020-11-16T21:12:15.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-568 td/osd-backfill-space/datafile 2020-11-16T21:12:15.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-568 td/osd-backfill-space/datafile 2020-11-16T21:12:15.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-569 td/osd-backfill-space/datafile 2020-11-16T21:12:15.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-569 td/osd-backfill-space/datafile 2020-11-16T21:12:15.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-570 td/osd-backfill-space/datafile 2020-11-16T21:12:15.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-570 td/osd-backfill-space/datafile 2020-11-16T21:12:15.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-571 td/osd-backfill-space/datafile 2020-11-16T21:12:15.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-571 td/osd-backfill-space/datafile 2020-11-16T21:12:15.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-572 td/osd-backfill-space/datafile 2020-11-16T21:12:15.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-572 td/osd-backfill-space/datafile 2020-11-16T21:12:15.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-573 td/osd-backfill-space/datafile 2020-11-16T21:12:15.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-573 td/osd-backfill-space/datafile 2020-11-16T21:12:15.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-574 td/osd-backfill-space/datafile 2020-11-16T21:12:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-574 td/osd-backfill-space/datafile 2020-11-16T21:12:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-575 td/osd-backfill-space/datafile 2020-11-16T21:12:15.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-575 td/osd-backfill-space/datafile 2020-11-16T21:12:15.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-576 td/osd-backfill-space/datafile 2020-11-16T21:12:15.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-576 td/osd-backfill-space/datafile 2020-11-16T21:12:15.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-577 td/osd-backfill-space/datafile 2020-11-16T21:12:15.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-577 td/osd-backfill-space/datafile 2020-11-16T21:12:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-578 td/osd-backfill-space/datafile 2020-11-16T21:12:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-578 td/osd-backfill-space/datafile 2020-11-16T21:12:15.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-579 td/osd-backfill-space/datafile 2020-11-16T21:12:15.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-579 td/osd-backfill-space/datafile 2020-11-16T21:12:15.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-580 td/osd-backfill-space/datafile 2020-11-16T21:12:15.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-580 td/osd-backfill-space/datafile 2020-11-16T21:12:15.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-581 td/osd-backfill-space/datafile 2020-11-16T21:12:15.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-581 td/osd-backfill-space/datafile 2020-11-16T21:12:15.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-582 td/osd-backfill-space/datafile 2020-11-16T21:12:15.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-582 td/osd-backfill-space/datafile 2020-11-16T21:12:15.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-583 td/osd-backfill-space/datafile 2020-11-16T21:12:15.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-583 td/osd-backfill-space/datafile 2020-11-16T21:12:15.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-584 td/osd-backfill-space/datafile 2020-11-16T21:12:15.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-584 td/osd-backfill-space/datafile 2020-11-16T21:12:15.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-585 td/osd-backfill-space/datafile 2020-11-16T21:12:15.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-585 td/osd-backfill-space/datafile 2020-11-16T21:12:15.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-586 td/osd-backfill-space/datafile 2020-11-16T21:12:15.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-586 td/osd-backfill-space/datafile 2020-11-16T21:12:15.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-587 td/osd-backfill-space/datafile 2020-11-16T21:12:15.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-587 td/osd-backfill-space/datafile 2020-11-16T21:12:15.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-588 td/osd-backfill-space/datafile 2020-11-16T21:12:15.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-588 td/osd-backfill-space/datafile 2020-11-16T21:12:15.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.906 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-589 td/osd-backfill-space/datafile 2020-11-16T21:12:15.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-589 td/osd-backfill-space/datafile 2020-11-16T21:12:15.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-590 td/osd-backfill-space/datafile 2020-11-16T21:12:15.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-590 td/osd-backfill-space/datafile 2020-11-16T21:12:15.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.908 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-591 td/osd-backfill-space/datafile 2020-11-16T21:12:15.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-591 td/osd-backfill-space/datafile 2020-11-16T21:12:15.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.909 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-592 td/osd-backfill-space/datafile 2020-11-16T21:12:15.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-592 td/osd-backfill-space/datafile 2020-11-16T21:12:15.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-593 td/osd-backfill-space/datafile 2020-11-16T21:12:15.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-593 td/osd-backfill-space/datafile 2020-11-16T21:12:15.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-594 td/osd-backfill-space/datafile 2020-11-16T21:12:15.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-594 td/osd-backfill-space/datafile 2020-11-16T21:12:15.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-595 td/osd-backfill-space/datafile 2020-11-16T21:12:15.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-595 td/osd-backfill-space/datafile 2020-11-16T21:12:15.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-596 td/osd-backfill-space/datafile 2020-11-16T21:12:15.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-596 td/osd-backfill-space/datafile 2020-11-16T21:12:15.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-597 td/osd-backfill-space/datafile 2020-11-16T21:12:15.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-597 td/osd-backfill-space/datafile 2020-11-16T21:12:15.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-598 td/osd-backfill-space/datafile 2020-11-16T21:12:15.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-598 td/osd-backfill-space/datafile 2020-11-16T21:12:15.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-599 td/osd-backfill-space/datafile 2020-11-16T21:12:15.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-599 td/osd-backfill-space/datafile 2020-11-16T21:12:15.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2020-11-16T21:12:15.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-600 td/osd-backfill-space/datafile 2020-11-16T21:12:15.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2020-11-16T21:12:15.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-600 td/osd-backfill-space/datafile 2020-11-16T21:12:15.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:508: TEST_backfill_multi_partial: kill_daemon td/osd-backfill-space/osd.2.pid TERM 2020-11-16T21:12:15.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/osd-backfill-space/osd.2.pid 2020-11-16T21:12:15.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=199659 2020-11-16T21:12:15.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2020-11-16T21:12:15.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2020-11-16T21:12:15.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2020-11-16T21:12:15.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2020-11-16T21:12:15.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 199659 2020-11-16T21:12:15.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2020-11-16T21:12:15.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2020-11-16T21:12:15.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2020-11-16T21:12:15.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2020-11-16T21:12:15.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 199659 2020-11-16T21:12:15.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2020-11-16T21:12:15.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2020-11-16T21:12:15.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2020-11-16T21:12:15.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:509: TEST_backfill_multi_partial: ceph osd out osd.2 2020-11-16T21:12:15.919 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already out. 2020-11-16T21:12:15.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:511: TEST_backfill_multi_partial: activate_osd td/osd-backfill-space 1 2020-11-16T21:12:15.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:15.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T21:12:15.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T21:12:15.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T21:12:15.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/1 2020-11-16T21:12:15.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:15.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:15.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:15.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:15.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2020-11-16T21:12:15.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2020-11-16T21:12:15.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:15.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T21:12:15.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:15.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T21:12:15.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:15.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:15.922 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:15.922 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:15.922 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:15.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:15.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:15.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:15.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:15.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:15.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:15.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:15.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:15.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T21:12:15.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T21:12:15.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/1 2020-11-16T21:12:15.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T21:12:15.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:15.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/1/whoami 2020-11-16T21:12:15.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T21:12:15.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T21:12:15.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T21:12:15.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T21:12:15.925 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:48:52.281+0000 7f70bdcbdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:15.925 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:48:52.281+0000 7f70bdcbdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:15.925 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:48:52.285+0000 7f70bdcbdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:15.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T21:12:15.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:15.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:12:15.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:15.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:15.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:15.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:15.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:15.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:12:15.927 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:48:52.821+0000 7f70bdcbdd80 -1 Falling back to public interface 2020-11-16T21:12:15.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:15.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:15.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:15.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:15.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:15.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:12:15.928 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:48:54.253+0000 7f70bdcbdd80 -1 osd.1 44 log_to_monitors {default=true} 2020-11-16T21:12:15.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:15.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:15.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:15.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:512: TEST_backfill_multi_partial: ceph osd in osd.1 2020-11-16T21:12:15.929 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already in. 2020-11-16T21:12:15.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:513: TEST_backfill_multi_partial: sleep 30 2020-11-16T21:12:15.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:515: TEST_backfill_multi_partial: wait_for_not_backfilling 240 2020-11-16T21:12:15.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2020-11-16T21:12:15.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2020-11-16T21:12:15.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2020-11-16T21:12:15.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:12:15.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:12:15.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:12:15.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2020-11-16T21:12:15.930 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:12:15.930 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:12:15.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:12:15.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:12:15.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:12:15.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:12:15.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:12:15.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:12:15.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:12:15.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:12:15.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:12:15.932 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:12:15.932 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T21:12:15.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:12:15.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:15.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:12:15.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836579 2020-11-16T21:12:15.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836579 2020-11-16T21:12:15.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836579' 2020-11-16T21:12:15.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:15.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:12:15.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397513 2020-11-16T21:12:15.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397513 2020-11-16T21:12:15.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836579 1-210453397513' 2020-11-16T21:12:15.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:15.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:12:15.934 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.2 2020-11-16T21:12:15.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T21:12:15.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T21:12:15.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T21:12:15.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:15.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836579 2020-11-16T21:12:15.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:15.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:12:15.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836579 2020-11-16T21:12:15.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:15.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836579 2020-11-16T21:12:15.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2020-11-16T21:12:15.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:12:15.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836577 -lt 21474836579 2020-11-16T21:12:15.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:12:15.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:12:15.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:12:15.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836580 -lt 21474836579 2020-11-16T21:12:15.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:15.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397513 2020-11-16T21:12:15.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:15.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:12:15.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397513 2020-11-16T21:12:15.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:15.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397513 2020-11-16T21:12:15.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397513' 2020-11-16T21:12:15.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:12:15.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397513 -lt 210453397513 2020-11-16T21:12:15.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:12:15.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:12:15.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:12:15.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 3 == 0 2020-11-16T21:12:15.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:12:15.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:12:15.940 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:12:15.940 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:12:15.940 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:12:15.940 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:12:15.940 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:12:15.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:12:15.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:12:15.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' -1 2020-11-16T21:12:15.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:68: wait_for_not_state: loop=0 2020-11-16T21:12:15.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:69: wait_for_not_state: num_in_state=1 2020-11-16T21:12:15.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 5 2020-11-16T21:12:15.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:12:15.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:12:15.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:12:15.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:12:15.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:12:15.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:12:15.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:12:15.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:12:15.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:12:15.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:12:15.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:12:15.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 1 >= 17 )) 2020-11-16T21:12:15.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 10 2020-11-16T21:12:15.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:12:15.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:12:15.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:12:15.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:12:15.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:12:15.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:12:15.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:12:15.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:12:15.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:12:15.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:12:15.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:12:15.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 2 >= 17 )) 2020-11-16T21:12:15.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:12:15.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:12:15.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:12:15.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:12:15.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:12:15.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:12:15.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:12:15.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:12:15.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:12:15.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:12:15.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:12:15.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:12:15.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 3 >= 17 )) 2020-11-16T21:12:15.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:12:15.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:12:15.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:12:15.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:12:15.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:12:15.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:12:15.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:12:15.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:12:15.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:12:15.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:12:15.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:12:15.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:12:15.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 4 >= 17 )) 2020-11-16T21:12:15.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:12:15.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:12:15.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:12:15.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:12:15.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:12:15.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:12:15.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:12:15.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:12:15.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:12:15.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:12:15.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:12:15.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:12:15.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:12:15.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:516: TEST_backfill_multi_partial: wait_for_not_activating 60 2020-11-16T21:12:15.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2020-11-16T21:12:15.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2020-11-16T21:12:15.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2020-11-16T21:12:15.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:12:15.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:12:15.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:12:15.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2020-11-16T21:12:15.952 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:12:15.952 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:12:15.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:12:15.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:12:15.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:12:15.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:12:15.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:12:15.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:12:15.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:12:15.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:12:15.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:12:15.953 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:12:15.954 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T21:12:15.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:12:15.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:15.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:12:15.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836594 2020-11-16T21:12:15.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836594 2020-11-16T21:12:15.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836594' 2020-11-16T21:12:15.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:15.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:12:15.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397528 2020-11-16T21:12:15.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397528 2020-11-16T21:12:15.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836594 1-210453397528' 2020-11-16T21:12:15.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:15.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:12:15.956 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.2 2020-11-16T21:12:15.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T21:12:15.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T21:12:15.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T21:12:15.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:15.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836594 2020-11-16T21:12:15.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:15.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:12:15.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836594 2020-11-16T21:12:15.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:15.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836594 2020-11-16T21:12:15.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836594' 2020-11-16T21:12:15.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:12:15.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836592 -lt 21474836594 2020-11-16T21:12:15.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:12:15.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:12:15.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:12:15.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836595 -lt 21474836594 2020-11-16T21:12:15.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:15.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:15.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397528 2020-11-16T21:12:15.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:12:15.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397528 2020-11-16T21:12:15.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:15.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397528 2020-11-16T21:12:15.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397528' 2020-11-16T21:12:15.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:12:15.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397528 -lt 210453397528 2020-11-16T21:12:15.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:12:15.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:12:15.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:12:15.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 3 == 0 2020-11-16T21:12:15.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:12:15.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2020-11-16T21:12:15.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2020-11-16T21:12:15.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:12:15.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2020-11-16T21:12:15.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:12:15.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2020-11-16T21:12:15.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:12:15.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:12:15.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:12:15.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:12:15.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:518: TEST_backfill_multi_partial: flush_pg_stats 2020-11-16T21:12:15.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:12:15.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:12:15.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:12:15.962 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:12:15.963 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T21:12:15.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:12:15.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:15.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:12:15.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836597 2020-11-16T21:12:15.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836597 2020-11-16T21:12:15.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836597' 2020-11-16T21:12:15.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:15.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:12:15.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397531 2020-11-16T21:12:15.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397531 2020-11-16T21:12:15.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836597 1-210453397531' 2020-11-16T21:12:15.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:15.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:12:15.965 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.2 2020-11-16T21:12:15.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T21:12:15.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T21:12:15.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T21:12:15.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:15.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836597 2020-11-16T21:12:15.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:15.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:12:15.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836597 2020-11-16T21:12:15.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:15.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836597 2020-11-16T21:12:15.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2020-11-16T21:12:15.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:12:15.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836595 -lt 21474836597 2020-11-16T21:12:15.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:12:15.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:12:15.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:12:15.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836598 -lt 21474836597 2020-11-16T21:12:15.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:15.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397531 2020-11-16T21:12:15.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:15.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:12:15.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397531 2020-11-16T21:12:15.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:15.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397531 2020-11-16T21:12:15.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397531' 2020-11-16T21:12:15.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:12:15.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397531 -lt 210453397531 2020-11-16T21:12:15.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:519: TEST_backfill_multi_partial: ceph pg dump pgs 2020-11-16T21:12:15.969 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:12:15.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:521: TEST_backfill_multi_partial: ERRORS=0 2020-11-16T21:12:15.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:522: TEST_backfill_multi_partial: get_num_in_state backfill_toofull 2020-11-16T21:12:15.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfill_toofull 2020-11-16T21:12:15.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:12:15.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfill_toofull"))' 2020-11-16T21:12:15.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:12:15.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfill_toofull"))] | length' 2020-11-16T21:12:15.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:522: TEST_backfill_multi_partial: '[' 1 '!=' 1 ']' 2020-11-16T21:12:15.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:528: TEST_backfill_multi_partial: get_num_in_state active+clean 2020-11-16T21:12:15.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=active+clean 2020-11-16T21:12:15.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:12:15.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("active+clean"))' 2020-11-16T21:12:15.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:12:15.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("active+clean"))] | length' 2020-11-16T21:12:15.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:528: TEST_backfill_multi_partial: '[' 2 '!=' 2 ']' 2020-11-16T21:12:15.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:534: TEST_backfill_multi_partial: '[' 0 '!=' 0 ']' 2020-11-16T21:12:15.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:539: TEST_backfill_multi_partial: delete_pool fillpool 2020-11-16T21:12:15.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=fillpool 2020-11-16T21:12:15.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete fillpool fillpool --yes-i-really-really-mean-it 2020-11-16T21:12:15.972 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fillpool' does not exist 2020-11-16T21:12:15.972 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:540: TEST_backfill_multi_partial: seq 1 2 2020-11-16T21:12:15.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:540: TEST_backfill_multi_partial: for i in $(seq 1 $pools) 2020-11-16T21:12:15.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:542: TEST_backfill_multi_partial: delete_pool test1 2020-11-16T21:12:15.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2020-11-16T21:12:15.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-11-16T21:12:15.972 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' does not exist 2020-11-16T21:12:15.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:540: TEST_backfill_multi_partial: for i in $(seq 1 $pools) 2020-11-16T21:12:15.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:542: TEST_backfill_multi_partial: delete_pool test2 2020-11-16T21:12:15.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2020-11-16T21:12:15.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-11-16T21:12:15.973 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' does not exist 2020-11-16T21:12:15.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:544: TEST_backfill_multi_partial: kill_daemons td/osd-backfill-space 2020-11-16T21:12:15.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:12:15.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:12:15.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:12:15.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:12:15.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:12:15.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:12:15.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:545: TEST_backfill_multi_partial: grep -q 'num_bytes mismatch' td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log 2020-11-16T21:12:15.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2020-11-16T21:12:15.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:12:15.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:12:15.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:12:15.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:12:15.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:12:15.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:12:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:12:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:12:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:12:15.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:12:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:12:15.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:12:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:12:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:12:15.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:12:15.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:12:15.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:12:15.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:12:15.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:12:15.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:12:15.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:12:15.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:12:15.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:12:15.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:15.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:15.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:12:15.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:12:15.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:12:15.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2020-11-16T21:12:15.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2020-11-16T21:12:15.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2020-11-16T21:12:15.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2020-11-16T21:12:15.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:12:15.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:12:15.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:12:15.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:12:15.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:12:15.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:12:15.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:12:15.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:12:15.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:12:15.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:12:15.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:12:15.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:12:15.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:12:15.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:12:15.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:12:15.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:12:15.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:12:15.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:12:15.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:12:15.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:12:15.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:12:15.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:12:15.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:12:15.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:15.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:15.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:12:15.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:12:15.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:12:15.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2020-11-16T21:12:15.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:12:15.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:15.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:15.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137323 2020-11-16T21:12:15.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:12:15.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:12:15.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:12:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2020-11-16T21:12:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_backfill_test_multi td/osd-backfill-space 2020-11-16T21:12:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:191: TEST_backfill_test_multi: local dir=td/osd-backfill-space 2020-11-16T21:12:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:192: TEST_backfill_test_multi: local pools=8 2020-11-16T21:12:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:193: TEST_backfill_test_multi: local OSDS=20 2020-11-16T21:12:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:195: TEST_backfill_test_multi: run_mon td/osd-backfill-space a 2020-11-16T21:12:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2020-11-16T21:12:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:12:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:12:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:12:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2020-11-16T21:12:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2020-11-16T21:12:15.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:12:15.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:15.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:15.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:12:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:12:15.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:12:15.988 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:12:15.988 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:12:15.988 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:12:15.988 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:12:15.988 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:12:15.988 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:12:15.988 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:12:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:12:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:12:15.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:12:15.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get fsid 2020-11-16T21:12:15.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:12:15.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:12:15.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:12:15.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:12:15.990 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:12:15.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:12:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:12:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:12:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:12:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:12:15.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:12:15.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get mon_host 2020-11-16T21:12:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:196: TEST_backfill_test_multi: run_mgr td/osd-backfill-space x 2020-11-16T21:12:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2020-11-16T21:12:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:12:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:12:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:12:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2020-11-16T21:12:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:12:15.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:12:15.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:15.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:15.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:15.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:12:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:12:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:197: TEST_backfill_test_multi: export CEPH_ARGS 2020-11-16T21:12:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: expr 20 - 1 2020-11-16T21:12:15.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: seq 0 19 2020-11-16T21:12:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 0 2020-11-16T21:12:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:12:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2020-11-16T21:12:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2020-11-16T21:12:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2020-11-16T21:12:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:15.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:15.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:15.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:15.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2020-11-16T21:12:16.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7ee53c15-f82b-4104-add3-8389cc45af48 2020-11-16T21:12:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 7ee53c15-f82b-4104-add3-8389cc45af48' 2020-11-16T21:12:16.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAj5rJf/9VKChAA8XYLtTgP3gOwjAziTl/qJA== 2020-11-16T21:12:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAj5rJf/9VKChAA8XYLtTgP3gOwjAziTl/qJA=="}' 2020-11-16T21:12:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7ee53c15-f82b-4104-add3-8389cc45af48 -i td/osd-backfill-space/0/new.json 2020-11-16T21:12:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2020-11-16T21:12:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAj5rJf/9VKChAA8XYLtTgP3gOwjAziTl/qJA== --osd-uuid 7ee53c15-f82b-4104-add3-8389cc45af48 2020-11-16T21:12:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:43.551+0000 7f896a51ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:43.555+0000 7f896a51ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:43.555+0000 7f896a51ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:43.559+0000 7f896a51ad80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:12:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:43.559+0000 7f896a51ad80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:12:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:43.559+0000 7f896a51ad80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:12:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:43.611+0000 7f896a51ad80 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2020-11-16T21:12:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2020-11-16T21:12:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:12:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:12:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:48.499+0000 7fc46f303d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:48.499+0000 7fc46f303d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:48.503+0000 7fc46f303d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:12:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:12:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:12:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:49.039+0000 7fc46f303d80 -1 Falling back to public interface 2020-11-16T21:12:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.006 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:49.907+0000 7fc46f303d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:12:16.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:16.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:12:16.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:16.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:16.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:16.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:16.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 1 2020-11-16T21:12:16.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:16.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:16.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:12:16.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:16.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2020-11-16T21:12:16.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:16.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:16.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:16.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:16.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2020-11-16T21:12:16.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2020-11-16T21:12:16.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:16.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:16.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:16.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:16.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:16.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:16.011 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:16.011 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:16.011 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:16.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:16.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:16.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:16.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:16.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:16.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:16.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:16.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2020-11-16T21:12:16.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ba099e1a-b7ef-4d93-ad31-b8bbd7f0d1ad 2020-11-16T21:12:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 ba099e1a-b7ef-4d93-ad31-b8bbd7f0d1ad' 2020-11-16T21:12:16.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAq5rJf70wIKRAAHfJAMHAAyyYzStDs0JDPbg== 2020-11-16T21:12:16.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAq5rJf70wIKRAAHfJAMHAAyyYzStDs0JDPbg=="}' 2020-11-16T21:12:16.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ba099e1a-b7ef-4d93-ad31-b8bbd7f0d1ad -i td/osd-backfill-space/1/new.json 2020-11-16T21:12:16.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2020-11-16T21:12:16.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAq5rJf70wIKRAAHfJAMHAAyyYzStDs0JDPbg== --osd-uuid ba099e1a-b7ef-4d93-ad31-b8bbd7f0d1ad 2020-11-16T21:12:16.014 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:51.191+0000 7fece946cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.014 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:51.195+0000 7fece946cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.015 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:51.195+0000 7fece946cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.015 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:51.199+0000 7fece946cd80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:12:16.015 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:51.199+0000 7fece946cd80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:12:16.015 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:51.199+0000 7fece946cd80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:12:16.015 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:51.243+0000 7fece946cd80 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2020-11-16T21:12:16.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2020-11-16T21:12:16.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:12:16.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:12:16.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:56.115+0000 7f1478b90d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:56.119+0000 7f1478b90d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:56.119+0000 7f1478b90d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:12:16.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:12:16.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:12:16.019 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:56.659+0000 7f1478b90d80 -1 Falling back to public interface 2020-11-16T21:12:16.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.019 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:57.351+0000 7f1478b90d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:12:16.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:16.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:12:16.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:16.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:16.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:16.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:16.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 2 2020-11-16T21:12:16.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:16.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:16.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:12:16.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:16.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2020-11-16T21:12:16.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:16.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:16.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:16.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:16.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2020-11-16T21:12:16.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2020-11-16T21:12:16.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:16.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:16.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:16.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:16.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:16.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:16.024 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:16.024 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:16.024 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:16.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:16.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:16.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:16.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:16.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:16.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:16.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:16.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:16.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2020-11-16T21:12:16.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=377dd9af-483f-486e-9c3c-126cfe3237b6 2020-11-16T21:12:16.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 377dd9af-483f-486e-9c3c-126cfe3237b6' 2020-11-16T21:12:16.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAy5rJffyiwFBAAzHDqW3hwYgTmV0q37g4LNQ== 2020-11-16T21:12:16.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAy5rJffyiwFBAAzHDqW3hwYgTmV0q37g4LNQ=="}' 2020-11-16T21:12:16.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 377dd9af-483f-486e-9c3c-126cfe3237b6 -i td/osd-backfill-space/2/new.json 2020-11-16T21:12:16.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2020-11-16T21:12:16.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAy5rJffyiwFBAAzHDqW3hwYgTmV0q37g4LNQ== --osd-uuid 377dd9af-483f-486e-9c3c-126cfe3237b6 2020-11-16T21:12:16.028 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:58.931+0000 7f184b714d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.028 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:58.935+0000 7f184b714d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.028 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:58.935+0000 7f184b714d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.028 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:58.935+0000 7f184b714d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:12:16.028 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:58.935+0000 7f184b714d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:12:16.029 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:58.935+0000 7f184b714d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:12:16.029 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:50:59.015+0000 7f184b714d80 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2020-11-16T21:12:16.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2020-11-16T21:12:16.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:12:16.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:12:16.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.030 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:03.931+0000 7f5712e66d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.030 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:03.935+0000 7f5712e66d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.030 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:03.935+0000 7f5712e66d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:12:16.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:12:16.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:12:16.032 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:04.479+0000 7f5712e66d80 -1 Falling back to public interface 2020-11-16T21:12:16.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.032 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:05.371+0000 7f5712e66d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:12:16.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:16.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:12:16.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:16.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:16.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:16.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:16.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 3 2020-11-16T21:12:16.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:16.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:16.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:12:16.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:16.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/3 2020-11-16T21:12:16.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:16.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:16.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:16.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:16.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/3' 2020-11-16T21:12:16.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/3/journal' 2020-11-16T21:12:16.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:16.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:16.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:16.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:16.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:16.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:16.036 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:16.037 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:16.037 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:16.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:16.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:16.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:16.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:16.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:16.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:16.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:16.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:16.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/3 2020-11-16T21:12:16.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=05ac7116-ae31-4b04-b82c-40e5dfeb90af 2020-11-16T21:12:16.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 05ac7116-ae31-4b04-b82c-40e5dfeb90af' 2020-11-16T21:12:16.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA65rJfsnlYBxAA+2i7iNbJkTI2LNYbUlog5g== 2020-11-16T21:12:16.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA65rJfsnlYBxAA+2i7iNbJkTI2LNYbUlog5g=="}' 2020-11-16T21:12:16.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 05ac7116-ae31-4b04-b82c-40e5dfeb90af -i td/osd-backfill-space/3/new.json 2020-11-16T21:12:16.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/3/new.json 2020-11-16T21:12:16.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA65rJfsnlYBxAA+2i7iNbJkTI2LNYbUlog5g== --osd-uuid 05ac7116-ae31-4b04-b82c-40e5dfeb90af 2020-11-16T21:12:16.040 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:06.759+0000 7fcceae15d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.040 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:06.763+0000 7fcceae15d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.040 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:06.767+0000 7fcceae15d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.040 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:06.767+0000 7fcceae15d80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:12:16.041 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:06.767+0000 7fcceae15d80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:12:16.041 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:06.767+0000 7fcceae15d80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:12:16.041 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:06.811+0000 7fcceae15d80 -1 bluestore(td/osd-backfill-space/3) _read_fsid unparsable uuid 2020-11-16T21:12:16.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/3/keyring 2020-11-16T21:12:16.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:12:16.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:12:16.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.042 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:11.655+0000 7fdd65dd1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.042 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:11.659+0000 7fdd65dd1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.043 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:11.659+0000 7fdd65dd1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:12:16.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:12:16.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:12:16.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.044 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:12.207+0000 7fdd65dd1d80 -1 Falling back to public interface 2020-11-16T21:12:16.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.044 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:13.095+0000 7fdd65dd1d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:12:16.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:12:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:16.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:16.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:16.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 4 2020-11-16T21:12:16.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:16.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:16.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:12:16.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:16.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/4 2020-11-16T21:12:16.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:16.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:16.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:16.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:16.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/4' 2020-11-16T21:12:16.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/4/journal' 2020-11-16T21:12:16.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:16.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:16.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:16.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:16.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:16.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:16.048 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:16.048 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:16.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:16.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:16.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:16.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:16.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:16.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/4 2020-11-16T21:12:16.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a0bf3665-d2ef-4edc-8b39-65d7deaf24ea 2020-11-16T21:12:16.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 a0bf3665-d2ef-4edc-8b39-65d7deaf24ea' 2020-11-16T21:12:16.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBB5rJf6x3dMRAAqZ3I0AwQ49Pge+kKhczMmg== 2020-11-16T21:12:16.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBB5rJf6x3dMRAAqZ3I0AwQ49Pge+kKhczMmg=="}' 2020-11-16T21:12:16.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a0bf3665-d2ef-4edc-8b39-65d7deaf24ea -i td/osd-backfill-space/4/new.json 2020-11-16T21:12:16.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/4/new.json 2020-11-16T21:12:16.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBB5rJf6x3dMRAAqZ3I0AwQ49Pge+kKhczMmg== --osd-uuid a0bf3665-d2ef-4edc-8b39-65d7deaf24ea 2020-11-16T21:12:16.052 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:14.399+0000 7ff79395dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.052 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:14.399+0000 7ff79395dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.052 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:14.403+0000 7ff79395dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.052 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:14.403+0000 7ff79395dd80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:12:16.052 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:14.403+0000 7ff79395dd80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:12:16.052 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:14.403+0000 7ff79395dd80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:12:16.052 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:14.471+0000 7ff79395dd80 -1 bluestore(td/osd-backfill-space/4) _read_fsid unparsable uuid 2020-11-16T21:12:16.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/4/keyring 2020-11-16T21:12:16.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:12:16.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:12:16.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.054 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:19.623+0000 7fd93ef9dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.054 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:19.627+0000 7fd93ef9dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:19.627+0000 7fd93ef9dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:12:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:12:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:12:16.056 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:20.159+0000 7fd93ef9dd80 -1 Falling back to public interface 2020-11-16T21:12:16.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.056 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:20.783+0000 7fd93ef9dd80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:12:16.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:12:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:16.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 5 2020-11-16T21:12:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T21:12:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/5 2020-11-16T21:12:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/5' 2020-11-16T21:12:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/5/journal' 2020-11-16T21:12:16.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:16.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:16.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:16.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:16.060 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:16.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/5 2020-11-16T21:12:16.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d4e12b60-777c-4297-91cf-53e605d2d67f 2020-11-16T21:12:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 d4e12b60-777c-4297-91cf-53e605d2d67f' 2020-11-16T21:12:16.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBJ5rJfOOQiMBAAqxh+bueeKA5HzNoH5n03gA== 2020-11-16T21:12:16.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBJ5rJfOOQiMBAAqxh+bueeKA5HzNoH5n03gA=="}' 2020-11-16T21:12:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d4e12b60-777c-4297-91cf-53e605d2d67f -i td/osd-backfill-space/5/new.json 2020-11-16T21:12:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/5/new.json 2020-11-16T21:12:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBJ5rJfOOQiMBAAqxh+bueeKA5HzNoH5n03gA== --osd-uuid d4e12b60-777c-4297-91cf-53e605d2d67f 2020-11-16T21:12:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:22.430+0000 7fde3fbced80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:22.430+0000 7fde3fbced80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:22.434+0000 7fde3fbced80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:22.434+0000 7fde3fbced80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:12:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:22.434+0000 7fde3fbced80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:12:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:22.434+0000 7fde3fbced80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:12:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:22.482+0000 7fde3fbced80 -1 bluestore(td/osd-backfill-space/5) _read_fsid unparsable uuid 2020-11-16T21:12:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/5/keyring 2020-11-16T21:12:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T21:12:16.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T21:12:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:27.378+0000 7fc92af9bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.066 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:27.382+0000 7fc92af9bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:27.386+0000 7fc92af9bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T21:12:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T21:12:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:12:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:27.922+0000 7fc92af9bd80 -1 Falling back to public interface 2020-11-16T21:12:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:28.758+0000 7fc92af9bd80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T21:12:16.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:12:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 6 2020-11-16T21:12:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=6 2020-11-16T21:12:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/6 2020-11-16T21:12:16.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:16.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:16.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:16.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:16.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/6' 2020-11-16T21:12:16.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/6/journal' 2020-11-16T21:12:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:16.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:16.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:16.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:16.073 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:16.073 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:16.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:16.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:16.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:16.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:16.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:16.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:16.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:16.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:16.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/6 2020-11-16T21:12:16.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5152a9e7-7900-4d0d-818a-0323f6092dea 2020-11-16T21:12:16.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd6 5152a9e7-7900-4d0d-818a-0323f6092dea' 2020-11-16T21:12:16.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBR5rJfnc6/HxAAog9HRAukn+GCL/Xv3XvCoQ== 2020-11-16T21:12:16.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBR5rJfnc6/HxAAog9HRAukn+GCL/Xv3XvCoQ=="}' 2020-11-16T21:12:16.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5152a9e7-7900-4d0d-818a-0323f6092dea -i td/osd-backfill-space/6/new.json 2020-11-16T21:12:16.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/6/new.json 2020-11-16T21:12:16.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 6 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/6 --osd-journal=td/osd-backfill-space/6/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBR5rJfnc6/HxAAog9HRAukn+GCL/Xv3XvCoQ== --osd-uuid 5152a9e7-7900-4d0d-818a-0323f6092dea 2020-11-16T21:12:16.076 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:30.154+0000 7f3c64017d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.076 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:30.158+0000 7f3c64017d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.076 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:30.162+0000 7f3c64017d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.077 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:30.162+0000 7f3c64017d80 -1 bluestore(td/osd-backfill-space/6/block) _read_bdev_label failed to open td/osd-backfill-space/6/block: (2) No such file or directory 2020-11-16T21:12:16.077 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:30.162+0000 7f3c64017d80 -1 bluestore(td/osd-backfill-space/6/block) _read_bdev_label failed to open td/osd-backfill-space/6/block: (2) No such file or directory 2020-11-16T21:12:16.077 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:30.162+0000 7f3c64017d80 -1 bluestore(td/osd-backfill-space/6/block) _read_bdev_label failed to open td/osd-backfill-space/6/block: (2) No such file or directory 2020-11-16T21:12:16.077 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:30.210+0000 7f3c64017d80 -1 bluestore(td/osd-backfill-space/6) _read_fsid unparsable uuid 2020-11-16T21:12:16.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/6/keyring 2020-11-16T21:12:16.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd6 key to auth repository 2020-11-16T21:12:16.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.6 2020-11-16T21:12:16.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 6 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/6 --osd-journal=td/osd-backfill-space/6/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.078 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:35.286+0000 7f0a576f5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.078 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:35.290+0000 7f0a576f5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.079 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:35.290+0000 7f0a576f5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 6 2020-11-16T21:12:16.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=6 2020-11-16T21:12:16.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-11-16T21:12:16.080 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:35.838+0000 7f0a576f5d80 -1 Falling back to public interface 2020-11-16T21:12:16.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.080 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:36.690+0000 7f0a576f5d80 -1 osd.6 0 log_to_monitors {default=true} 2020-11-16T21:12:16.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:16.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-11-16T21:12:16.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:16.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:16.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:16.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:16.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 7 2020-11-16T21:12:16.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:16.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:16.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=7 2020-11-16T21:12:16.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:16.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/7 2020-11-16T21:12:16.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:16.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:16.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:16.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:16.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/7' 2020-11-16T21:12:16.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/7/journal' 2020-11-16T21:12:16.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:16.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:16.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:16.084 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:16.084 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:16.084 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:16.085 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:16.085 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:16.085 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:16.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:16.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:16.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:16.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:16.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:16.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:16.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:16.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:16.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/7 2020-11-16T21:12:16.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7051b692-ea86-4f5b-abaa-d8fb1fbe02be 2020-11-16T21:12:16.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd7 7051b692-ea86-4f5b-abaa-d8fb1fbe02be' 2020-11-16T21:12:16.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBZ5rJfTaVdHhAAP2lr4BYAqyd2nFRAd1sBgg== 2020-11-16T21:12:16.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBZ5rJfTaVdHhAAP2lr4BYAqyd2nFRAd1sBgg=="}' 2020-11-16T21:12:16.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7051b692-ea86-4f5b-abaa-d8fb1fbe02be -i td/osd-backfill-space/7/new.json 2020-11-16T21:12:16.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/7/new.json 2020-11-16T21:12:16.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 7 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/7 --osd-journal=td/osd-backfill-space/7/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBZ5rJfTaVdHhAAP2lr4BYAqyd2nFRAd1sBgg== --osd-uuid 7051b692-ea86-4f5b-abaa-d8fb1fbe02be 2020-11-16T21:12:16.088 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:38.202+0000 7f97b7e3ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.088 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:38.206+0000 7f97b7e3ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.088 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:38.206+0000 7f97b7e3ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.089 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:38.206+0000 7f97b7e3ed80 -1 bluestore(td/osd-backfill-space/7/block) _read_bdev_label failed to open td/osd-backfill-space/7/block: (2) No such file or directory 2020-11-16T21:12:16.089 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:38.206+0000 7f97b7e3ed80 -1 bluestore(td/osd-backfill-space/7/block) _read_bdev_label failed to open td/osd-backfill-space/7/block: (2) No such file or directory 2020-11-16T21:12:16.089 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:38.206+0000 7f97b7e3ed80 -1 bluestore(td/osd-backfill-space/7/block) _read_bdev_label failed to open td/osd-backfill-space/7/block: (2) No such file or directory 2020-11-16T21:12:16.089 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:38.318+0000 7f97b7e3ed80 -1 bluestore(td/osd-backfill-space/7) _read_fsid unparsable uuid 2020-11-16T21:12:16.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/7/keyring 2020-11-16T21:12:16.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd7 key to auth repository 2020-11-16T21:12:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.7 2020-11-16T21:12:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 7 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/7 --osd-journal=td/osd-backfill-space/7/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.091 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:43.186+0000 7f4539d35d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.091 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:43.186+0000 7f4539d35d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.091 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:43.186+0000 7f4539d35d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 7 2020-11-16T21:12:16.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=7 2020-11-16T21:12:16.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.7 up' 2020-11-16T21:12:16.092 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:43.734+0000 7f4539d35d80 -1 Falling back to public interface 2020-11-16T21:12:16.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.093 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:44.618+0000 7f4539d35d80 -1 osd.7 0 log_to_monitors {default=true} 2020-11-16T21:12:16.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:16.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.7 up' 2020-11-16T21:12:16.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:16.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:16.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:16.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:16.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 8 2020-11-16T21:12:16.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:16.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:16.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=8 2020-11-16T21:12:16.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:16.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/8 2020-11-16T21:12:16.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:16.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:16.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:16.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:16.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/8' 2020-11-16T21:12:16.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/8/journal' 2020-11-16T21:12:16.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:16.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:16.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:16.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:16.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:16.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:16.097 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:16.097 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:16.097 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:16.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:16.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:16.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:16.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:16.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:16.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:16.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:16.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:16.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/8 2020-11-16T21:12:16.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e22a2b02-f4e5-42a0-8724-23fd0b5b7904 2020-11-16T21:12:16.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd8 e22a2b02-f4e5-42a0-8724-23fd0b5b7904' 2020-11-16T21:12:16.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBh5rJfEt68GxAArpGw7exEQUzIcHLgXE9MfA== 2020-11-16T21:12:16.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBh5rJfEt68GxAArpGw7exEQUzIcHLgXE9MfA=="}' 2020-11-16T21:12:16.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e22a2b02-f4e5-42a0-8724-23fd0b5b7904 -i td/osd-backfill-space/8/new.json 2020-11-16T21:12:16.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/8/new.json 2020-11-16T21:12:16.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 8 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/8 --osd-journal=td/osd-backfill-space/8/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBh5rJfEt68GxAArpGw7exEQUzIcHLgXE9MfA== --osd-uuid e22a2b02-f4e5-42a0-8724-23fd0b5b7904 2020-11-16T21:12:16.100 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:46.122+0000 7f84beacfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.101 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:46.126+0000 7f84beacfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.101 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:46.126+0000 7f84beacfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.101 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:46.126+0000 7f84beacfd80 -1 bluestore(td/osd-backfill-space/8/block) _read_bdev_label failed to open td/osd-backfill-space/8/block: (2) No such file or directory 2020-11-16T21:12:16.101 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:46.126+0000 7f84beacfd80 -1 bluestore(td/osd-backfill-space/8/block) _read_bdev_label failed to open td/osd-backfill-space/8/block: (2) No such file or directory 2020-11-16T21:12:16.101 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:46.126+0000 7f84beacfd80 -1 bluestore(td/osd-backfill-space/8/block) _read_bdev_label failed to open td/osd-backfill-space/8/block: (2) No such file or directory 2020-11-16T21:12:16.101 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:46.174+0000 7f84beacfd80 -1 bluestore(td/osd-backfill-space/8) _read_fsid unparsable uuid 2020-11-16T21:12:16.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/8/keyring 2020-11-16T21:12:16.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd8 key to auth repository 2020-11-16T21:12:16.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.8 2020-11-16T21:12:16.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 8 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/8 --osd-journal=td/osd-backfill-space/8/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.103 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:51.550+0000 7f5ca0a6cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.103 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:51.554+0000 7f5ca0a6cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.103 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:51.554+0000 7f5ca0a6cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 8 2020-11-16T21:12:16.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=8 2020-11-16T21:12:16.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.8 up' 2020-11-16T21:12:16.104 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:52.098+0000 7f5ca0a6cd80 -1 Falling back to public interface 2020-11-16T21:12:16.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.105 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:52.694+0000 7f5ca0a6cd80 -1 osd.8 0 log_to_monitors {default=true} 2020-11-16T21:12:16.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:16.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.8 up' 2020-11-16T21:12:16.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:16.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:16.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:16.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:16.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 9 2020-11-16T21:12:16.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:16.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:16.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=9 2020-11-16T21:12:16.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:16.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/9 2020-11-16T21:12:16.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:16.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:16.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:16.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:16.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/9' 2020-11-16T21:12:16.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/9/journal' 2020-11-16T21:12:16.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:16.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:16.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:16.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:16.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:16.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:16.109 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:16.109 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:16.109 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:16.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:16.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:16.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:16.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:16.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:16.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:16.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:16.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:16.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/9 2020-11-16T21:12:16.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=74a825de-57e6-4fee-9e1a-ea3010a9016b 2020-11-16T21:12:16.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd9 74a825de-57e6-4fee-9e1a-ea3010a9016b' 2020-11-16T21:12:16.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBp5rJfg9UiLBAAVreMtc5QjkEsaZ3u1N+Czw== 2020-11-16T21:12:16.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBp5rJfg9UiLBAAVreMtc5QjkEsaZ3u1N+Czw=="}' 2020-11-16T21:12:16.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 74a825de-57e6-4fee-9e1a-ea3010a9016b -i td/osd-backfill-space/9/new.json 2020-11-16T21:12:16.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/9/new.json 2020-11-16T21:12:16.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 9 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/9 --osd-journal=td/osd-backfill-space/9/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBp5rJfg9UiLBAAVreMtc5QjkEsaZ3u1N+Czw== --osd-uuid 74a825de-57e6-4fee-9e1a-ea3010a9016b 2020-11-16T21:12:16.112 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:54.326+0000 7f3085a41d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.112 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:54.326+0000 7f3085a41d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.113 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:54.330+0000 7f3085a41d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.113 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:54.330+0000 7f3085a41d80 -1 bluestore(td/osd-backfill-space/9/block) _read_bdev_label failed to open td/osd-backfill-space/9/block: (2) No such file or directory 2020-11-16T21:12:16.113 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:54.330+0000 7f3085a41d80 -1 bluestore(td/osd-backfill-space/9/block) _read_bdev_label failed to open td/osd-backfill-space/9/block: (2) No such file or directory 2020-11-16T21:12:16.113 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:54.330+0000 7f3085a41d80 -1 bluestore(td/osd-backfill-space/9/block) _read_bdev_label failed to open td/osd-backfill-space/9/block: (2) No such file or directory 2020-11-16T21:12:16.113 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:54.378+0000 7f3085a41d80 -1 bluestore(td/osd-backfill-space/9) _read_fsid unparsable uuid 2020-11-16T21:12:16.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/9/keyring 2020-11-16T21:12:16.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd9 key to auth repository 2020-11-16T21:12:16.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.9 2020-11-16T21:12:16.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 9 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/9 --osd-journal=td/osd-backfill-space/9/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.115 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:59.802+0000 7fd64b4e7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.115 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:59.806+0000 7fd64b4e7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.115 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:51:59.806+0000 7fd64b4e7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 9 2020-11-16T21:12:16.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=9 2020-11-16T21:12:16.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.9 up' 2020-11-16T21:12:16.117 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:00.362+0000 7fd64b4e7d80 -1 Falling back to public interface 2020-11-16T21:12:16.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.117 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:01.210+0000 7fd64b4e7d80 -1 osd.9 0 log_to_monitors {default=true} 2020-11-16T21:12:16.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:16.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.9 up' 2020-11-16T21:12:16.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:12:16.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.9 up' 2020-11-16T21:12:16.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:16.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:16.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:16.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:16.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 10 2020-11-16T21:12:16.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:16.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:16.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=10 2020-11-16T21:12:16.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:16.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/10 2020-11-16T21:12:16.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:16.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:16.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:16.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:16.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/10' 2020-11-16T21:12:16.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/10/journal' 2020-11-16T21:12:16.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:16.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:16.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:16.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:16.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:16.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:16.122 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:16.122 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:16.122 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:16.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:16.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:16.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:16.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:16.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:16.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:16.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:16.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:16.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/10 2020-11-16T21:12:16.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e368b490-ada9-4609-a55a-1d53fd53353e 2020-11-16T21:12:16.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd10 e368b490-ada9-4609-a55a-1d53fd53353e' 2020-11-16T21:12:16.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBz5rJfWG/1FxAABTeJQrEtjg8FU86JGsL3MQ== 2020-11-16T21:12:16.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBz5rJfWG/1FxAABTeJQrEtjg8FU86JGsL3MQ=="}' 2020-11-16T21:12:16.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e368b490-ada9-4609-a55a-1d53fd53353e -i td/osd-backfill-space/10/new.json 2020-11-16T21:12:16.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/10/new.json 2020-11-16T21:12:16.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 10 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/10 --osd-journal=td/osd-backfill-space/10/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBz5rJfWG/1FxAABTeJQrEtjg8FU86JGsL3MQ== --osd-uuid e368b490-ada9-4609-a55a-1d53fd53353e 2020-11-16T21:12:16.126 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:04.038+0000 7f8ac0b3fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.127 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:04.038+0000 7f8ac0b3fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.127 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:04.038+0000 7f8ac0b3fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.127 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:04.042+0000 7f8ac0b3fd80 -1 bluestore(td/osd-backfill-space/10/block) _read_bdev_label failed to open td/osd-backfill-space/10/block: (2) No such file or directory 2020-11-16T21:12:16.127 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:04.042+0000 7f8ac0b3fd80 -1 bluestore(td/osd-backfill-space/10/block) _read_bdev_label failed to open td/osd-backfill-space/10/block: (2) No such file or directory 2020-11-16T21:12:16.127 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:04.042+0000 7f8ac0b3fd80 -1 bluestore(td/osd-backfill-space/10/block) _read_bdev_label failed to open td/osd-backfill-space/10/block: (2) No such file or directory 2020-11-16T21:12:16.127 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:04.122+0000 7f8ac0b3fd80 -1 bluestore(td/osd-backfill-space/10) _read_fsid unparsable uuid 2020-11-16T21:12:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/10/keyring 2020-11-16T21:12:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd10 key to auth repository 2020-11-16T21:12:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.10 2020-11-16T21:12:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 10 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/10 --osd-journal=td/osd-backfill-space/10/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.129 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:09.438+0000 7fbf7a0c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.129 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:09.442+0000 7fbf7a0c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.129 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:09.442+0000 7fbf7a0c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 10 2020-11-16T21:12:16.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=10 2020-11-16T21:12:16.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.10 up' 2020-11-16T21:12:16.131 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:09.978+0000 7fbf7a0c0d80 -1 Falling back to public interface 2020-11-16T21:12:16.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.131 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:10.850+0000 7fbf7a0c0d80 -1 osd.10 0 log_to_monitors {default=true} 2020-11-16T21:12:16.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:16.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.10 up' 2020-11-16T21:12:16.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:16.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:16.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:12:16.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.10 up' 2020-11-16T21:12:16.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:16.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:16.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:16.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:16.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 11 2020-11-16T21:12:16.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:16.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:16.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=11 2020-11-16T21:12:16.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:16.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/11 2020-11-16T21:12:16.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:16.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:16.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:16.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:16.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/11' 2020-11-16T21:12:16.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/11/journal' 2020-11-16T21:12:16.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:16.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:16.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:16.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:16.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:16.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:16.137 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:16.137 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:16.137 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:16.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:16.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:16.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:16.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:16.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:16.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:16.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:16.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:16.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:16.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:16.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/11 2020-11-16T21:12:16.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:16.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=005d10e5-1349-480c-a878-cef198c556d6 2020-11-16T21:12:16.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd11 005d10e5-1349-480c-a878-cef198c556d6' 2020-11-16T21:12:16.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB95rJfIpGEBhAASS+HlOCeh2dplHIM9QfVvQ== 2020-11-16T21:12:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB95rJfIpGEBhAASS+HlOCeh2dplHIM9QfVvQ=="}' 2020-11-16T21:12:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 005d10e5-1349-480c-a878-cef198c556d6 -i td/osd-backfill-space/11/new.json 2020-11-16T21:12:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/11/new.json 2020-11-16T21:12:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 11 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/11 --osd-journal=td/osd-backfill-space/11/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB95rJfIpGEBhAASS+HlOCeh2dplHIM9QfVvQ== --osd-uuid 005d10e5-1349-480c-a878-cef198c556d6 2020-11-16T21:12:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:13.645+0000 7fdd0d442d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:13.649+0000 7fdd0d442d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:13.649+0000 7fdd0d442d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:13.649+0000 7fdd0d442d80 -1 bluestore(td/osd-backfill-space/11/block) _read_bdev_label failed to open td/osd-backfill-space/11/block: (2) No such file or directory 2020-11-16T21:12:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:13.649+0000 7fdd0d442d80 -1 bluestore(td/osd-backfill-space/11/block) _read_bdev_label failed to open td/osd-backfill-space/11/block: (2) No such file or directory 2020-11-16T21:12:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:13.649+0000 7fdd0d442d80 -1 bluestore(td/osd-backfill-space/11/block) _read_bdev_label failed to open td/osd-backfill-space/11/block: (2) No such file or directory 2020-11-16T21:12:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:13.697+0000 7fdd0d442d80 -1 bluestore(td/osd-backfill-space/11) _read_fsid unparsable uuid 2020-11-16T21:12:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/11/keyring 2020-11-16T21:12:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd11 key to auth repository 2020-11-16T21:12:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/11/keyring auth add osd.11 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.11 2020-11-16T21:12:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 11 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/11 --osd-journal=td/osd-backfill-space/11/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:18.989+0000 7f1c0c736d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:18.989+0000 7f1c0c736d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:18.989+0000 7f1c0c736d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 11 2020-11-16T21:12:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=11 2020-11-16T21:12:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.11 up' 2020-11-16T21:12:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:19.533+0000 7f1c0c736d80 -1 Falling back to public interface 2020-11-16T21:12:20.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.322 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:20.377+0000 7f1c0c736d80 -1 osd.11 0 log_to_monitors {default=true} 2020-11-16T21:12:20.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:20.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.11 up' 2020-11-16T21:12:20.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:12:20.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.11 up' 2020-11-16T21:12:20.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:20.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:20.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:20.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:20.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 12 2020-11-16T21:12:20.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:20.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:20.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=12 2020-11-16T21:12:20.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:20.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/12 2020-11-16T21:12:20.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:20.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:20.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:20.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:20.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/12' 2020-11-16T21:12:20.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/12/journal' 2020-11-16T21:12:20.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:20.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:20.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:20.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:20.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:20.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:20.327 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:20.327 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:20.327 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:20.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:20.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:20.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:20.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:20.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:20.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:20.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:20.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:20.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:20.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/12 2020-11-16T21:12:20.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:20.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9aa9ef44-2ca1-452e-8fa3-f3d180643e79 2020-11-16T21:12:20.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd12 9aa9ef44-2ca1-452e-8fa3-f3d180643e79' 2020-11-16T21:12:20.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:20.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCG5rJf1X4/KBAAHWR7KD9EcOC+05akNH9Caw== 2020-11-16T21:12:20.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCG5rJf1X4/KBAAHWR7KD9EcOC+05akNH9Caw=="}' 2020-11-16T21:12:20.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9aa9ef44-2ca1-452e-8fa3-f3d180643e79 -i td/osd-backfill-space/12/new.json 2020-11-16T21:12:20.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/12/new.json 2020-11-16T21:12:20.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 12 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/12 --osd-journal=td/osd-backfill-space/12/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCG5rJf1X4/KBAAHWR7KD9EcOC+05akNH9Caw== --osd-uuid 9aa9ef44-2ca1-452e-8fa3-f3d180643e79 2020-11-16T21:12:20.331 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:23.485+0000 7f2354692d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.331 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:23.489+0000 7f2354692d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.331 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:23.493+0000 7f2354692d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.331 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:23.493+0000 7f2354692d80 -1 bluestore(td/osd-backfill-space/12/block) _read_bdev_label failed to open td/osd-backfill-space/12/block: (2) No such file or directory 2020-11-16T21:12:20.331 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:23.493+0000 7f2354692d80 -1 bluestore(td/osd-backfill-space/12/block) _read_bdev_label failed to open td/osd-backfill-space/12/block: (2) No such file or directory 2020-11-16T21:12:20.331 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:23.493+0000 7f2354692d80 -1 bluestore(td/osd-backfill-space/12/block) _read_bdev_label failed to open td/osd-backfill-space/12/block: (2) No such file or directory 2020-11-16T21:12:20.332 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:23.577+0000 7f2354692d80 -1 bluestore(td/osd-backfill-space/12) _read_fsid unparsable uuid 2020-11-16T21:12:20.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/12/keyring 2020-11-16T21:12:20.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:20.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd12 key to auth repository 2020-11-16T21:12:20.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/12/keyring auth add osd.12 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:20.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.12 2020-11-16T21:12:20.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 12 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/12 --osd-journal=td/osd-backfill-space/12/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:20.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:20.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:20.333 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:28.753+0000 7ffba001dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.333 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:28.757+0000 7ffba001dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.333 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:28.757+0000 7ffba001dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 12 2020-11-16T21:12:20.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:20.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=12 2020-11-16T21:12:20.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:20.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:20.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:20.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.12 up' 2020-11-16T21:12:20.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.334 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:29.305+0000 7ffba001dd80 -1 Falling back to public interface 2020-11-16T21:12:20.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.335 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:29.913+0000 7ffba001dd80 -1 osd.12 0 log_to_monitors {default=true} 2020-11-16T21:12:20.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:20.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.12 up' 2020-11-16T21:12:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 13 2020-11-16T21:12:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:20.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=13 2020-11-16T21:12:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/13 2020-11-16T21:12:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/13' 2020-11-16T21:12:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/13/journal' 2020-11-16T21:12:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:20.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:20.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:20.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:20.339 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:20.339 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:20.339 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:20.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:20.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:20.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:20.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:20.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:20.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:20.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:20.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:20.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:20.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/13 2020-11-16T21:12:20.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:20.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d1e0d617-c34d-435f-bddb-7808c3ca712e 2020-11-16T21:12:20.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd13 d1e0d617-c34d-435f-bddb-7808c3ca712e' 2020-11-16T21:12:20.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:20.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCO5rJfhjIGORAA6vrFZv3mvYgogGKgqf1DPg== 2020-11-16T21:12:20.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCO5rJfhjIGORAA6vrFZv3mvYgogGKgqf1DPg=="}' 2020-11-16T21:12:20.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d1e0d617-c34d-435f-bddb-7808c3ca712e -i td/osd-backfill-space/13/new.json 2020-11-16T21:12:20.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/13/new.json 2020-11-16T21:12:20.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 13 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/13 --osd-journal=td/osd-backfill-space/13/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCO5rJfhjIGORAA6vrFZv3mvYgogGKgqf1DPg== --osd-uuid d1e0d617-c34d-435f-bddb-7808c3ca712e 2020-11-16T21:12:20.343 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:31.477+0000 7f37e869dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.343 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:31.481+0000 7f37e869dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.343 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:31.481+0000 7f37e869dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.343 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:31.481+0000 7f37e869dd80 -1 bluestore(td/osd-backfill-space/13/block) _read_bdev_label failed to open td/osd-backfill-space/13/block: (2) No such file or directory 2020-11-16T21:12:20.343 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:31.481+0000 7f37e869dd80 -1 bluestore(td/osd-backfill-space/13/block) _read_bdev_label failed to open td/osd-backfill-space/13/block: (2) No such file or directory 2020-11-16T21:12:20.343 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:31.481+0000 7f37e869dd80 -1 bluestore(td/osd-backfill-space/13/block) _read_bdev_label failed to open td/osd-backfill-space/13/block: (2) No such file or directory 2020-11-16T21:12:20.344 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:31.525+0000 7f37e869dd80 -1 bluestore(td/osd-backfill-space/13) _read_fsid unparsable uuid 2020-11-16T21:12:20.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/13/keyring 2020-11-16T21:12:20.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:20.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd13 key to auth repository 2020-11-16T21:12:20.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/13/keyring auth add osd.13 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:20.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.13 2020-11-16T21:12:20.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:20.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 13 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/13 --osd-journal=td/osd-backfill-space/13/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:36.949+0000 7f275de40d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:36.953+0000 7f275de40d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:36.953+0000 7f275de40d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 13 2020-11-16T21:12:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:20.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=13 2020-11-16T21:12:20.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:20.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:20.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:20.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.13 up' 2020-11-16T21:12:20.346 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:37.509+0000 7f275de40d80 -1 Falling back to public interface 2020-11-16T21:12:20.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.347 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:38.397+0000 7f275de40d80 -1 osd.13 0 log_to_monitors {default=true} 2020-11-16T21:12:20.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:20.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.13 up' 2020-11-16T21:12:20.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:12:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.13 up' 2020-11-16T21:12:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 14 2020-11-16T21:12:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=14 2020-11-16T21:12:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/14 2020-11-16T21:12:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/14' 2020-11-16T21:12:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/14/journal' 2020-11-16T21:12:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:20.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:20.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:20.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:20.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/14 2020-11-16T21:12:20.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=301430e9-8f0f-4858-b138-5dc5dfec9c70 2020-11-16T21:12:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd14 301430e9-8f0f-4858-b138-5dc5dfec9c70' 2020-11-16T21:12:20.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCY5rJf6VezIxAAXZYTgIYseFPdcg6CJKNing== 2020-11-16T21:12:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCY5rJf6VezIxAAXZYTgIYseFPdcg6CJKNing=="}' 2020-11-16T21:12:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 301430e9-8f0f-4858-b138-5dc5dfec9c70 -i td/osd-backfill-space/14/new.json 2020-11-16T21:12:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/14/new.json 2020-11-16T21:12:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 14 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/14 --osd-journal=td/osd-backfill-space/14/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCY5rJf6VezIxAAXZYTgIYseFPdcg6CJKNing== --osd-uuid 301430e9-8f0f-4858-b138-5dc5dfec9c70 2020-11-16T21:12:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:41.209+0000 7f661e950d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.356 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:41.213+0000 7f661e950d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.356 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:41.213+0000 7f661e950d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.356 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:41.213+0000 7f661e950d80 -1 bluestore(td/osd-backfill-space/14/block) _read_bdev_label failed to open td/osd-backfill-space/14/block: (2) No such file or directory 2020-11-16T21:12:20.356 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:41.213+0000 7f661e950d80 -1 bluestore(td/osd-backfill-space/14/block) _read_bdev_label failed to open td/osd-backfill-space/14/block: (2) No such file or directory 2020-11-16T21:12:20.356 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:41.213+0000 7f661e950d80 -1 bluestore(td/osd-backfill-space/14/block) _read_bdev_label failed to open td/osd-backfill-space/14/block: (2) No such file or directory 2020-11-16T21:12:20.356 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:41.337+0000 7f661e950d80 -1 bluestore(td/osd-backfill-space/14) _read_fsid unparsable uuid 2020-11-16T21:12:20.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/14/keyring 2020-11-16T21:12:20.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:20.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd14 key to auth repository 2020-11-16T21:12:20.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/14/keyring auth add osd.14 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:20.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.14 2020-11-16T21:12:20.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 14 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/14 --osd-journal=td/osd-backfill-space/14/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:20.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:20.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:20.358 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:46.705+0000 7f1479f22d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.358 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:46.705+0000 7f1479f22d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.358 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:46.709+0000 7f1479f22d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 14 2020-11-16T21:12:20.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:20.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=14 2020-11-16T21:12:20.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:20.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:20.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:20.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.14 up' 2020-11-16T21:12:20.359 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:47.249+0000 7f1479f22d80 -1 Falling back to public interface 2020-11-16T21:12:20.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:48.205+0000 7f1479f22d80 -1 osd.14 0 log_to_monitors {default=true} 2020-11-16T21:12:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.14 up' 2020-11-16T21:12:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:12:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.14 up' 2020-11-16T21:12:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 15 2020-11-16T21:12:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=15 2020-11-16T21:12:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/15 2020-11-16T21:12:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/15' 2020-11-16T21:12:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/15/journal' 2020-11-16T21:12:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:20.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:20.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:20.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:20.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/15 2020-11-16T21:12:20.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9b59bd85-b530-48f0-9ac0-38da75971adc 2020-11-16T21:12:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd15 9b59bd85-b530-48f0-9ac0-38da75971adc' 2020-11-16T21:12:20.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCi5rJfOrQ2GBAA4ePissfh1PAVtshWmPxxHA== 2020-11-16T21:12:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCi5rJfOrQ2GBAA4ePissfh1PAVtshWmPxxHA=="}' 2020-11-16T21:12:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9b59bd85-b530-48f0-9ac0-38da75971adc -i td/osd-backfill-space/15/new.json 2020-11-16T21:12:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/15/new.json 2020-11-16T21:12:20.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 15 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/15 --osd-journal=td/osd-backfill-space/15/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCi5rJfOrQ2GBAA4ePissfh1PAVtshWmPxxHA== --osd-uuid 9b59bd85-b530-48f0-9ac0-38da75971adc 2020-11-16T21:12:20.368 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:51.033+0000 7f2c99878d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.368 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:51.037+0000 7f2c99878d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.368 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:51.037+0000 7f2c99878d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.368 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:51.037+0000 7f2c99878d80 -1 bluestore(td/osd-backfill-space/15/block) _read_bdev_label failed to open td/osd-backfill-space/15/block: (2) No such file or directory 2020-11-16T21:12:20.368 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:51.037+0000 7f2c99878d80 -1 bluestore(td/osd-backfill-space/15/block) _read_bdev_label failed to open td/osd-backfill-space/15/block: (2) No such file or directory 2020-11-16T21:12:20.369 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:51.037+0000 7f2c99878d80 -1 bluestore(td/osd-backfill-space/15/block) _read_bdev_label failed to open td/osd-backfill-space/15/block: (2) No such file or directory 2020-11-16T21:12:20.369 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:51.105+0000 7f2c99878d80 -1 bluestore(td/osd-backfill-space/15) _read_fsid unparsable uuid 2020-11-16T21:12:20.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/15/keyring 2020-11-16T21:12:20.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:20.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd15 key to auth repository 2020-11-16T21:12:20.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/15/keyring auth add osd.15 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:20.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.15 2020-11-16T21:12:20.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 15 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/15 --osd-journal=td/osd-backfill-space/15/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:20.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:20.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:20.371 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:56.545+0000 7fd6ca8c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.371 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:56.545+0000 7fd6ca8c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.371 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:56.545+0000 7fd6ca8c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 15 2020-11-16T21:12:20.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:20.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=15 2020-11-16T21:12:20.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:20.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:20.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:20.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.15 up' 2020-11-16T21:12:20.372 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:57.089+0000 7fd6ca8c0d80 -1 Falling back to public interface 2020-11-16T21:12:20.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.372 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:52:57.969+0000 7fd6ca8c0d80 -1 osd.15 0 log_to_monitors {default=true} 2020-11-16T21:12:20.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:20.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.15 up' 2020-11-16T21:12:20.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:12:20.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.15 up' 2020-11-16T21:12:20.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:20.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:20.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 16 2020-11-16T21:12:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=16 2020-11-16T21:12:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/16 2020-11-16T21:12:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:20.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:20.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:20.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:20.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/16' 2020-11-16T21:12:20.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/16/journal' 2020-11-16T21:12:20.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:20.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:20.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:20.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:20.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:20.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:20.377 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:20.377 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:20.377 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:20.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:20.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:20.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:20.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:20.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:20.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:20.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:20.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:20.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:20.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/16 2020-11-16T21:12:20.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:20.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=808f55de-c834-4dda-ba03-66260ea3da9b 2020-11-16T21:12:20.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd16 808f55de-c834-4dda-ba03-66260ea3da9b' 2020-11-16T21:12:20.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:20.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCs5rJfWvLnBhAAB+CKFXkuxRUaqJYN8KOkJA== 2020-11-16T21:12:20.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCs5rJfWvLnBhAAB+CKFXkuxRUaqJYN8KOkJA=="}' 2020-11-16T21:12:20.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 808f55de-c834-4dda-ba03-66260ea3da9b -i td/osd-backfill-space/16/new.json 2020-11-16T21:12:20.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/16/new.json 2020-11-16T21:12:20.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 16 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/16 --osd-journal=td/osd-backfill-space/16/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCs5rJfWvLnBhAAB+CKFXkuxRUaqJYN8KOkJA== --osd-uuid 808f55de-c834-4dda-ba03-66260ea3da9b 2020-11-16T21:12:20.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:00.745+0000 7f1f6622dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:00.749+0000 7f1f6622dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:00.749+0000 7f1f6622dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:00.753+0000 7f1f6622dd80 -1 bluestore(td/osd-backfill-space/16/block) _read_bdev_label failed to open td/osd-backfill-space/16/block: (2) No such file or directory 2020-11-16T21:12:20.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:00.753+0000 7f1f6622dd80 -1 bluestore(td/osd-backfill-space/16/block) _read_bdev_label failed to open td/osd-backfill-space/16/block: (2) No such file or directory 2020-11-16T21:12:20.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:00.753+0000 7f1f6622dd80 -1 bluestore(td/osd-backfill-space/16/block) _read_bdev_label failed to open td/osd-backfill-space/16/block: (2) No such file or directory 2020-11-16T21:12:20.382 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:00.849+0000 7f1f6622dd80 -1 bluestore(td/osd-backfill-space/16) _read_fsid unparsable uuid 2020-11-16T21:12:20.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/16/keyring 2020-11-16T21:12:20.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:20.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd16 key to auth repository 2020-11-16T21:12:20.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/16/keyring auth add osd.16 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:20.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.16 2020-11-16T21:12:20.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 16 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/16 --osd-journal=td/osd-backfill-space/16/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:20.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:20.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:20.383 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:06.256+0000 7fd2b7a50d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.383 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:06.260+0000 7fd2b7a50d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.383 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:06.260+0000 7fd2b7a50d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 16 2020-11-16T21:12:20.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:20.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=16 2020-11-16T21:12:20.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:20.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:20.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:20.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.16 up' 2020-11-16T21:12:20.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.385 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:06.796+0000 7fd2b7a50d80 -1 Falling back to public interface 2020-11-16T21:12:20.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.385 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:07.668+0000 7fd2b7a50d80 -1 osd.16 0 log_to_monitors {default=true} 2020-11-16T21:12:20.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:20.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.16 up' 2020-11-16T21:12:20.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:12:20.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.16 up' 2020-11-16T21:12:20.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:20.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:20.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:20.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:20.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 17 2020-11-16T21:12:20.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:20.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:20.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=17 2020-11-16T21:12:20.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:20.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/17 2020-11-16T21:12:20.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:20.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:20.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:20.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:20.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/17' 2020-11-16T21:12:20.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/17/journal' 2020-11-16T21:12:20.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:20.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:20.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:20.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:20.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:20.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:20.391 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:20.391 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:20.391 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:20.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:20.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:20.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:20.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:20.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:20.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:20.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:20.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:20.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:20.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/17 2020-11-16T21:12:20.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:20.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=11997534-baef-4fbe-b8b3-3d02488918ee 2020-11-16T21:12:20.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd17 11997534-baef-4fbe-b8b3-3d02488918ee' 2020-11-16T21:12:20.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:20.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC15rJfVQSTOxAAsqvmmAl83dUy36r1FsxGcg== 2020-11-16T21:12:20.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC15rJfVQSTOxAAsqvmmAl83dUy36r1FsxGcg=="}' 2020-11-16T21:12:20.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 11997534-baef-4fbe-b8b3-3d02488918ee -i td/osd-backfill-space/17/new.json 2020-11-16T21:12:20.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/17/new.json 2020-11-16T21:12:20.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 17 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/17 --osd-journal=td/osd-backfill-space/17/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC15rJfVQSTOxAAsqvmmAl83dUy36r1FsxGcg== --osd-uuid 11997534-baef-4fbe-b8b3-3d02488918ee 2020-11-16T21:12:20.396 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:10.652+0000 7f992ceefd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.396 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:10.656+0000 7f992ceefd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.396 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:10.656+0000 7f992ceefd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.396 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:10.656+0000 7f992ceefd80 -1 bluestore(td/osd-backfill-space/17/block) _read_bdev_label failed to open td/osd-backfill-space/17/block: (2) No such file or directory 2020-11-16T21:12:20.396 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:10.656+0000 7f992ceefd80 -1 bluestore(td/osd-backfill-space/17/block) _read_bdev_label failed to open td/osd-backfill-space/17/block: (2) No such file or directory 2020-11-16T21:12:20.397 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:10.656+0000 7f992ceefd80 -1 bluestore(td/osd-backfill-space/17/block) _read_bdev_label failed to open td/osd-backfill-space/17/block: (2) No such file or directory 2020-11-16T21:12:20.397 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:10.736+0000 7f992ceefd80 -1 bluestore(td/osd-backfill-space/17) _read_fsid unparsable uuid 2020-11-16T21:12:20.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/17/keyring 2020-11-16T21:12:20.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:20.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd17 key to auth repository 2020-11-16T21:12:20.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/17/keyring auth add osd.17 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:20.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.17 2020-11-16T21:12:20.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 17 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/17 --osd-journal=td/osd-backfill-space/17/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:20.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:20.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:20.399 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:16.220+0000 7f9fad100d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.399 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:16.224+0000 7f9fad100d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.399 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:16.224+0000 7f9fad100d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 17 2020-11-16T21:12:20.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:20.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=17 2020-11-16T21:12:20.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:20.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:20.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:20.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.17 up' 2020-11-16T21:12:20.400 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:16.760+0000 7f9fad100d80 -1 Falling back to public interface 2020-11-16T21:12:20.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:17.604+0000 7f9fad100d80 -1 osd.17 0 log_to_monitors {default=true} 2020-11-16T21:12:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.17 up' 2020-11-16T21:12:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:12:20.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.17 up' 2020-11-16T21:12:20.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:20.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:20.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:20.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:20.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 18 2020-11-16T21:12:20.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:20.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:20.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=18 2020-11-16T21:12:20.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:20.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/18 2020-11-16T21:12:20.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:20.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:20.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:20.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:20.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/18' 2020-11-16T21:12:20.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/18/journal' 2020-11-16T21:12:20.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:20.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:20.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:20.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:20.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:20.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:20.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:20.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:20.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:20.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:20.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:20.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:20.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:20.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:20.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:20.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:20.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:20.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:20.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/18 2020-11-16T21:12:20.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:20.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6c5aed7d-0a46-4541-a1a4-345da649cb11 2020-11-16T21:12:20.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd18 6c5aed7d-0a46-4541-a1a4-345da649cb11' 2020-11-16T21:12:20.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:20.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC/5rJfRiKSNhAAlCcw2qbDou+LbPDw3eEeqw== 2020-11-16T21:12:20.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC/5rJfRiKSNhAAlCcw2qbDou+LbPDw3eEeqw=="}' 2020-11-16T21:12:20.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6c5aed7d-0a46-4541-a1a4-345da649cb11 -i td/osd-backfill-space/18/new.json 2020-11-16T21:12:20.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/18/new.json 2020-11-16T21:12:20.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 18 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/18 --osd-journal=td/osd-backfill-space/18/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC/5rJfRiKSNhAAlCcw2qbDou+LbPDw3eEeqw== --osd-uuid 6c5aed7d-0a46-4541-a1a4-345da649cb11 2020-11-16T21:12:20.409 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:20.484+0000 7f48a9da5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.409 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:20.488+0000 7f48a9da5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.409 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:20.488+0000 7f48a9da5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.409 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:20.488+0000 7f48a9da5d80 -1 bluestore(td/osd-backfill-space/18/block) _read_bdev_label failed to open td/osd-backfill-space/18/block: (2) No such file or directory 2020-11-16T21:12:20.409 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:20.488+0000 7f48a9da5d80 -1 bluestore(td/osd-backfill-space/18/block) _read_bdev_label failed to open td/osd-backfill-space/18/block: (2) No such file or directory 2020-11-16T21:12:20.410 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:20.488+0000 7f48a9da5d80 -1 bluestore(td/osd-backfill-space/18/block) _read_bdev_label failed to open td/osd-backfill-space/18/block: (2) No such file or directory 2020-11-16T21:12:20.410 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:20.612+0000 7f48a9da5d80 -1 bluestore(td/osd-backfill-space/18) _read_fsid unparsable uuid 2020-11-16T21:12:20.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/18/keyring 2020-11-16T21:12:20.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:20.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd18 key to auth repository 2020-11-16T21:12:20.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/18/keyring auth add osd.18 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:20.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.18 2020-11-16T21:12:20.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 18 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/18 --osd-journal=td/osd-backfill-space/18/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:20.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:20.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:20.411 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:26.584+0000 7f394b7f6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.411 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:26.584+0000 7f394b7f6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.412 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:26.584+0000 7f394b7f6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 18 2020-11-16T21:12:20.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:20.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=18 2020-11-16T21:12:20.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:20.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:20.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:20.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.18 up' 2020-11-16T21:12:20.413 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:27.136+0000 7f394b7f6d80 -1 Falling back to public interface 2020-11-16T21:12:20.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.413 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:28.036+0000 7f394b7f6d80 -1 osd.18 0 log_to_monitors {default=true} 2020-11-16T21:12:20.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:20.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.18 up' 2020-11-16T21:12:20.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:12:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.18 up' 2020-11-16T21:12:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:12:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 19 2020-11-16T21:12:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:12:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:12:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=19 2020-11-16T21:12:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:12:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/19 2020-11-16T21:12:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:12:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:12:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:12:20.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:12:20.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/19' 2020-11-16T21:12:20.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/19/journal' 2020-11-16T21:12:20.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:12:20.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:12:20.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:12:20.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:12:20.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:12:20.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:12:20.418 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:12:20.418 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:12:20.418 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:12:20.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:12:20.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:12:20.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:12:20.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:12:20.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:12:20.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:12:20.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:12:20.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:12:20.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:12:20.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:12:20.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/19 2020-11-16T21:12:20.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:12:20.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=456df5ff-f0a4-4a3d-add6-4a1ac301bc08 2020-11-16T21:12:20.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd19 456df5ff-f0a4-4a3d-add6-4a1ac301bc08' 2020-11-16T21:12:20.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:12:20.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDK5rJfWVyiCxAAKbbiZJ5GDYN7WLccS02S/Q== 2020-11-16T21:12:20.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDK5rJfWVyiCxAAKbbiZJ5GDYN7WLccS02S/Q=="}' 2020-11-16T21:12:20.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 456df5ff-f0a4-4a3d-add6-4a1ac301bc08 -i td/osd-backfill-space/19/new.json 2020-11-16T21:12:20.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/19/new.json 2020-11-16T21:12:20.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 19 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/19 --osd-journal=td/osd-backfill-space/19/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDK5rJfWVyiCxAAKbbiZJ5GDYN7WLccS02S/Q== --osd-uuid 456df5ff-f0a4-4a3d-add6-4a1ac301bc08 2020-11-16T21:12:20.422 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:30.724+0000 7fae686e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.422 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:30.724+0000 7fae686e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.422 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:30.728+0000 7fae686e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.422 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:30.728+0000 7fae686e9d80 -1 bluestore(td/osd-backfill-space/19/block) _read_bdev_label failed to open td/osd-backfill-space/19/block: (2) No such file or directory 2020-11-16T21:12:20.422 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:30.728+0000 7fae686e9d80 -1 bluestore(td/osd-backfill-space/19/block) _read_bdev_label failed to open td/osd-backfill-space/19/block: (2) No such file or directory 2020-11-16T21:12:20.422 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:30.728+0000 7fae686e9d80 -1 bluestore(td/osd-backfill-space/19/block) _read_bdev_label failed to open td/osd-backfill-space/19/block: (2) No such file or directory 2020-11-16T21:12:20.422 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:30.824+0000 7fae686e9d80 -1 bluestore(td/osd-backfill-space/19) _read_fsid unparsable uuid 2020-11-16T21:12:20.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/19/keyring 2020-11-16T21:12:20.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:12:20.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd19 key to auth repository 2020-11-16T21:12:20.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/19/keyring auth add osd.19 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:12:20.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.19 2020-11-16T21:12:20.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:12:20.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 19 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/19 --osd-journal=td/osd-backfill-space/19/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:12:20.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:12:20.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:12:20.424 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:36.920+0000 7f006ef8cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.424 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:36.924+0000 7f006ef8cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.424 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:36.924+0000 7f006ef8cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:12:20.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 19 2020-11-16T21:12:20.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:12:20.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=19 2020-11-16T21:12:20.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:12:20.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:12:20.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:12:20.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.19 up' 2020-11-16T21:12:20.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.425 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:37.472+0000 7f006ef8cd80 -1 Falling back to public interface 2020-11-16T21:12:20.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.426 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T20:53:38.380+0000 7f006ef8cd80 -1 osd.19 0 log_to_monitors {default=true} 2020-11-16T21:12:20.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:12:20.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.19 up' 2020-11-16T21:12:20.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:12:20.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:12:20.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:12:20.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:12:20.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:12:20.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.19 up' 2020-11-16T21:12:20.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:12:20.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:12:20.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:12:20.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:204: TEST_backfill_test_multi: ceph osd set-backfillfull-ratio .85 2020-11-16T21:12:20.428 INFO:tasks.workunit.client.0.smithi189.stderr:osd set-backfillfull-ratio 0.85 2020-11-16T21:12:20.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test1 1 1 2020-11-16T21:12:20.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2020-11-16T21:12:20.429 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' already exists 2020-11-16T21:12:20.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:12:20.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2020-11-16T21:12:20.429 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 1 2020-11-16T21:12:20.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test2 1 1 2020-11-16T21:12:20.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2020-11-16T21:12:20.430 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' already exists 2020-11-16T21:12:20.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:12:20.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2020-11-16T21:12:20.430 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 1 2020-11-16T21:12:20.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test3 1 1 2020-11-16T21:12:20.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2020-11-16T21:12:20.431 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' already exists 2020-11-16T21:12:20.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:12:20.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2020-11-16T21:12:20.431 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 1 2020-11-16T21:12:20.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test4 1 1 2020-11-16T21:12:20.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test4 1 1 2020-11-16T21:12:20.432 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test4' already exists 2020-11-16T21:12:20.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:12:20.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test4 size 1 --yes-i-really-mean-it 2020-11-16T21:12:20.432 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 1 2020-11-16T21:12:20.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test5 1 1 2020-11-16T21:12:20.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test5 1 1 2020-11-16T21:12:20.433 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test5' already exists 2020-11-16T21:12:20.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:12:20.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test5 size 1 --yes-i-really-mean-it 2020-11-16T21:12:20.433 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 5 size to 1 2020-11-16T21:12:20.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test6 1 1 2020-11-16T21:12:20.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test6 1 1 2020-11-16T21:12:20.434 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test6' already exists 2020-11-16T21:12:20.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:12:20.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test6 size 1 --yes-i-really-mean-it 2020-11-16T21:12:20.434 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 6 size to 1 2020-11-16T21:12:20.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test7 1 1 2020-11-16T21:12:20.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test7 1 1 2020-11-16T21:12:20.435 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test7' already exists 2020-11-16T21:12:20.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:12:20.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2020-11-16T21:12:20.435 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 7 size to 1 2020-11-16T21:12:20.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test8 1 1 2020-11-16T21:12:20.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test8 1 1 2020-11-16T21:12:20.436 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test8' already exists 2020-11-16T21:12:20.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:12:20.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test8 size 1 --yes-i-really-mean-it 2020-11-16T21:12:20.436 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 8 size to 1 2020-11-16T21:12:20.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:212: TEST_backfill_test_multi: wait_for_clean 2020-11-16T21:12:20.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:12:20.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:12:20.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:12:20.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:12:20.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:12:20.437 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:12:20.437 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:12:20.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:12:20.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:12:20.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:12:20.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:12:20.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:12:20.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:12:20.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:12:20.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:12:20.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:12:20.439 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:12:20.439 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:12:20.439 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:12:20.439 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:12:20.439 INFO:tasks.workunit.client.0.smithi189.stderr:5 2020-11-16T21:12:20.440 INFO:tasks.workunit.client.0.smithi189.stderr:6 2020-11-16T21:12:20.440 INFO:tasks.workunit.client.0.smithi189.stderr:7 2020-11-16T21:12:20.440 INFO:tasks.workunit.client.0.smithi189.stderr:8 2020-11-16T21:12:20.440 INFO:tasks.workunit.client.0.smithi189.stderr:9 2020-11-16T21:12:20.440 INFO:tasks.workunit.client.0.smithi189.stderr:10 2020-11-16T21:12:20.440 INFO:tasks.workunit.client.0.smithi189.stderr:11 2020-11-16T21:12:20.441 INFO:tasks.workunit.client.0.smithi189.stderr:12 2020-11-16T21:12:20.441 INFO:tasks.workunit.client.0.smithi189.stderr:13 2020-11-16T21:12:20.441 INFO:tasks.workunit.client.0.smithi189.stderr:14 2020-11-16T21:12:20.441 INFO:tasks.workunit.client.0.smithi189.stderr:15 2020-11-16T21:12:20.441 INFO:tasks.workunit.client.0.smithi189.stderr:16 2020-11-16T21:12:20.441 INFO:tasks.workunit.client.0.smithi189.stderr:17 2020-11-16T21:12:20.441 INFO:tasks.workunit.client.0.smithi189.stderr:18 2020-11-16T21:12:20.442 INFO:tasks.workunit.client.0.smithi189.stderr:19' 2020-11-16T21:12:20.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:12:20.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:12:20.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836521 2020-11-16T21:12:20.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836521 2020-11-16T21:12:20.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521' 2020-11-16T21:12:20.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:12:20.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705705 2020-11-16T21:12:20.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705705 2020-11-16T21:12:20.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705' 2020-11-16T21:12:20.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:12:20.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542182 2020-11-16T21:12:20.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542182 2020-11-16T21:12:20.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182' 2020-11-16T21:12:20.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:12:20.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378662 2020-11-16T21:12:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378662 2020-11-16T21:12:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662' 2020-11-16T21:12:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:12:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247843 2020-11-16T21:12:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247843 2020-11-16T21:12:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843' 2020-11-16T21:12:20.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:12:20.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084322 2020-11-16T21:12:20.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084322 2020-11-16T21:12:20.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322' 2020-11-16T21:12:20.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-11-16T21:12:20.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920800 2020-11-16T21:12:20.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920800 2020-11-16T21:12:20.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800' 2020-11-16T21:12:20.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-11-16T21:12:20.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757279 2020-11-16T21:12:20.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757279 2020-11-16T21:12:20.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279' 2020-11-16T21:12:20.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-11-16T21:12:20.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=184683593757 2020-11-16T21:12:20.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 184683593757 2020-11-16T21:12:20.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757' 2020-11-16T21:12:20.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-11-16T21:12:20.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=206158430235 2020-11-16T21:12:20.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 206158430235 2020-11-16T21:12:20.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757 9-206158430235' 2020-11-16T21:12:20.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-11-16T21:12:20.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=227633266713 2020-11-16T21:12:20.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 227633266713 2020-11-16T21:12:20.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757 9-206158430235 10-227633266713' 2020-11-16T21:12:20.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.11 flush_pg_stats 2020-11-16T21:12:20.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=249108103193 2020-11-16T21:12:20.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 249108103193 2020-11-16T21:12:20.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757 9-206158430235 10-227633266713 11-249108103193' 2020-11-16T21:12:20.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.12 flush_pg_stats 2020-11-16T21:12:20.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=270582939671 2020-11-16T21:12:20.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 270582939671 2020-11-16T21:12:20.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757 9-206158430235 10-227633266713 11-249108103193 12-270582939671' 2020-11-16T21:12:20.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.13 flush_pg_stats 2020-11-16T21:12:20.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=292057776148 2020-11-16T21:12:20.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 292057776148 2020-11-16T21:12:20.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757 9-206158430235 10-227633266713 11-249108103193 12-270582939671 13-292057776148' 2020-11-16T21:12:20.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.14 flush_pg_stats 2020-11-16T21:12:20.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=313532612627 2020-11-16T21:12:20.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 313532612627 2020-11-16T21:12:20.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757 9-206158430235 10-227633266713 11-249108103193 12-270582939671 13-292057776148 14-313532612627' 2020-11-16T21:12:20.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.15 flush_pg_stats 2020-11-16T21:12:20.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=335007449104 2020-11-16T21:12:20.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 335007449104 2020-11-16T21:12:20.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757 9-206158430235 10-227633266713 11-249108103193 12-270582939671 13-292057776148 14-313532612627 15-335007449104' 2020-11-16T21:12:20.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.16 flush_pg_stats 2020-11-16T21:12:20.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=356482285582 2020-11-16T21:12:20.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 356482285582 2020-11-16T21:12:20.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757 9-206158430235 10-227633266713 11-249108103193 12-270582939671 13-292057776148 14-313532612627 15-335007449104 16-356482285582' 2020-11-16T21:12:20.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.17 flush_pg_stats 2020-11-16T21:12:20.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=377957122060 2020-11-16T21:12:20.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 377957122060 2020-11-16T21:12:20.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757 9-206158430235 10-227633266713 11-249108103193 12-270582939671 13-292057776148 14-313532612627 15-335007449104 16-356482285582 17-377957122060' 2020-11-16T21:12:20.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.18 flush_pg_stats 2020-11-16T21:12:20.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=399431958538 2020-11-16T21:12:20.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 399431958538 2020-11-16T21:12:20.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757 9-206158430235 10-227633266713 11-249108103193 12-270582939671 13-292057776148 14-313532612627 15-335007449104 16-356482285582 17-377957122060 18-399431958538' 2020-11-16T21:12:20.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:12:20.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.19 flush_pg_stats 2020-11-16T21:12:20.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=420906795016 2020-11-16T21:12:20.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 420906795016 2020-11-16T21:12:20.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836521 1-38654705705 2-60129542182 3-81604378662 4-98784247843 5-120259084322 6-141733920800 7-163208757279 8-184683593757 9-206158430235 10-227633266713 11-249108103193 12-270582939671 13-292057776148 14-313532612627 15-335007449104 16-356482285582 17-377957122060 18-399431958538 19-420906795016' 2020-11-16T21:12:20.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836521 2020-11-16T21:12:20.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:12:20.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836521 2020-11-16T21:12:20.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836521 2020-11-16T21:12:20.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2020-11-16T21:12:20.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:12:20.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836522 -lt 21474836521 2020-11-16T21:12:20.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705705 2020-11-16T21:12:20.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:12:20.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705705 2020-11-16T21:12:20.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705705 2020-11-16T21:12:20.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705705' 2020-11-16T21:12:20.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:12:20.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705706 -lt 38654705705 2020-11-16T21:12:20.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542182 2020-11-16T21:12:20.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:12:20.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542182 2020-11-16T21:12:20.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542182 2020-11-16T21:12:20.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542182' 2020-11-16T21:12:20.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:12:20.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542183 -lt 60129542182 2020-11-16T21:12:20.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378662 2020-11-16T21:12:20.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:12:20.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378662 2020-11-16T21:12:20.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378662 2020-11-16T21:12:20.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378662' 2020-11-16T21:12:20.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:12:20.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378662 -lt 81604378662 2020-11-16T21:12:20.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247843 2020-11-16T21:12:20.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:12:20.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247843 2020-11-16T21:12:20.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247843 2020-11-16T21:12:20.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247843' 2020-11-16T21:12:20.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:12:20.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247844 -lt 98784247843 2020-11-16T21:12:20.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084322 2020-11-16T21:12:20.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:12:20.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084322 2020-11-16T21:12:20.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084322 2020-11-16T21:12:20.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084322' 2020-11-16T21:12:20.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:12:20.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084323 -lt 120259084322 2020-11-16T21:12:20.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-141733920800 2020-11-16T21:12:20.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-11-16T21:12:20.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-141733920800 2020-11-16T21:12:20.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920800 2020-11-16T21:12:20.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 141733920800' 2020-11-16T21:12:20.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-11-16T21:12:20.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920801 -lt 141733920800 2020-11-16T21:12:20.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-163208757279 2020-11-16T21:12:20.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-11-16T21:12:20.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-163208757279 2020-11-16T21:12:20.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757279 2020-11-16T21:12:20.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 163208757279' 2020-11-16T21:12:20.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-11-16T21:12:20.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757279 -lt 163208757279 2020-11-16T21:12:20.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-184683593757 2020-11-16T21:12:20.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-11-16T21:12:20.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-184683593757 2020-11-16T21:12:20.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=184683593757 2020-11-16T21:12:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 184683593757' 2020-11-16T21:12:20.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-11-16T21:12:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 184683593758 -lt 184683593757 2020-11-16T21:12:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-206158430235 2020-11-16T21:12:20.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-11-16T21:12:20.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-206158430235 2020-11-16T21:12:20.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=206158430235 2020-11-16T21:12:20.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 206158430235' 2020-11-16T21:12:20.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-11-16T21:12:20.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 206158430236 -lt 206158430235 2020-11-16T21:12:20.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-227633266713 2020-11-16T21:12:20.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-11-16T21:12:20.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-227633266713 2020-11-16T21:12:20.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=227633266713 2020-11-16T21:12:20.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 227633266713' 2020-11-16T21:12:20.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-11-16T21:12:20.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 227633266715 -lt 227633266713 2020-11-16T21:12:20.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 11-249108103193 2020-11-16T21:12:20.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=11 2020-11-16T21:12:20.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 11-249108103193 2020-11-16T21:12:20.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=249108103193 2020-11-16T21:12:20.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.11 seq 249108103193' 2020-11-16T21:12:20.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 11 2020-11-16T21:12:20.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 249108103194 -lt 249108103193 2020-11-16T21:12:20.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 12-270582939671 2020-11-16T21:12:20.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=12 2020-11-16T21:12:20.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 12-270582939671 2020-11-16T21:12:20.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=270582939671 2020-11-16T21:12:20.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.12 seq 270582939671' 2020-11-16T21:12:20.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 12 2020-11-16T21:12:20.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 270582939672 -lt 270582939671 2020-11-16T21:12:20.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 13-292057776148 2020-11-16T21:12:20.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=13 2020-11-16T21:12:20.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 13-292057776148 2020-11-16T21:12:20.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=292057776148 2020-11-16T21:12:20.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.13 seq 292057776148' 2020-11-16T21:12:20.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 13 2020-11-16T21:12:20.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 292057776149 -lt 292057776148 2020-11-16T21:12:20.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 14-313532612627 2020-11-16T21:12:20.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=14 2020-11-16T21:12:20.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 14-313532612627 2020-11-16T21:12:20.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=313532612627 2020-11-16T21:12:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.14 seq 313532612627' 2020-11-16T21:12:20.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 14 2020-11-16T21:12:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 313532612628 -lt 313532612627 2020-11-16T21:12:20.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 15-335007449104 2020-11-16T21:12:20.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=15 2020-11-16T21:12:20.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 15-335007449104 2020-11-16T21:12:20.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=335007449104 2020-11-16T21:12:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.15 seq 335007449104' 2020-11-16T21:12:20.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 15 2020-11-16T21:12:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 335007449106 -lt 335007449104 2020-11-16T21:12:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 16-356482285582 2020-11-16T21:12:20.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=16 2020-11-16T21:12:20.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 16-356482285582 2020-11-16T21:12:20.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=356482285582 2020-11-16T21:12:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.16 seq 356482285582' 2020-11-16T21:12:20.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 16 2020-11-16T21:12:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 356482285584 -lt 356482285582 2020-11-16T21:12:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 17-377957122060 2020-11-16T21:12:20.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=17 2020-11-16T21:12:20.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 17-377957122060 2020-11-16T21:12:20.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=377957122060 2020-11-16T21:12:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.17 seq 377957122060' 2020-11-16T21:12:20.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 17 2020-11-16T21:12:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 377957122062 -lt 377957122060 2020-11-16T21:12:20.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 18-399431958538 2020-11-16T21:12:20.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=18 2020-11-16T21:12:20.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 18-399431958538 2020-11-16T21:12:20.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=399431958538 2020-11-16T21:12:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.18 seq 399431958538' 2020-11-16T21:12:20.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 18 2020-11-16T21:12:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 399431958540 -lt 399431958538 2020-11-16T21:12:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:12:20.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 19-420906795016 2020-11-16T21:12:20.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:12:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=19 2020-11-16T21:12:20.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 19-420906795016 2020-11-16T21:12:20.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:12:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=420906795016 2020-11-16T21:12:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.19 seq 420906795016' 2020-11-16T21:12:20.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 19 2020-11-16T21:12:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 420906795018 -lt 420906795016 2020-11-16T21:12:20.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:12:20.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:12:20.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:12:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 8 == 0 2020-11-16T21:12:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:12:20.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:12:20.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:12:20.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:12:20.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:12:20.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:12:20.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:12:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2020-11-16T21:12:20.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:12:20.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:12:20.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:12:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 8 2020-11-16T21:12:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:12:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:12:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:214: TEST_backfill_test_multi: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=4 2020-11-16T21:12:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:4+0 records in 2020-11-16T21:12:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:4+0 records out 2020-11-16T21:12:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000228059 s, 18.0 MB/s 2020-11-16T21:12:20.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: seq 1 600 2020-11-16T21:12:20.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:12:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:12:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:12:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:12:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:12:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:12:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:12:20.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:12:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:12:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:12:20.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:12:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:12:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:12:20.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:12:20.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:12:20.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:12:20.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:12:20.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:12:20.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:12:20.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:12:20.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:12:20.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:12:20.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:12:20.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:12:20.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:12:20.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:12:20.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:12:20.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:12:20.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:12:20.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:12:20.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:12:20.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:12:20.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:12:20.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:12:20.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:12:20.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:12:20.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:12:20.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:12:20.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:12:20.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:12:20.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:12:20.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:12:20.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:12:20.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:12:20.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:12:20.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:12:20.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:12:20.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:12:20.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:12:20.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:12:20.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:12:20.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:12:20.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:12:20.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:12:20.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:12:20.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:12:20.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:12:20.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:12:20.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:12:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:12:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:12:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:12:20.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:12:20.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:12:20.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:12:20.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:12:20.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:12:20.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:12:20.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:12:20.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:12:20.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:12:20.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:12:20.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:12:20.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:12:20.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:12:20.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:12:20.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:12:20.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:12:20.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:12:20.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:12:20.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:12:20.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:12:20.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:12:20.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:12:20.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:12:20.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:12:20.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:12:20.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:12:20.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:12:20.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:12:20.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:12:20.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:12:20.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:12:20.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:12:20.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:12:20.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:12:20.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:12:20.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:12:20.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:12:20.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:12:20.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:12:20.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:12:20.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:12:20.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:12:20.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:12:20.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:12:20.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:12:20.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:12:20.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:12:20.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:12:20.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:12:20.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:12:20.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:12:20.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:12:20.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:12:20.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:12:20.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:12:20.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:12:20.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:12:20.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:12:20.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:12:20.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:12:20.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:12:20.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:12:20.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:12:20.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:12:20.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:12:20.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:12:20.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:12:20.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:12:20.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:12:20.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:12:20.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:12:20.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:12:20.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:12:20.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:12:20.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:12:20.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:12:20.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:12:20.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:12:20.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:12:20.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:12:20.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:12:20.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:12:20.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:12:20.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:12:20.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:12:20.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:12:20.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:12:20.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:12:20.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:12:20.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:12:20.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:12:20.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:12:20.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:12:20.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:12:20.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:12:20.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:12:20.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:12:20.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:12:20.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:12:20.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:12:20.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:12:20.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:12:20.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:12:20.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:12:20.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:12:20.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:12:20.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:12:20.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:12:20.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:12:20.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:12:20.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:12:20.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:12:20.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:12:20.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:12:20.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:12:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:12:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:12:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:12:20.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:12:20.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:12:20.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:12:20.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:12:20.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:12:20.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:12:20.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:12:20.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:12:20.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:12:20.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:12:20.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:12:20.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:12:20.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:12:20.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:12:20.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:12:20.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:12:20.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:12:20.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:12:20.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:12:20.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:12:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:12:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:12:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:12:20.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:12:20.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:12:20.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:12:20.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:12:20.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:12:20.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:12:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:12:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:12:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:12:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:12:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:12:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:12:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:12:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:12:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:12:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:12:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:12:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:12:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:12:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:12:20.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:12:20.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:12:20.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:12:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:12:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:12:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:12:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:12:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:12:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:12:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:12:20.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:12:20.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:12:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:12:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:12:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:12:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:12:20.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:12:20.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:12:20.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:12:20.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:12:20.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:12:20.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:12:20.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:12:20.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:12:20.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:12:20.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:12:20.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:12:20.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:12:20.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:12:20.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:12:20.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:12:20.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:12:20.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:12:20.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:12:20.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:12:20.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:12:20.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:12:20.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:12:20.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:12:20.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:12:20.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:12:20.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:12:20.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:12:20.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:12:20.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:12:20.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:12:20.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:12:20.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:12:20.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:12:20.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:12:20.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:12:20.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:12:20.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:12:20.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:12:20.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:12:20.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:12:20.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:12:20.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:12:20.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:12:20.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:12:20.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:12:20.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:12:20.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:12:20.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:12:20.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:12:20.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.576 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:12:20.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:12:20.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:12:20.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:12:20.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:12:20.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:12:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:12:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:12:20.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:12:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:12:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:12:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:12:20.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:12:20.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:12:20.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:12:20.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:12:20.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:12:20.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:12:20.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:12:20.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:12:20.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:12:20.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:12:20.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:12:20.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:12:20.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:12:20.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:12:20.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:12:20.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:12:20.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:12:20.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:12:20.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:12:20.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:12:20.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:12:20.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:12:20.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:12:20.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:12:20.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:12:20.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:12:20.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:12:20.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:12:20.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.588 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:12:20.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:12:20.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:12:20.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:12:20.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:12:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:12:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:12:20.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:12:20.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:12:20.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:12:20.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:12:20.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:12:20.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:12:20.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:12:20.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:12:20.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:12:20.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:12:20.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:12:20.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:12:20.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:12:20.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:12:20.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:12:20.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:12:20.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:12:20.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:12:20.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:12:20.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:12:20.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:12:20.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:12:20.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:12:20.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:12:20.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:12:20.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:12:20.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:12:20.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:12:20.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:12:20.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:12:20.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:12:20.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:12:20.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:12:20.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:12:20.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:12:20.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:12:20.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:12:20.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:12:20.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:12:20.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:12:20.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:12:20.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:12:20.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:12:20.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:12:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:12:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:12:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:12:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:12:20.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:12:20.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:12:20.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:12:20.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:12:20.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:12:20.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:12:20.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:12:20.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:12:20.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:12:20.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:12:20.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:12:20.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:12:20.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:12:20.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:12:20.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:12:20.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:12:20.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:12:20.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:12:20.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:12:20.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:12:20.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:12:20.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:12:20.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:12:20.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:12:20.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:12:20.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:12:20.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:12:20.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:12:20.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:12:20.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:12:20.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:12:20.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:12:20.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:12:20.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:12:20.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:12:20.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:12:20.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:12:20.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:12:20.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:12:20.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:12:20.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:12:20.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:12:20.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:12:20.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:12:20.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:12:20.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:12:20.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:12:20.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:12:20.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:12:20.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:12:20.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:12:20.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:12:20.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:12:20.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:12:20.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:12:20.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:12:20.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:12:20.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:12:20.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:12:20.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:12:20.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:12:20.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:12:20.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:12:20.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:12:20.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:12:20.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:12:20.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:12:20.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:12:20.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:12:20.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:12:20.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:12:20.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:12:20.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:12:20.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:12:20.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:12:20.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:12:20.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:12:20.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:12:20.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:12:20.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:12:20.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:12:20.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:12:20.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:12:20.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:12:20.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:12:20.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:12:20.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:12:20.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:12:20.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:12:20.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:12:20.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:12:20.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:12:20.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:12:20.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:12:20.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:12:20.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:12:20.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:12:20.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:12:20.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:12:20.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:12:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:12:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:12:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:12:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:12:20.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:12:20.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:12:20.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:12:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:12:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:12:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:12:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:12:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:12:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:12:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:12:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:12:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:12:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:12:20.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:12:20.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:12:20.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:12:20.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:12:20.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:12:20.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:12:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:12:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:12:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:12:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:12:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:12:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:12:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:12:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:12:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:12:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:12:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:12:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:12:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:12:20.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:12:20.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:12:20.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:12:20.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:12:20.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:12:20.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:12:20.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:12:20.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:12:20.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:12:20.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:12:20.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:12:20.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:12:20.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:12:20.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:12:20.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:12:20.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:12:20.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:12:20.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:12:20.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:12:20.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:12:20.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:12:20.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:12:20.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:12:20.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:12:20.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:12:20.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:12:20.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:12:20.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:12:20.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:12:20.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:12:20.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:12:20.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:12:20.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:12:20.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:12:20.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:12:20.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:12:20.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:12:20.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:12:20.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:12:20.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:12:20.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:12:20.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:12:20.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:12:20.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:12:20.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:12:20.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:12:20.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:12:20.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:12:20.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:12:20.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:12:20.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:12:20.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:12:20.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:12:20.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:12:20.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:12:20.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:12:20.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:12:20.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:12:20.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:12:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:12:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:12:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:12:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:12:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:12:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:12:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:12:20.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:12:20.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:12:20.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:12:20.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:12:20.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:12:20.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:12:20.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:12:20.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:12:20.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:12:20.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:12:20.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:12:20.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:12:20.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:12:20.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:12:20.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:12:20.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:12:20.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:12:20.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:12:20.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:12:20.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:12:20.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:12:20.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:12:20.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:12:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:12:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:12:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:12:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:12:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:12:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:12:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:12:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:12:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:12:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:12:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:12:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:12:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:12:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:12:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:12:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:12:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.679 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:12:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:12:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:12:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:12:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:12:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:12:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:12:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:12:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.682 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:12:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:12:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:12:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:12:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:12:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:12:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:12:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:12:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:12:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:12:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:12:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:12:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:12:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:12:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:12:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:12:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:12:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:12:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:12:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:12:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:12:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:12:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:12:20.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:12:20.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:12:20.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:12:20.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:12:20.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:12:20.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:12:20.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:12:20.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:12:20.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:12:20.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:12:20.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:12:20.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:12:20.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:12:20.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:12:20.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:12:20.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:12:20.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:12:20.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:12:20.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:12:20.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:12:20.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:12:20.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:12:20.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:12:20.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:12:20.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:12:20.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:12:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:12:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:12:20.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:12:20.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:12:20.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:12:20.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:12:20.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:12:20.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:12:20.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:12:20.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:12:20.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:12:20.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:12:20.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:12:20.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:12:20.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:12:20.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:12:20.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:12:20.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:12:20.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:12:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:12:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:12:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:12:20.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:12:20.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:12:20.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:12:20.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:12:20.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:12:20.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:12:20.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:12:20.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:12:20.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:12:20.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:20.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:20.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:12:20.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:12:20.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:12:20.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:12:20.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:12:20.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:12:20.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:20.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:12:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:12:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:12:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:12:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:12:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:12:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:12:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:12:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:12:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:12:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:12:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:12:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:12:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:12:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:12:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:12:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:12:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:12:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:12:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:12:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:12:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:12:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:12:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:12:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:12:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:12:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:12:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:12:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:12:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:12:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:12:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:12:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:12:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:12:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:12:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:12:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:12:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:12:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:12:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:12:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:12:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:12:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:12:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:12:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:12:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:12:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:12:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:12:24.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:12:24.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:12:24.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:12:24.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:12:24.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:12:24.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:12:24.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:12:24.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:12:24.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:12:24.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:12:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:12:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:12:24.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:12:24.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:12:24.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:12:24.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:12:24.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:12:24.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:12:24.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:12:24.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:12:24.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:12:24.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:12:24.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:12:24.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:12:24.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:12:24.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:12:24.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:12:24.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:12:24.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:12:24.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:12:24.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:12:24.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:12:24.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:12:24.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:12:24.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:12:24.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:12:24.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:12:24.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:12:24.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:12:24.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:12:24.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:12:24.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:24.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:12:24.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:24.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:24.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:12:25.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:12:25.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:12:25.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:12:25.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:12:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:12:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:12:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:12:25.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:12:25.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:12:25.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:12:25.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:12:25.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:12:25.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:12:25.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:12:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:12:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.004 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:12:25.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:12:25.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:12:25.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:12:25.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:12:25.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:12:25.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:12:25.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:12:25.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:12:25.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:12:25.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:12:25.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:12:25.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:12:25.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:12:25.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:12:25.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:12:25.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:12:25.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:12:25.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:12:25.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:12:25.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:12:25.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:12:25.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:12:25.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:12:25.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:12:25.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:12:25.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:12:25.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:12:25.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:12:25.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:12:25.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:12:25.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:12:25.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:12:25.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:12:25.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:12:25.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:12:25.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:12:25.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:12:25.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:12:25.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:12:25.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:12:25.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:12:25.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:12:25.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:12:25.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:12:25.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:12:25.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:12:25.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:12:25.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:12:25.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:12:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:12:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:12:25.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:12:25.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:12:25.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:12:25.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:12:25.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:12:25.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:12:25.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:12:25.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:12:25.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:12:25.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:12:25.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:12:25.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:12:25.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:12:25.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:12:25.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:12:25.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:12:25.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:12:25.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:12:25.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:12:25.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:12:25.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:12:25.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:12:25.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:12:25.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:12:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:12:25.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:12:25.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:12:25.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:12:25.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:12:25.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:12:25.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:12:25.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:12:25.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:12:25.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:12:25.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:12:25.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:12:25.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:12:25.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:12:25.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:12:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:12:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:12:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:12:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:12:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:12:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:12:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:12:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:12:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:12:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:12:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:12:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:12:25.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:12:25.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:12:25.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:12:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:12:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:12:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:12:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:12:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:12:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:12:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:12:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:12:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:12:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:12:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:12:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:12:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:12:25.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:12:25.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:12:25.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:12:25.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:12:25.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:12:25.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:12:25.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:12:25.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:12:25.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:12:25.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.074 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:12:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:12:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:12:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:12:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:12:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:12:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:12:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:12:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:12:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:12:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:12:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:12:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:12:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:12:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:12:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:12:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:12:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:12:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:12:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:12:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:12:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:12:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:12:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:12:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:12:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:12:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:12:25.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:12:25.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:12:25.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:12:25.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:12:25.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:12:25.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.084 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:12:25.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:12:25.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:12:25.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:12:25.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:12:25.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:12:25.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:12:25.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:12:25.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:12:25.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:12:25.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:12:25.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:12:25.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:12:25.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:12:25.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:12:25.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:12:25.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:12:25.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:12:25.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:12:25.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:12:25.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:12:25.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:12:25.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:12:25.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:12:25.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:12:25.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:12:25.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:12:25.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:12:25.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:12:25.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:12:25.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:12:25.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:12:25.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:12:25.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:12:25.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:12:25.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:12:25.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:12:25.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:12:25.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:12:25.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:12:25.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:12:25.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:12:25.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:12:25.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:12:25.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:12:25.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:12:25.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:12:25.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:12:25.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:12:25.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:12:25.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:12:25.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:12:25.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:12:25.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:12:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:12:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:12:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:12:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:12:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:12:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:12:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:12:25.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:12:25.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:12:25.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:12:25.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:12:25.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:12:25.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:12:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:12:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:12:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:12:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:12:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:12:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:12:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:12:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:12:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:12:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:12:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:12:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:12:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:12:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:12:25.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:12:25.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:12:25.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:12:25.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:12:25.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:12:25.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:12:25.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:12:25.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:12:25.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:12:25.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:12:25.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:12:25.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:12:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:12:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:12:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:12:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:12:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:12:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:12:25.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:12:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:12:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:12:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:12:25.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:12:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:12:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:12:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:12:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:12:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:12:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:12:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:12:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:12:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:12:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:12:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:12:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:12:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:12:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:12:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:12:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:12:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:12:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:12:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:12:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:12:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:12:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:12:25.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:12:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:12:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:12:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:12:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:12:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:12:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:12:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:12:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:12:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:12:25.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:12:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:12:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:12:25.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:12:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:12:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:12:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:12:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:12:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:12:25.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:12:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:12:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:12:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:12:25.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:12:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:12:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:12:25.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:12:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:12:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:12:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:12:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:12:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:12:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:12:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:12:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:12:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:12:25.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:12:25.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:12:25.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:12:25.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:12:25.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:12:25.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:12:25.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:12:25.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:12:25.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:12:25.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:12:25.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:12:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:12:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:12:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:12:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:12:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:12:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:12:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:12:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:12:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:12:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:12:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:12:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:12:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:12:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:12:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:12:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:12:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:12:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:12:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:12:25.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:12:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:12:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:12:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:12:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:12:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:12:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:12:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:12:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:12:25.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:12:25.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:12:25.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:12:25.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:12:25.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:12:25.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:12:25.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:12:25.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:12:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:12:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:12:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:12:25.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:12:25.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:12:25.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:12:25.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:12:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:12:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:12:25.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:12:25.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:12:25.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:12:25.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:12:25.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:12:25.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:12:25.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:12:25.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:12:25.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:12:25.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:12:25.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:12:25.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:12:25.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:12:25.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:12:25.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:12:25.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:12:25.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:12:25.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:12:25.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:12:25.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:12:25.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:12:25.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:12:25.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:12:25.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:12:25.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:12:25.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:12:25.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:12:25.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:12:25.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:12:25.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:12:25.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:12:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:12:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:12:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:12:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:12:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:12:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:12:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:12:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:12:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:12:25.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:12:25.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:12:25.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:12:25.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:12:25.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:12:25.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:12:25.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:12:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:12:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:12:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:12:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:12:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:12:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:12:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:12:25.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:12:25.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:12:25.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:12:25.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:12:25.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:12:25.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:12:25.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:12:25.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:12:25.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:12:25.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:12:25.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:12:25.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:12:25.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:12:25.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:12:25.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:12:25.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:12:25.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:12:25.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:12:25.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:12:25.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:12:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:12:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:12:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:12:25.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:12:25.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:12:25.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:12:25.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:12:25.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:12:25.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:12:25.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:12:25.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:12:25.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:12:25.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:12:25.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:12:25.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:12:25.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:12:25.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:12:25.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:12:25.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:12:25.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:12:25.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:12:25.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:12:25.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:12:25.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:12:25.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:12:25.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:12:25.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:12:25.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:12:25.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:12:25.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:12:25.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:12:25.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:12:25.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:12:25.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:12:25.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:12:25.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:12:25.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:12:25.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:12:25.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:12:25.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:12:25.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:12:25.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:12:25.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:12:25.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:12:25.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:12:25.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:12:25.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:12:25.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:12:25.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:12:25.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:12:25.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:12:25.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:12:25.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:12:25.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:12:25.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:12:25.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:12:25.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:12:25.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:12:25.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:12:25.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:12:25.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:12:25.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:12:25.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:12:25.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:12:25.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:12:25.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:12:25.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:12:25.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:12:25.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.193 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:12:25.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:12:25.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:12:25.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:12:25.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:12:25.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:12:25.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:12:25.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:12:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.196 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:12:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:12:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:12:25.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:12:25.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:12:25.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:12:25.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:12:25.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:12:25.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:12:25.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:12:25.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:12:25.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:12:25.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:12:25.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:12:25.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:12:25.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:12:25.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:12:25.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:12:25.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:12:25.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:12:25.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:12:25.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:12:25.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:12:25.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:12:25.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:12:25.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:12:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:12:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:12:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:12:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:12:25.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:12:25.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:12:25.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:12:25.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:12:25.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:12:25.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:12:25.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:12:25.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:12:25.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:12:25.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:12:25.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:12:25.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:12:25.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:12:25.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:12:25.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:12:25.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:12:25.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:12:25.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:12:25.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:12:25.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:12:25.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:12:25.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:12:25.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:12:25.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:12:25.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:12:25.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:12:25.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:12:25.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:12:25.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:12:25.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:12:25.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:12:25.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:12:25.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:12:25.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:12:25.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:12:25.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:12:25.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:12:25.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:12:25.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:12:25.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:12:25.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:12:25.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:12:25.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:12:25.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:12:25.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:12:25.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:12:25.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:12:25.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:12:25.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:12:25.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:12:25.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:12:25.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:12:25.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:12:25.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:12:25.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:12:25.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:12:25.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:12:25.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:12:25.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:12:25.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:12:25.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:12:25.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:12:25.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:12:25.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:12:25.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:12:25.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:12:25.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:12:25.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:12:25.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:12:25.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:12:25.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:12:25.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:12:25.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:12:25.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:12:25.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:12:25.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:12:25.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:12:25.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:12:25.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:12:25.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:12:25.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:12:25.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:12:25.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.230 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:12:25.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:12:25.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:12:25.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:12:25.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:12:25.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:12:25.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:12:25.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:12:25.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:12:25.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:12:25.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:12:25.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:12:25.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:12:25.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:12:25.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:12:25.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:12:25.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.235 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:12:25.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:12:25.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:12:25.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:12:25.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:12:25.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:12:25.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:12:25.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:12:25.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:12:25.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:12:25.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:12:25.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:12:25.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:12:25.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:12:25.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:12:25.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:12:25.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.239 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:12:25.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:12:25.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:12:25.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:12:25.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:12:25.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:12:25.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:12:25.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:12:25.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:12:25.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:12:25.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:12:25.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:12:25.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:12:25.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:12:25.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:12:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:12:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:12:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:12:25.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:12:25.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:12:25.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:12:25.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:12:25.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:12:25.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:12:25.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:12:25.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:12:25.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:12:25.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:12:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:12:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:12:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:12:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:12:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:12:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:12:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:12:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:12:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:12:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:12:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:12:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:12:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:12:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:12:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:12:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:12:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:12:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:12:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:12:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:12:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:12:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:12:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:12:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:12:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:12:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:12:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:12:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:12:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:12:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:12:25.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:12:25.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:12:25.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:12:25.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:12:25.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:12:25.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:12:25.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:12:25.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:12:25.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:12:25.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:12:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:12:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:12:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:12:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:12:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:12:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:12:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:12:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:12:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:12:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:12:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:12:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:12:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:12:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:12:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:12:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:12:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:12:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:12:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:12:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:12:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:12:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:12:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:12:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:12:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:12:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:12:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:12:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:12:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:12:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:12:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:12:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:12:25.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:12:25.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:12:25.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:12:25.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:12:25.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:12:25.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:12:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:12:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:12:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:12:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:12:25.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:12:25.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:12:25.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:12:25.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:12:25.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:12:25.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:12:25.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:12:25.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:12:25.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:12:25.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:12:25.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:12:25.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:12:25.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:12:25.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:12:25.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:12:25.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:12:25.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:12:25.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:12:25.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:12:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:12:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:12:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:12:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:12:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:12:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:12:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:12:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:12:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:12:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:12:25.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:12:25.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:12:25.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:12:25.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:12:25.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:12:25.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:12:25.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:12:25.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:12:25.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:12:25.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:12:25.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:12:25.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:12:25.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:12:25.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:12:25.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:12:25.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:12:25.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:12:25.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:12:25.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:12:25.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:12:25.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:12:25.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:12:25.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:12:25.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:12:25.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:12:25.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:12:25.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:12:25.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:12:25.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:12:25.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:12:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:12:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:12:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:12:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:12:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:12:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:12:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:12:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:12:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:12:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:12:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:12:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:12:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:12:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:12:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:12:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:12:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:12:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:12:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:12:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:12:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:12:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:12:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:12:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:12:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:12:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:12:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:12:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:12:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:12:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:12:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:12:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.301 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:12:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:12:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:12:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:12:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:12:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:12:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:12:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:12:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:12:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:12:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:12:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:12:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:12:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:12:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:12:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:12:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:12:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:12:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:12:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:12:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:12:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:12:25.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:12:25.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:12:25.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.308 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:12:25.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:12:25.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:12:25.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:12:25.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:12:25.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:12:25.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:12:25.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:12:25.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:12:25.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:12:25.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:12:25.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:12:25.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:12:25.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:12:25.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:12:25.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:12:25.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:12:25.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:12:25.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:12:25.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:12:25.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:12:25.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:12:25.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:12:25.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:12:25.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:12:25.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:12:25.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:12:25.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:12:25.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:12:25.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:12:25.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:12:25.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:12:25.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:12:25.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:12:25.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:12:25.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:12:25.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:12:25.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:12:25.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:12:25.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:12:25.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:12:25.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:12:25.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:12:25.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:12:25.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:12:25.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:12:25.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:12:25.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:12:25.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:12:25.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:12:25.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:12:25.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:12:25.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:12:25.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:12:25.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:12:25.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:12:25.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:12:25.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:12:25.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:12:25.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:12:25.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:12:25.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:12:25.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:12:25.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:12:25.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:12:25.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:12:25.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:12:25.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:12:25.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:12:25.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:12:25.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:12:25.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:12:25.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:12:25.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:12:25.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:12:25.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:12:25.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:12:25.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:12:25.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:12:25.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:12:25.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:12:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:12:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:12:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:12:25.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:12:25.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:12:25.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:12:25.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:12:25.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:12:25.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:12:25.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:12:25.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:12:25.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:12:25.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:12:25.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:12:25.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:12:25.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:12:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:12:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:12:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:12:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:12:25.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:12:25.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:12:25.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:12:25.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:12:25.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:12:25.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:12:25.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:12:25.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:12:25.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:12:25.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:12:25.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:12:25.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:12:25.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:12:25.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:12:25.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:12:25.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:12:25.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:12:25.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:12:25.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:12:25.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:12:25.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:12:25.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:12:25.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:12:25.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:12:25.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:12:25.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:12:25.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:12:25.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:12:25.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:12:25.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:12:25.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:12:25.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:12:25.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:12:25.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:12:25.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:12:25.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:12:25.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:12:25.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:12:25.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:12:25.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:12:25.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:12:25.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:12:25.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:12:25.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:12:25.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:12:25.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:12:25.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:12:25.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:12:25.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:12:25.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:12:25.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:12:25.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:12:25.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:12:25.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:12:25.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:12:25.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:12:25.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:12:25.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:12:25.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:12:25.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:12:25.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:12:25.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:12:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:12:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:12:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:12:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:12:25.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:12:25.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:12:25.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:12:25.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:12:25.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:12:25.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:12:25.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:12:25.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:12:25.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:12:25.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.361 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:12:25.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:12:25.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:12:25.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:12:25.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:12:25.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:12:25.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:12:25.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:12:25.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:12:25.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:12:25.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:12:25.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:12:25.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:12:25.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:12:25.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:12:25.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:12:25.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:12:25.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:12:25.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:12:25.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:12:25.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:12:25.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:12:25.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:12:25.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:12:25.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:12:25.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:12:25.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:12:25.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:12:25.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:12:25.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:12:25.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:12:25.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:12:25.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:12:25.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:12:25.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:12:25.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:12:25.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:12:25.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:12:25.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:12:25.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:12:25.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:12:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:12:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:12:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:12:25.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:12:25.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:12:25.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:12:25.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:12:25.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:12:25.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:12:25.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:12:25.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:12:25.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:12:25.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:12:25.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:12:25.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:12:25.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:12:25.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:12:25.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:12:25.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:12:25.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:12:25.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:12:25.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:12:25.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:12:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:12:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:12:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:12:25.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:12:25.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:12:25.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:12:25.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:12:25.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:12:25.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:12:25.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:12:25.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:12:25.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:12:25.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:12:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:12:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:12:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:12:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:12:25.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:12:25.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:12:25.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:12:25.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:12:25.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:12:25.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:12:25.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:12:25.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:12:25.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:12:25.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:12:25.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:12:25.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:12:25.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:12:25.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:12:25.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:12:25.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:12:25.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:12:25.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:12:25.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:12:25.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:12:25.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:12:25.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:12:25.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:12:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:12:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:12:25.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:12:25.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:12:25.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:12:25.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:12:25.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:12:25.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:12:25.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:12:25.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:12:25.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:12:25.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:12:25.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:12:25.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:12:25.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:12:25.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:12:25.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:12:25.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:12:25.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:12:25.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:12:25.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:12:25.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:12:25.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:12:25.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:12:25.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:12:25.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:12:25.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:12:25.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:12:25.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:12:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:12:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:12:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:12:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:12:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:12:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:12:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:12:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:12:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:12:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:12:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:12:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:12:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:12:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:12:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:12:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:12:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:12:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:12:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:12:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:12:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:12:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:12:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:12:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:12:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:12:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:12:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:12:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:12:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:12:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:12:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:12:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:12:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:12:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:12:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:12:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:12:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:12:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:12:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:12:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:12:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:12:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:12:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:12:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:12:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:12:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:12:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:12:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:12:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:12:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:12:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:12:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:12:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:12:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:12:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:12:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:12:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:12:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:12:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:12:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:12:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:12:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:12:25.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:12:25.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:12:25.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:12:25.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:12:25.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:12:25.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:12:25.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:12:25.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:12:25.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:12:25.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:12:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:12:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:12:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:12:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:12:25.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:12:25.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:12:25.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:12:25.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:12:25.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:12:25.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:12:25.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:12:25.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:12:25.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:12:25.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:12:25.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:12:25.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:12:25.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:12:25.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:12:25.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:12:25.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:12:25.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:12:25.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:12:25.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:12:25.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:12:25.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:12:25.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:12:25.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:12:25.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:12:25.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:12:25.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:12:25.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:12:25.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:12:25.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:12:25.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:12:25.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:12:25.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:12:25.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:12:25.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:12:25.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:12:25.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:12:25.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:12:25.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:12:25.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:12:25.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:12:25.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:12:25.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:12:25.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:12:25.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:12:25.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:12:25.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:12:25.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:12:25.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:12:25.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:12:25.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:12:25.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:12:25.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:12:25.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:12:25.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:12:25.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:12:25.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:12:25.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:12:25.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:12:25.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:12:25.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:12:25.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:12:25.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:12:25.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:12:25.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:12:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:12:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:12:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:12:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:12:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:12:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:12:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:12:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:12:25.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:12:25.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:12:25.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:12:25.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:12:25.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:12:25.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:12:25.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:12:25.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:12:25.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:12:25.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:12:25.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:12:25.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:12:25.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:12:25.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:12:25.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:12:25.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:12:25.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:12:25.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:12:25.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:12:25.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:12:25.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:12:25.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:12:25.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:12:25.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:12:25.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:12:25.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:12:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:12:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:12:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:12:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:12:25.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:12:25.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:12:25.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:12:25.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:12:25.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:12:25.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:12:25.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:12:25.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:12:25.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:12:25.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:12:25.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:12:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:12:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:12:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:12:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:12:25.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:12:25.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:12:25.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:12:25.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:12:25.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:12:25.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:12:25.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:12:25.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:12:25.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:12:25.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:12:25.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:12:25.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:12:25.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:12:25.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:12:25.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:12:25.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:12:25.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:12:25.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:12:25.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:12:25.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:12:25.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:12:25.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:12:25.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:12:25.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:12:25.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:12:25.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:12:25.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:12:25.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:12:25.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:12:25.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:12:25.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:12:25.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:12:25.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:12:25.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:12:25.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:12:25.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:12:25.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:12:25.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:12:25.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:12:25.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:12:25.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:12:25.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:12:25.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:12:25.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:12:25.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:12:25.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:12:25.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:12:25.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:12:25.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:12:25.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:12:25.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:12:25.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:12:25.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:12:25.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:12:25.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:12:25.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:12:25.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:12:25.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:12:25.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:12:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:12:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:12:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:12:25.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:12:25.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:12:25.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:12:25.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:12:25.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:12:25.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:12:25.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:12:25.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:12:25.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:12:25.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:12:25.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:12:25.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:12:25.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:12:25.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:12:25.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:12:25.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:12:25.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:12:25.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:12:25.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:12:25.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:12:25.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:12:25.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:12:25.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:12:25.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:12:25.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:12:25.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:12:25.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:12:25.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:12:25.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:12:25.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:12:25.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:12:25.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:12:25.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:12:25.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:12:25.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:12:25.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:12:25.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:12:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:12:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:12:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:12:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:12:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:12:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:12:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:12:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:12:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:12:25.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:12:25.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:12:25.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:12:25.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:12:25.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:12:25.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:12:25.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:12:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:12:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:12:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:12:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:12:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:12:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:12:25.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:12:25.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:12:25.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:12:25.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:12:25.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:12:25.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:12:25.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:12:25.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:12:25.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:12:25.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:12:25.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:12:25.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:12:25.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:12:25.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:12:25.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:12:25.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:12:25.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:12:25.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:12:25.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:12:25.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:12:25.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:12:25.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:12:25.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:12:25.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:12:25.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:12:25.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:12:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:12:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:12:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:12:25.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:12:25.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:12:25.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:12:25.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:12:25.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:12:25.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:12:25.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:12:25.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:12:25.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:12:25.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:12:25.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:12:25.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:12:25.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:12:25.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:12:25.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:12:25.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:12:25.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:12:25.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:12:25.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:12:25.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:12:25.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:12:25.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:12:25.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:12:25.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:12:25.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:12:25.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:12:25.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:12:25.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:12:25.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:12:25.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:12:25.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:12:25.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:12:25.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:12:25.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:12:25.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:12:25.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:12:25.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:12:25.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:12:25.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:12:25.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:12:25.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:12:25.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:12:25.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:12:25.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:12:25.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:12:25.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:12:25.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:12:25.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:12:25.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:12:25.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:12:25.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:12:25.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:12:25.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:12:25.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:12:25.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:12:25.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:12:25.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:12:25.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:12:25.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:12:25.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:12:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:12:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:12:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:12:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:12:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:12:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:12:25.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:12:25.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:12:25.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:12:25.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:12:25.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:12:25.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:12:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:12:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:12:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:12:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:12:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:12:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:12:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:12:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:12:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:12:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:12:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:12:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:12:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:12:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:12:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:12:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:12:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:12:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:12:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:12:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:12:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:12:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:12:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:12:25.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:12:25.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:12:25.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:12:25.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:12:25.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:12:25.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:12:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:12:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:12:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:12:25.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:12:25.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:12:25.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:12:25.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:12:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:12:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:12:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:12:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:12:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:12:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:12:25.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:12:25.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:12:25.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:12:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:12:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:12:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:12:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:12:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:12:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:12:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:12:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:12:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:12:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:12:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:12:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:12:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:12:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:12:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:12:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:12:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:12:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:12:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:12:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:12:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:12:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:12:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:12:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:12:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:12:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:12:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:12:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:12:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:12:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:12:25.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:12:25.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:12:25.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:12:25.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:12:25.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:12:25.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:12:25.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:12:25.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:12:25.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:12:25.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:12:25.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:12:25.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:12:25.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:12:25.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:12:25.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:12:25.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:12:25.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:12:25.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:25.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:12:25.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:12:25.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:12:25.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:12:25.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:12:25.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:12:25.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:12:25.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:25.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:12:25.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:25.565 INFO:tasks.workunit.client.0.smithi189.stdout: marking collection for removal 2020-11-16T21:12:25.565 INFO:tasks.workunit.client.0.smithi189.stdout:setting '_remove' omap key 2020-11-16T21:12:25.565 INFO:tasks.workunit.client.0.smithi189.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-11-16T21:12:25.565 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:12:25.565 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:25.565 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:25.566 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up out weight 0 up_from 40 up_thru 34 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/186933,v1:127.0.0.1:6811/186933] [v2:127.0.0.1:6812/186933,v1:127.0.0.1:6813/186933] exists,up 929f76af-e446-4277-aece-f953facd3c63 2020-11-16T21:12:25.566 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:12:25.566 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:43:06.227685+0000 0'0 39:19 [2,0] 2 [2,0] 2 0'0 2020-11-16T20:41:56.209471+0000 0'0 2020-11-16T20:41:56.209471+0000 0 2020-11-16T21:12:25.566 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 566 0 0 0 0 1477826 0 0 66 66 stale+active+clean 2020-11-16T20:41:53.479578+0000 36'566 36:589 [1] 1 [1] 1 0'0 2020-11-16T20:41:51.380521+0000 0'0 2020-11-16T20:41:51.380521+0000 0 2020-11-16T21:12:25.566 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:43:06.228231+0000 0'0 39:31 [2,0] 2 [2,0] 2 0'0 2020-11-16T20:41:53.771402+0000 0'0 2020-11-16T20:41:53.771402+0000 0 2020-11-16T21:12:25.566 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:12:25.566 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:12:25.567 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:12:25.567 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:43:06.227685+0000 0'0 39:20 [2,0] 2 [2,0] 2 0'0 2020-11-16T20:41:56.209471+0000 0'0 2020-11-16T20:41:56.209471+0000 0 2020-11-16T21:12:25.567 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 566 0 0 0 0 1477826 0 0 66 66 stale+active+clean 2020-11-16T20:41:53.479578+0000 36'566 36:589 [1] 1 [1] 1 0'0 2020-11-16T20:41:51.380521+0000 0'0 2020-11-16T20:41:51.380521+0000 0 2020-11-16T21:12:25.567 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 574 0 0 0 0 587776 0 0 74 74 active+clean 2020-11-16T20:43:06.228231+0000 40'574 40:606 [2,0] 2 [2,0] 2 0'0 2020-11-16T20:41:53.771402+0000 0'0 2020-11-16T20:41:53.771402+0000 0 2020-11-16T21:12:25.567 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:12:25.567 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:12:25.567 INFO:tasks.workunit.client.0.smithi189.stdout:get_pg_num_history pg_num_history pg_num_history(e44 pg_nums {1={21=1},2={26=1},3={31=1}} deleted_pools ) 2020-11-16T21:12:25.567 INFO:tasks.workunit.client.0.smithi189.stdout:Importing pgid 2.0 2020-11-16T21:12:25.568 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:00f101c6:::obj-1-121:head# 2020-11-16T21:12:25.568 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.568 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0168051e:::obj-1-571:head# 2020-11-16T21:12:25.568 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.568 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:01b67738:::obj-1-182:head# 2020-11-16T21:12:25.568 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.568 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:040dbc59:::obj-1-23:head# 2020-11-16T21:12:25.569 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.569 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0415ce6e:::obj-1-20:head# 2020-11-16T21:12:25.569 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.569 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:058b28d7:::obj-1-572:head# 2020-11-16T21:12:25.569 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.569 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:059d2e5c:::obj-1-87:head# 2020-11-16T21:12:25.569 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.569 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:05a823ca:::obj-1-52:head# 2020-11-16T21:12:25.570 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.570 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:05b8b7fe:::obj-1-119:head# 2020-11-16T21:12:25.570 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.570 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:05c9492b:::obj-1-78:head# 2020-11-16T21:12:25.570 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.570 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:05e91e96:::obj-1-393:head# 2020-11-16T21:12:25.570 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.571 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:07bdb780:::obj-1-204:head# 2020-11-16T21:12:25.571 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.571 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:07cfbd49:::obj-1-318:head# 2020-11-16T21:12:25.571 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.571 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:084a829f:::obj-1-112:head# 2020-11-16T21:12:25.571 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:25.571 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:08cedd34:::obj-1-438:head# 2020-11-16T21:12:25.572 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.718 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0941af69:::obj-1-117:head# 2020-11-16T21:12:29.718 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.718 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:09607265:::obj-1-429:head# 2020-11-16T21:12:29.719 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.719 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0be86f7e:::obj-1-568:head# 2020-11-16T21:12:29.719 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.719 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0ca5819d:::obj-1-327:head# 2020-11-16T21:12:29.719 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.719 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0cd02051:::obj-1-208:head# 2020-11-16T21:12:29.719 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.720 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0d0264b5:::obj-1-171:head# 2020-11-16T21:12:29.720 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.720 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0d195060:::obj-1-420:head# 2020-11-16T21:12:29.720 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.720 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0d362b19:::obj-1-219:head# 2020-11-16T21:12:29.720 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.720 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0dea6f4d:::obj-1-61:head# 2020-11-16T21:12:29.721 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.721 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0ecd8390:::obj-1-132:head# 2020-11-16T21:12:29.721 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.721 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0f49e2fa:::obj-1-200:head# 2020-11-16T21:12:29.721 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.721 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0f7bb95f:::obj-1-268:head# 2020-11-16T21:12:29.721 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.721 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0fabf252:::obj-1-441:head# 2020-11-16T21:12:29.722 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.722 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:0fb1697a:::obj-1-443:head# 2020-11-16T21:12:29.722 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.722 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:105d3efe:::obj-1-490:head# 2020-11-16T21:12:29.722 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.722 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:10f52c01:::obj-1-449:head# 2020-11-16T21:12:29.722 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.723 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:11965425:::obj-1-335:head# 2020-11-16T21:12:29.723 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.723 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:11c9f5f8:::obj-1-565:head# 2020-11-16T21:12:29.723 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.723 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:11cfcfb6:::obj-1-588:head# 2020-11-16T21:12:29.723 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.723 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:11d46f0a:::obj-1-387:head# 2020-11-16T21:12:29.724 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.724 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:11ebb0d6:::obj-1-214:head# 2020-11-16T21:12:29.724 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.724 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1295e402:::obj-1-178:head# 2020-11-16T21:12:29.724 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.724 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:12c996c1:::obj-1-517:head# 2020-11-16T21:12:29.724 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.724 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:133dec15:::obj-1-475:head# 2020-11-16T21:12:29.725 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.725 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1475c981:::obj-1-505:head# 2020-11-16T21:12:29.725 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.725 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:148e92a4:::obj-1-509:head# 2020-11-16T21:12:29.725 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.725 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1490b121:::obj-1-172:head# 2020-11-16T21:12:29.725 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.726 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:14ce082c:::obj-1-99:head# 2020-11-16T21:12:29.726 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.726 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1552ad99:::obj-1-41:head# 2020-11-16T21:12:29.726 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.726 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:15544270:::obj-1-271:head# 2020-11-16T21:12:29.726 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.726 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1589d957:::obj-1-86:head# 2020-11-16T21:12:29.726 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.727 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:16db3947:::obj-1-477:head# 2020-11-16T21:12:29.727 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.727 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:16dc98bd:::obj-1-375:head# 2020-11-16T21:12:29.727 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.727 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:17e54db3:::obj-1-196:head# 2020-11-16T21:12:29.727 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.727 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:17f2ad3f:::obj-1-10:head# 2020-11-16T21:12:29.728 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.728 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:18075e41:::obj-1-55:head# 2020-11-16T21:12:29.728 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.728 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:18086253:::obj-1-71:head# 2020-11-16T21:12:29.728 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.728 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:191cf8d8:::obj-1-372:head# 2020-11-16T21:12:29.728 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.729 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:19311f70:::obj-1-454:head# 2020-11-16T21:12:29.729 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.729 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:198318e4:::obj-1-464:head# 2020-11-16T21:12:29.729 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.729 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1999d806:::obj-1-163:head# 2020-11-16T21:12:29.729 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.729 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1a92cab7:::obj-1-412:head# 2020-11-16T21:12:29.729 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.730 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1b16c0f6:::obj-1-210:head# 2020-11-16T21:12:29.730 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.730 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1b53ed46:::obj-1-577:head# 2020-11-16T21:12:29.730 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.730 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1b5c5a8d:::obj-1-556:head# 2020-11-16T21:12:29.730 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.731 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1c12c05e:::obj-1-79:head# 2020-11-16T21:12:29.731 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.731 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1c571ba5:::obj-1-262:head# 2020-11-16T21:12:29.731 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.731 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1c5f42b7:::obj-1-92:head# 2020-11-16T21:12:29.731 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.731 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1c6ad881:::obj-1-312:head# 2020-11-16T21:12:29.731 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.732 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1c89877b:::obj-1-142:head# 2020-11-16T21:12:29.732 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.732 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1e4168ca:::obj-1-282:head# 2020-11-16T21:12:29.732 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.732 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1e424445:::obj-1-484:head# 2020-11-16T21:12:29.732 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.732 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1e467959:::obj-1-22:head# 2020-11-16T21:12:29.733 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.733 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1eaf7e7c:::obj-1-392:head# 2020-11-16T21:12:29.733 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.733 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1f06c86d:::obj-1-307:head# 2020-11-16T21:12:29.733 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.733 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1f2ee1f8:::obj-1-306:head# 2020-11-16T21:12:29.733 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.733 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:1f4ef040:::obj-1-349:head# 2020-11-16T21:12:29.734 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.734 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2039481f:::obj-1-545:head# 2020-11-16T21:12:29.734 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.734 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:22233df0:::obj-1-543:head# 2020-11-16T21:12:29.734 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.734 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2225184e:::obj-1-266:head# 2020-11-16T21:12:29.734 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.735 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:234766bb:::obj-1-281:head# 2020-11-16T21:12:29.735 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.735 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2372e022:::obj-1-205:head# 2020-11-16T21:12:29.735 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.735 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2379547b:::obj-1-113:head# 2020-11-16T21:12:29.735 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.735 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:23ae14bf:::obj-1-531:head# 2020-11-16T21:12:29.735 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.736 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:23bc5f90:::obj-1-452:head# 2020-11-16T21:12:29.736 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.736 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:23fbef3d:::obj-1-82:head# 2020-11-16T21:12:29.736 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.736 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:24106c2b:::obj-1-232:head# 2020-11-16T21:12:29.736 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.736 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2529b4e4:::obj-1-75:head# 2020-11-16T21:12:29.737 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.737 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:25a24ae7:::obj-1-459:head# 2020-11-16T21:12:29.737 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.737 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:26627aa9:::obj-1-7:head# 2020-11-16T21:12:29.737 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.737 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:26b77221:::obj-1-227:head# 2020-11-16T21:12:29.737 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.738 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:27450826:::obj-1-344:head# 2020-11-16T21:12:29.738 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.738 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:278905af:::obj-1-592:head# 2020-11-16T21:12:29.738 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.738 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:27ad0d2d:::obj-1-136:head# 2020-11-16T21:12:29.738 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.738 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:28a8c213:::obj-1-118:head# 2020-11-16T21:12:29.739 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.739 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:28c0c009:::obj-1-74:head# 2020-11-16T21:12:29.739 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.739 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2a214708:::obj-1-260:head# 2020-11-16T21:12:29.739 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.739 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2a67e195:::obj-1-474:head# 2020-11-16T21:12:29.739 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.740 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2aa852ff:::obj-1-5:head# 2020-11-16T21:12:29.740 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.740 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2ba1fb52:::obj-1-563:head# 2020-11-16T21:12:29.740 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.740 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2bae8cb9:::obj-1-320:head# 2020-11-16T21:12:29.740 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.740 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2c569d02:::obj-1-139:head# 2020-11-16T21:12:29.740 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.741 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2c71d691:::obj-1-201:head# 2020-11-16T21:12:29.741 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.741 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2c88e92c:::obj-1-389:head# 2020-11-16T21:12:29.741 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.741 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2d659109:::obj-1-580:head# 2020-11-16T21:12:29.741 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.741 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2d74ea60:::obj-1-230:head# 2020-11-16T21:12:29.742 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.742 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2de96edf:::obj-1-233:head# 2020-11-16T21:12:29.742 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.742 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2e43da40:::obj-1-382:head# 2020-11-16T21:12:29.742 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.742 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2e530c9f:::obj-1-358:head# 2020-11-16T21:12:29.742 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.742 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:2ed02239:::obj-1-81:head# 2020-11-16T21:12:29.743 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.743 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:30264a0c:::obj-1-373:head# 2020-11-16T21:12:29.743 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.743 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:302ca5e7:::obj-1-206:head# 2020-11-16T21:12:29.743 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.743 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:309878e8:::obj-1-508:head# 2020-11-16T21:12:29.743 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.744 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:309a0ac3:::obj-1-188:head# 2020-11-16T21:12:29.744 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.744 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:30f030e9:::obj-1-53:head# 2020-11-16T21:12:29.744 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.744 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:31242a73:::obj-1-528:head# 2020-11-16T21:12:29.744 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.744 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:31463825:::obj-1-34:head# 2020-11-16T21:12:29.744 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.745 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3166dc86:::obj-1-355:head# 2020-11-16T21:12:29.745 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.745 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:31baa2fa:::obj-1-252:head# 2020-11-16T21:12:29.745 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.745 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:31d8935e:::obj-1-193:head# 2020-11-16T21:12:29.745 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.745 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:32670b14:::obj-1-305:head# 2020-11-16T21:12:29.746 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.746 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:33da02fa:::obj-1-114:head# 2020-11-16T21:12:29.746 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.746 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:34373133:::obj-1-446:head# 2020-11-16T21:12:29.746 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.746 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:347b7083:::obj-1-89:head# 2020-11-16T21:12:29.746 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.746 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:349125c1:::obj-1-148:head# 2020-11-16T21:12:29.747 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.747 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:34b80eee:::obj-1-138:head# 2020-11-16T21:12:29.747 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.747 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:34db48f8:::obj-1-49:head# 2020-11-16T21:12:29.747 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.747 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3669ff0c:::obj-1-313:head# 2020-11-16T21:12:29.747 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.748 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:36c1b610:::obj-1-249:head# 2020-11-16T21:12:29.748 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.748 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:374c250b:::obj-1-324:head# 2020-11-16T21:12:29.748 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.748 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3761a149:::obj-1-342:head# 2020-11-16T21:12:29.748 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.748 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:379ff95a:::obj-1-341:head# 2020-11-16T21:12:29.748 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.749 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:37dc15a6:::obj-1-26:head# 2020-11-16T21:12:29.749 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.749 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:38130e33:::obj-1-552:head# 2020-11-16T21:12:29.749 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.749 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:39c44880:::obj-1-538:head# 2020-11-16T21:12:29.749 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.749 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3a08dec9:::obj-1-368:head# 2020-11-16T21:12:29.750 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.750 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3a2b46a6:::obj-1-407:head# 2020-11-16T21:12:29.750 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.750 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3a7dfe0a:::obj-1-285:head# 2020-11-16T21:12:29.750 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.750 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3aebb144:::obj-1-248:head# 2020-11-16T21:12:29.750 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.750 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3b08b36f:::obj-1-310:head# 2020-11-16T21:12:29.751 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.751 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3b28ffa3:::obj-1-295:head# 2020-11-16T21:12:29.751 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.751 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3ba79bd1:::obj-1-444:head# 2020-11-16T21:12:29.751 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.751 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3c05df8d:::obj-1-319:head# 2020-11-16T21:12:29.751 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.752 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3c57e6a9:::obj-1-436:head# 2020-11-16T21:12:29.752 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.752 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3c73d8ee:::obj-1-298:head# 2020-11-16T21:12:29.752 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.752 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3cb6ae2a:::obj-1-400:head# 2020-11-16T21:12:29.752 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.752 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3cb71491:::obj-1-328:head# 2020-11-16T21:12:29.752 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.753 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3cb842fe:::obj-1-541:head# 2020-11-16T21:12:29.753 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.753 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3cf6835e:::obj-1-496:head# 2020-11-16T21:12:29.753 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.753 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3cf78507:::obj-1-573:head# 2020-11-16T21:12:29.753 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.753 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3d0dab84:::obj-1-554:head# 2020-11-16T21:12:29.754 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.754 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3d1a5f9d:::obj-1-323:head# 2020-11-16T21:12:29.754 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.754 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3da5f3a2:::obj-1-414:head# 2020-11-16T21:12:29.754 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.754 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3dbb65f0:::obj-1-440:head# 2020-11-16T21:12:29.754 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.754 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3dc94618:::obj-1-223:head# 2020-11-16T21:12:29.755 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.755 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3df3d5ab:::obj-1-25:head# 2020-11-16T21:12:29.755 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.755 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3e167988:::obj-1-336:head# 2020-11-16T21:12:29.755 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.755 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3e4ef3bd:::obj-1-104:head# 2020-11-16T21:12:29.755 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.756 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3e9e1efd:::obj-1-357:head# 2020-11-16T21:12:29.756 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.756 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3ebbe3a7:::obj-1-42:head# 2020-11-16T21:12:29.756 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.756 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3ec80373:::obj-1-403:head# 2020-11-16T21:12:29.756 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.756 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3ed086e4:::obj-1-13:head# 2020-11-16T21:12:29.756 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.757 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:3fe8e157:::obj-1-164:head# 2020-11-16T21:12:29.757 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.757 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:401ddfd4:::obj-1-379:head# 2020-11-16T21:12:29.757 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.757 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4092f6cd:::obj-1-430:head# 2020-11-16T21:12:29.757 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.757 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:414bd7cc:::obj-1-562:head# 2020-11-16T21:12:29.757 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.758 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:422bd9d0:::obj-1-287:head# 2020-11-16T21:12:29.758 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.758 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4255002d:::obj-1-578:head# 2020-11-16T21:12:29.758 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.758 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4270725c:::obj-1-109:head# 2020-11-16T21:12:29.758 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.758 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:42836669:::obj-1-31:head# 2020-11-16T21:12:29.759 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.759 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:428d93ef:::obj-1-32:head# 2020-11-16T21:12:29.759 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.759 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4327ef5b:::obj-1-453:head# 2020-11-16T21:12:29.759 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.759 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:440eb07b:::obj-1-236:head# 2020-11-16T21:12:29.759 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.759 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:442a028a:::obj-1-365:head# 2020-11-16T21:12:29.760 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.760 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:449ca54a:::obj-1-161:head# 2020-11-16T21:12:29.760 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.760 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:45128c3b:::obj-1-343:head# 2020-11-16T21:12:29.760 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.760 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:45801bf6:::obj-1-264:head# 2020-11-16T21:12:29.760 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.761 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:45896f61:::obj-1-48:head# 2020-11-16T21:12:29.761 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.761 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:459b7f1c:::obj-1-190:head# 2020-11-16T21:12:29.761 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.761 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:466af96a:::obj-1-486:head# 2020-11-16T21:12:29.761 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.761 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:47387ce4:::obj-1-575:head# 2020-11-16T21:12:29.761 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.762 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:47c94cc8:::obj-1-564:head# 2020-11-16T21:12:29.762 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.762 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:47d7f8ed:::obj-1-544:head# 2020-11-16T21:12:29.762 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.762 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:47fc0ec3:::obj-1-558:head# 2020-11-16T21:12:29.762 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.762 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:48ba42d8:::obj-1-156:head# 2020-11-16T21:12:29.763 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.763 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:494e269e:::obj-1-600:head# 2020-11-16T21:12:29.763 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.763 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:49646095:::obj-1-73:head# 2020-11-16T21:12:29.763 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.763 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:49eb65ab:::obj-1-374:head# 2020-11-16T21:12:29.763 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.763 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4a0075a4:::obj-1-432:head# 2020-11-16T21:12:29.764 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.764 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4af1ffc2:::obj-1-72:head# 2020-11-16T21:12:29.764 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.764 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4bc1c46f:::obj-1-346:head# 2020-11-16T21:12:29.764 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.764 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4bd55e68:::obj-1-93:head# 2020-11-16T21:12:29.764 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.765 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4c6d2760:::obj-1-198:head# 2020-11-16T21:12:29.765 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.765 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4cb9be3b:::obj-1-451:head# 2020-11-16T21:12:29.765 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.765 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4d260a64:::obj-1-514:head# 2020-11-16T21:12:29.765 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.765 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4d7d6284:::obj-1-583:head# 2020-11-16T21:12:29.765 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.766 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4e79ac02:::obj-1-437:head# 2020-11-16T21:12:29.766 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.766 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4e980099:::obj-1-507:head# 2020-11-16T21:12:29.766 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.766 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4ea8f99c:::obj-1-211:head# 2020-11-16T21:12:29.766 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.766 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4eabdc2a:::obj-1-590:head# 2020-11-16T21:12:29.767 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.767 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4f97fd73:::obj-1-243:head# 2020-11-16T21:12:29.767 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.767 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4fe00224:::obj-1-229:head# 2020-11-16T21:12:29.767 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.767 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4fead9ab:::obj-1-476:head# 2020-11-16T21:12:29.767 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.767 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:4ffddfd7:::obj-1-462:head# 2020-11-16T21:12:29.768 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.768 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5085c04d:::obj-1-100:head# 2020-11-16T21:12:29.768 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.768 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:50a8b465:::obj-1-574:head# 2020-11-16T21:12:29.768 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.768 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:518a151f:::obj-1-189:head# 2020-11-16T21:12:29.768 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.768 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:519afc95:::obj-1-314:head# 2020-11-16T21:12:29.769 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.769 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:52199781:::obj-1-202:head# 2020-11-16T21:12:29.769 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.769 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:52313062:::obj-1-337:head# 2020-11-16T21:12:29.769 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.769 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:526a5196:::obj-1-225:head# 2020-11-16T21:12:29.769 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.770 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:53128d32:::obj-1-483:head# 2020-11-16T21:12:29.770 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.770 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:53c2bb21:::obj-1-532:head# 2020-11-16T21:12:29.770 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.770 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:53fa4f3c:::obj-1-423:head# 2020-11-16T21:12:29.770 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.770 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:541c1d30:::obj-1-143:head# 2020-11-16T21:12:29.770 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.771 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:548d20c3:::obj-1-352:head# 2020-11-16T21:12:29.771 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.771 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:54e4c6f9:::obj-1-101:head# 2020-11-16T21:12:29.771 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.771 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:55016e06:::obj-1-276:head# 2020-11-16T21:12:29.771 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.771 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5510c784:::obj-1-491:head# 2020-11-16T21:12:29.772 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.772 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:551275ef:::obj-1-239:head# 2020-11-16T21:12:29.772 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.772 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:556df001:::obj-1-46:head# 2020-11-16T21:12:29.772 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.772 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:558fb5b1:::obj-1-536:head# 2020-11-16T21:12:29.772 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.772 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:56390d98:::obj-1-64:head# 2020-11-16T21:12:29.773 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.773 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:564f962a:::obj-1-125:head# 2020-11-16T21:12:29.773 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.773 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:56936e94:::obj-1-251:head# 2020-11-16T21:12:29.773 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.773 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:57585d3d:::obj-1-294:head# 2020-11-16T21:12:29.773 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.773 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5764cf7b:::obj-1-479:head# 2020-11-16T21:12:29.774 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.774 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5782341a:::obj-1-218:head# 2020-11-16T21:12:29.774 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.774 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:57d07a1a:::obj-1-317:head# 2020-11-16T21:12:29.774 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.774 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:58700205:::obj-1-176:head# 2020-11-16T21:12:29.774 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.775 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:58aee718:::obj-1-434:head# 2020-11-16T21:12:29.775 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.775 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:594c9d52:::obj-1-458:head# 2020-11-16T21:12:29.775 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.775 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5a70b555:::obj-1-279:head# 2020-11-16T21:12:29.775 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.775 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5b02568b:::obj-1-586:head# 2020-11-16T21:12:29.775 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.776 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5b69cdcf:::obj-1-391:head# 2020-11-16T21:12:29.776 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.776 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5cf151b5:::obj-1-510:head# 2020-11-16T21:12:29.776 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.776 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5d7fd07f:::obj-1-152:head# 2020-11-16T21:12:29.776 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.776 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5de288d7:::obj-1-235:head# 2020-11-16T21:12:29.777 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.777 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5ec497cd:::obj-1-44:head# 2020-11-16T21:12:29.777 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.777 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5fb3a09f:::obj-1-518:head# 2020-11-16T21:12:29.777 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.777 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:5ff3c2ef:::obj-1-470:head# 2020-11-16T21:12:29.777 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.777 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:601efcb1:::obj-1-549:head# 2020-11-16T21:12:29.778 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.778 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:60831338:::obj-1-263:head# 2020-11-16T21:12:29.778 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.778 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6092777b:::obj-1-145:head# 2020-11-16T21:12:29.778 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.778 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:60b17206:::obj-1-411:head# 2020-11-16T21:12:29.778 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.779 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:610c957c:::obj-1-278:head# 2020-11-16T21:12:29.779 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.779 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:613b5ef4:::obj-1-504:head# 2020-11-16T21:12:29.779 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.779 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:61ed7d60:::obj-1-274:head# 2020-11-16T21:12:29.779 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.779 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:621b6751:::obj-1-212:head# 2020-11-16T21:12:29.779 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.780 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:62b0aba9:::obj-1-253:head# 2020-11-16T21:12:29.780 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.780 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:64017240:::obj-1-102:head# 2020-11-16T21:12:29.780 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.780 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:645f4f49:::obj-1-332:head# 2020-11-16T21:12:29.780 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.780 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:65db82b9:::obj-1-334:head# 2020-11-16T21:12:29.780 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.781 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:667422e5:::obj-1-465:head# 2020-11-16T21:12:29.781 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.781 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:66c50bc2:::obj-1-322:head# 2020-11-16T21:12:29.781 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.781 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:681a4577:::obj-1-511:head# 2020-11-16T21:12:29.781 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.781 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6874e007:::obj-1-398:head# 2020-11-16T21:12:29.782 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.782 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:693746c1:::obj-1-582:head# 2020-11-16T21:12:29.782 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.782 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:69638b0c:::obj-1-8:head# 2020-11-16T21:12:29.782 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.782 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6967de6e:::obj-1-331:head# 2020-11-16T21:12:29.782 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.783 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:69da97aa:::obj-1-340:head# 2020-11-16T21:12:29.783 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.783 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6b3a50a6:::obj-1-144:head# 2020-11-16T21:12:29.783 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.783 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6be4ef26:::obj-1-579:head# 2020-11-16T21:12:29.783 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.783 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6d735549:::obj-1-599:head# 2020-11-16T21:12:29.784 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.784 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6e22c25e:::obj-1-9:head# 2020-11-16T21:12:29.784 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.784 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6e2ced4e:::obj-1-370:head# 2020-11-16T21:12:29.784 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.784 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6e961169:::obj-1-428:head# 2020-11-16T21:12:29.785 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.785 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6efcbbee:::obj-1-168:head# 2020-11-16T21:12:29.785 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.785 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6f20741c:::obj-1-33:head# 2020-11-16T21:12:29.785 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.785 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6fe4c885:::obj-1-273:head# 2020-11-16T21:12:29.785 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.786 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:6fffdea2:::obj-1-425:head# 2020-11-16T21:12:29.786 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.786 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:70716852:::obj-1-593:head# 2020-11-16T21:12:29.786 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.786 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:70919606:::obj-1-515:head# 2020-11-16T21:12:29.786 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.786 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:70c39d76:::obj-1-467:head# 2020-11-16T21:12:29.787 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.787 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:70f57ad3:::obj-1-361:head# 2020-11-16T21:12:29.787 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.787 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7177c7b5:::obj-1-503:head# 2020-11-16T21:12:29.787 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.788 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:719b69bd:::obj-1-497:head# 2020-11-16T21:12:29.788 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.788 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:724f9347:::obj-1-6:head# 2020-11-16T21:12:29.788 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.788 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:726d9f6b:::obj-1-111:head# 2020-11-16T21:12:29.788 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.788 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:72a91baf:::obj-1-439:head# 2020-11-16T21:12:29.789 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.789 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:72aa43ee:::obj-1-534:head# 2020-11-16T21:12:29.789 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.789 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:72c40e11:::obj-1-288:head# 2020-11-16T21:12:29.789 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.789 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:74196d45:::obj-1-246:head# 2020-11-16T21:12:29.789 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.790 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:743f42b1:::obj-1-326:head# 2020-11-16T21:12:29.790 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.790 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7523b538:::obj-1-494:head# 2020-11-16T21:12:29.790 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.790 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:75310ed6:::obj-1-364:head# 2020-11-16T21:12:29.790 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.790 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7568e0bc:::obj-1-585:head# 2020-11-16T21:12:29.791 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.791 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:760219a3:::obj-1-353:head# 2020-11-16T21:12:29.791 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.791 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:763338c9:::obj-1-555:head# 2020-11-16T21:12:29.791 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.791 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:77bc2d65:::obj-1-386:head# 2020-11-16T21:12:29.791 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.792 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:77ed0c7a:::obj-1-492:head# 2020-11-16T21:12:29.792 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.792 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:785a4052:::obj-1-456:head# 2020-11-16T21:12:29.792 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.792 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:796799c3:::obj-1-195:head# 2020-11-16T21:12:29.792 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.792 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:79caf0c1:::obj-1-445:head# 2020-11-16T21:12:29.792 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.793 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7af5b3e4:::obj-1-217:head# 2020-11-16T21:12:29.793 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.793 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7b3a9063:::obj-1-169:head# 2020-11-16T21:12:29.793 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.793 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7b509a92:::obj-1-561:head# 2020-11-16T21:12:29.793 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.793 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7bc646b1:::obj-1-415:head# 2020-11-16T21:12:29.794 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.794 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7c2f9188:::obj-1-482:head# 2020-11-16T21:12:29.794 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.794 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7d0da936:::obj-1-11:head# 2020-11-16T21:12:29.794 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.794 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7d25ca2d:::obj-1-242:head# 2020-11-16T21:12:29.794 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.794 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7d64f81e:::obj-1-473:head# 2020-11-16T21:12:29.795 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.795 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7ef2ea1b:::obj-1-417:head# 2020-11-16T21:12:29.795 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.795 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7f52b238:::obj-1-308:head# 2020-11-16T21:12:29.795 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.795 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:7f7a313c:::obj-1-551:head# 2020-11-16T21:12:29.795 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.796 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:806827a7:::obj-1-472:head# 2020-11-16T21:12:29.796 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.796 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8112c3ab:::obj-1-45:head# 2020-11-16T21:12:29.796 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.796 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:81365b55:::obj-1-315:head# 2020-11-16T21:12:29.796 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.796 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:818ffdc3:::obj-1-345:head# 2020-11-16T21:12:29.796 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.797 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:81babab8:::obj-1-535:head# 2020-11-16T21:12:29.797 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.797 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8288f652:::obj-1-347:head# 2020-11-16T21:12:29.797 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.797 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8312eadd:::obj-1-14:head# 2020-11-16T21:12:29.797 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.797 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:83b89bc7:::obj-1-137:head# 2020-11-16T21:12:29.798 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.798 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:84a12321:::obj-1-134:head# 2020-11-16T21:12:29.798 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.798 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:84b26d62:::obj-1-408:head# 2020-11-16T21:12:29.798 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.798 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:84bbb26e:::obj-1-43:head# 2020-11-16T21:12:29.798 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.798 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:86fc0014:::obj-1-222:head# 2020-11-16T21:12:29.799 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.799 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:87488bf4:::obj-1-174:head# 2020-11-16T21:12:29.799 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.799 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8754389a:::obj-1-65:head# 2020-11-16T21:12:29.799 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.799 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:87a6ad67:::obj-1-366:head# 2020-11-16T21:12:29.799 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.800 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:87d32b83:::obj-1-513:head# 2020-11-16T21:12:29.800 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.800 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8813a346:::obj-1-540:head# 2020-11-16T21:12:29.800 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.800 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:88146951:::obj-1-216:head# 2020-11-16T21:12:29.800 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.800 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8820800d:::obj-1-83:head# 2020-11-16T21:12:29.800 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.801 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:88319745:::obj-1-98:head# 2020-11-16T21:12:29.801 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.801 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:88c969b8:::obj-1-54:head# 2020-11-16T21:12:29.801 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.801 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:895a2d53:::obj-1-150:head# 2020-11-16T21:12:29.801 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.801 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8ae006a0:::obj-1-241:head# 2020-11-16T21:12:29.801 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.802 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8bd44e64:::obj-1-300:head# 2020-11-16T21:12:29.802 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.802 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8beb257f:::obj-1-3:head# 2020-11-16T21:12:29.802 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.802 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8cd8c743:::obj-1-527:head# 2020-11-16T21:12:29.802 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.802 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8daa99c7:::obj-1-237:head# 2020-11-16T21:12:29.803 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.803 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8f5bccbf:::obj-1-59:head# 2020-11-16T21:12:29.803 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.803 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8f6d9945:::obj-1-151:head# 2020-11-16T21:12:29.803 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.803 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8f7c568c:::obj-1-261:head# 2020-11-16T21:12:29.803 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.803 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:8fbc59a9:::obj-1-589:head# 2020-11-16T21:12:29.804 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.804 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:90225d94:::obj-1-194:head# 2020-11-16T21:12:29.804 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.804 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:904c50d3:::obj-1-63:head# 2020-11-16T21:12:29.804 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.804 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:90e1bd79:::obj-1-450:head# 2020-11-16T21:12:29.804 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.805 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9274b06d:::obj-1-460:head# 2020-11-16T21:12:29.805 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.805 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:92af41f5:::obj-1-409:head# 2020-11-16T21:12:29.805 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.805 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:935a08d6:::obj-1-192:head# 2020-11-16T21:12:29.805 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.805 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:935f8a13:::obj-1-24:head# 2020-11-16T21:12:29.805 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.806 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:95230148:::obj-1-280:head# 2020-11-16T21:12:29.806 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.806 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:95342eb0:::obj-1-390:head# 2020-11-16T21:12:29.806 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.806 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9659d09c:::obj-1-292:head# 2020-11-16T21:12:29.806 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.806 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:970e8cb4:::obj-1-18:head# 2020-11-16T21:12:29.807 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.807 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:97d21cf1:::obj-1-80:head# 2020-11-16T21:12:29.807 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.807 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:982bdb12:::obj-1-302:head# 2020-11-16T21:12:29.807 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.807 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:985dfd75:::obj-1-91:head# 2020-11-16T21:12:29.807 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.807 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:986d245a:::obj-1-154:head# 2020-11-16T21:12:29.808 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.808 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:98a03e99:::obj-1-529:head# 2020-11-16T21:12:29.808 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.808 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:98b7103e:::obj-1-275:head# 2020-11-16T21:12:29.808 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.808 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:990f0ecf:::obj-1-427:head# 2020-11-16T21:12:29.808 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.809 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:996c1f26:::obj-1-96:head# 2020-11-16T21:12:29.809 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.809 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:99bd2018:::obj-1-17:head# 2020-11-16T21:12:29.809 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.809 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9a7434c9:::obj-1-469:head# 2020-11-16T21:12:29.809 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.809 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9a8e36cc:::obj-1-598:head# 2020-11-16T21:12:29.809 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.810 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9abcc369:::obj-1-442:head# 2020-11-16T21:12:29.810 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.810 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9b0ea06e:::obj-1-231:head# 2020-11-16T21:12:29.810 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.810 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9b34e80d:::obj-1-378:head# 2020-11-16T21:12:29.810 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.810 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9b55a2a9:::obj-1-60:head# 2020-11-16T21:12:29.811 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.811 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9be3f2da:::obj-1-597:head# 2020-11-16T21:12:29.811 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.811 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9bf62c88:::obj-1-447:head# 2020-11-16T21:12:29.811 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.811 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9c31857b:::obj-1-272:head# 2020-11-16T21:12:29.811 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.811 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9c533e91:::obj-1-19:head# 2020-11-16T21:12:29.812 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.812 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9ca70688:::obj-1-570:head# 2020-11-16T21:12:29.812 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.812 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9ccce337:::obj-1-147:head# 2020-11-16T21:12:29.812 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.812 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9d4b2dfc:::obj-1-267:head# 2020-11-16T21:12:29.812 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.813 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9d4df969:::obj-1-286:head# 2020-11-16T21:12:29.813 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.813 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9d714651:::obj-1-495:head# 2020-11-16T21:12:29.813 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.813 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9da65518:::obj-1-498:head# 2020-11-16T21:12:29.813 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.813 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9e94f27b:::obj-1-209:head# 2020-11-16T21:12:29.813 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.814 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9f39e910:::obj-1-354:head# 2020-11-16T21:12:29.814 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.814 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9f8f9ef4:::obj-1-226:head# 2020-11-16T21:12:29.814 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.814 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:9fcd3f36:::obj-1-47:head# 2020-11-16T21:12:29.814 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.814 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a0326d7d:::obj-1-338:head# 2020-11-16T21:12:29.815 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.815 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a05e7475:::obj-1-62:head# 2020-11-16T21:12:29.815 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.815 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a0955052:::obj-1-533:head# 2020-11-16T21:12:29.815 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.815 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a0aa6848:::obj-1-38:head# 2020-11-16T21:12:29.815 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.815 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a0fdc285:::obj-1-133:head# 2020-11-16T21:12:29.816 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.816 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a10290b8:::obj-1-245:head# 2020-11-16T21:12:29.816 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.816 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a118a8a8:::obj-1-500:head# 2020-11-16T21:12:29.816 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.816 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a11a4079:::obj-1-199:head# 2020-11-16T21:12:29.816 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.816 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a1b3ec36:::obj-1-15:head# 2020-11-16T21:12:29.817 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.817 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a1f5079b:::obj-1-468:head# 2020-11-16T21:12:29.817 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.817 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a1ff04b5:::obj-1-167:head# 2020-11-16T21:12:29.817 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.817 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a21cc437:::obj-1-128:head# 2020-11-16T21:12:29.817 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.818 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a2966c77:::obj-1-140:head# 2020-11-16T21:12:29.818 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.818 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a321e537:::obj-1-184:head# 2020-11-16T21:12:29.818 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.818 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a33418d8:::obj-1-488:head# 2020-11-16T21:12:29.818 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.818 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a38b1c1a:::obj-1-584:head# 2020-11-16T21:12:29.818 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.819 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a3f83221:::obj-1-158:head# 2020-11-16T21:12:29.819 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.819 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a46e33eb:::obj-1-106:head# 2020-11-16T21:12:29.819 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.819 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a491460d:::obj-1-421:head# 2020-11-16T21:12:29.819 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.819 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a4dd813b:::obj-1-530:head# 2020-11-16T21:12:29.820 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.820 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a4e0d96d:::obj-1-351:head# 2020-11-16T21:12:29.820 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.820 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a52657b4:::obj-1-557:head# 2020-11-16T21:12:29.820 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.820 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a5b3bca6:::obj-1-277:head# 2020-11-16T21:12:29.820 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.820 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a5f5a9fb:::obj-1-170:head# 2020-11-16T21:12:29.821 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.821 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a6aeed57:::obj-1-141:head# 2020-11-16T21:12:29.821 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.821 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a6dda70f:::obj-1-471:head# 2020-11-16T21:12:29.821 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.821 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a71186c0:::obj-1-406:head# 2020-11-16T21:12:29.821 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.821 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a764ad8c:::obj-1-221:head# 2020-11-16T21:12:29.822 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.822 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a7b63473:::obj-1-131:head# 2020-11-16T21:12:29.822 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.822 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a88042fa:::obj-1-296:head# 2020-11-16T21:12:29.822 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.822 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:a8805fe0:::obj-1-213:head# 2020-11-16T21:12:29.822 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.823 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:aa4f12b1:::obj-1-270:head# 2020-11-16T21:12:29.823 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.823 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ab27f701:::obj-1-399:head# 2020-11-16T21:12:29.823 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.823 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ab8027c1:::obj-1-360:head# 2020-11-16T21:12:29.823 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.823 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ac3ef8ab:::obj-1-371:head# 2020-11-16T21:12:29.824 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.824 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ac4db76d:::obj-1-405:head# 2020-11-16T21:12:29.824 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.824 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ad53cb29:::obj-1-256:head# 2020-11-16T21:12:29.824 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.824 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ad704c16:::obj-1-485:head# 2020-11-16T21:12:29.824 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.824 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:adec4920:::obj-1-107:head# 2020-11-16T21:12:29.825 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.825 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ae3a94b3:::obj-1-95:head# 2020-11-16T21:12:29.825 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.825 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ae92616e:::obj-1-448:head# 2020-11-16T21:12:29.825 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.825 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ae975168:::obj-1-426:head# 2020-11-16T21:12:29.825 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.826 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:aee8fd62:::obj-1-27:head# 2020-11-16T21:12:29.826 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.826 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:af38aa87:::obj-1-50:head# 2020-11-16T21:12:29.826 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.826 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b0327eb3:::obj-1-380:head# 2020-11-16T21:12:29.826 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.826 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b03cd166:::obj-1-203:head# 2020-11-16T21:12:29.826 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.827 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b06f06e0:::obj-1-126:head# 2020-11-16T21:12:29.827 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.827 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b0972538:::obj-1-506:head# 2020-11-16T21:12:29.827 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.827 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b0cfb86a:::obj-1-359:head# 2020-11-16T21:12:29.827 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.827 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b10adb1c:::obj-1-333:head# 2020-11-16T21:12:29.827 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.828 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b20b9004:::obj-1-401:head# 2020-11-16T21:12:29.828 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.828 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b2113757:::obj-1-4:head# 2020-11-16T21:12:29.828 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.828 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b22441e9:::obj-1-29:head# 2020-11-16T21:12:29.828 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.828 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b2472384:::obj-1-207:head# 2020-11-16T21:12:29.829 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.829 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b2604227:::obj-1-502:head# 2020-11-16T21:12:29.829 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.829 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b3277f89:::obj-1-175:head# 2020-11-16T21:12:29.829 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.829 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b3d09e0d:::obj-1-258:head# 2020-11-16T21:12:29.829 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.829 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b3e9969a:::obj-1-499:head# 2020-11-16T21:12:29.830 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.830 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b4892e7d:::obj-1-155:head# 2020-11-16T21:12:29.830 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.830 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b4b76084:::obj-1-581:head# 2020-11-16T21:12:29.830 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.830 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b6e47ae5:::obj-1-215:head# 2020-11-16T21:12:29.830 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.831 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b75af5ff:::obj-1-228:head# 2020-11-16T21:12:29.831 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.831 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b8e2e20f:::obj-1-220:head# 2020-11-16T21:12:29.831 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.831 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:b8e80c4e:::obj-1-576:head# 2020-11-16T21:12:29.831 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.831 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ba502aee:::obj-1-255:head# 2020-11-16T21:12:29.832 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.832 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ba9c3e6e:::obj-1-57:head# 2020-11-16T21:12:29.832 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.832 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bb068062:::obj-1-461:head# 2020-11-16T21:12:29.832 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.832 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bb16c115:::obj-1-157:head# 2020-11-16T21:12:29.832 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.832 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bb447914:::obj-1-481:head# 2020-11-16T21:12:29.833 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.833 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bb6c8e0f:::obj-1-348:head# 2020-11-16T21:12:29.833 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.833 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bb77d6d0:::obj-1-301:head# 2020-11-16T21:12:29.833 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.833 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bc51c5f6:::obj-1-422:head# 2020-11-16T21:12:29.833 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.834 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bc838d5c:::obj-1-35:head# 2020-11-16T21:12:29.834 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.834 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bceae5eb:::obj-1-56:head# 2020-11-16T21:12:29.834 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.834 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bea17ef6:::obj-1-84:head# 2020-11-16T21:12:29.835 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.835 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:beb88be5:::obj-1-396:head# 2020-11-16T21:12:29.835 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.835 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:beed7b18:::obj-1-569:head# 2020-11-16T21:12:29.835 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.835 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bf0bff49:::obj-1-385:head# 2020-11-16T21:12:29.835 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.835 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bf545d5a:::obj-1-69:head# 2020-11-16T21:12:29.836 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.836 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bf560a7f:::obj-1-40:head# 2020-11-16T21:12:29.836 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.836 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:bff6999c:::obj-1-240:head# 2020-11-16T21:12:29.836 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.836 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c0a33d38:::obj-1-321:head# 2020-11-16T21:12:29.836 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.837 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c1e4c9b4:::obj-1-299:head# 2020-11-16T21:12:29.837 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.837 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c1ec515a:::obj-1-16:head# 2020-11-16T21:12:29.837 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.837 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c22de3c8:::obj-1-110:head# 2020-11-16T21:12:29.837 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.837 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c231cd7c:::obj-1-58:head# 2020-11-16T21:12:29.838 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.838 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c2ee6ace:::obj-1-186:head# 2020-11-16T21:12:29.838 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.838 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c32bfef2:::obj-1-181:head# 2020-11-16T21:12:29.838 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.838 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c575a298:::obj-1-466:head# 2020-11-16T21:12:29.838 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.838 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c58c3f80:::obj-1-493:head# 2020-11-16T21:12:29.839 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.839 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c6442cf2:::obj-1-455:head# 2020-11-16T21:12:29.839 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.839 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c6619413:::obj-1-37:head# 2020-11-16T21:12:29.839 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.839 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c6c2f8e5:::obj-1-519:head# 2020-11-16T21:12:29.839 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.840 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c6f811a2:::obj-1-180:head# 2020-11-16T21:12:29.840 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.840 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c72af975:::obj-1-21:head# 2020-11-16T21:12:29.840 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.840 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c73eb350:::obj-1-356:head# 2020-11-16T21:12:29.840 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.840 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c7a683da:::obj-1-410:head# 2020-11-16T21:12:29.840 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.841 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c7eb9bcc:::obj-1-553:head# 2020-11-16T21:12:29.841 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.841 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c886df99:::obj-1-269:head# 2020-11-16T21:12:29.841 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.841 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c932e3ec:::obj-1-135:head# 2020-11-16T21:12:29.841 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.841 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c9b6507a:::obj-1-520:head# 2020-11-16T21:12:29.842 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.842 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:c9e3b064:::obj-1-244:head# 2020-11-16T21:12:29.842 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.842 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:cb705d66:::obj-1-377:head# 2020-11-16T21:12:29.842 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.842 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:cc80e25c:::obj-1-115:head# 2020-11-16T21:12:29.842 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.842 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:cd1ec88e:::obj-1-297:head# 2020-11-16T21:12:29.843 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.843 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:cd7e6936:::obj-1-234:head# 2020-11-16T21:12:29.843 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.843 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ce32b69c:::obj-1-30:head# 2020-11-16T21:12:29.843 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.843 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:cf2b123c:::obj-1-293:head# 2020-11-16T21:12:29.843 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.844 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:cf2e80a9:::obj-1-542:head# 2020-11-16T21:12:29.844 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.844 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:cf51da6a:::obj-1-284:head# 2020-11-16T21:12:29.844 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.844 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:cf590f42:::obj-1-381:head# 2020-11-16T21:12:29.844 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.844 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:cf68a2b1:::obj-1-68:head# 2020-11-16T21:12:29.844 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.845 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:cf6aefb6:::obj-1-28:head# 2020-11-16T21:12:29.845 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.845 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:cfd8e6af:::obj-1-105:head# 2020-11-16T21:12:29.845 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.845 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d0640bad:::obj-1-254:head# 2020-11-16T21:12:29.845 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.845 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d101cda5:::obj-1-259:head# 2020-11-16T21:12:29.845 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.846 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d1be9386:::obj-1-416:head# 2020-11-16T21:12:29.846 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.846 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d243f036:::obj-1-265:head# 2020-11-16T21:12:29.846 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.846 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d2808c34:::obj-1-159:head# 2020-11-16T21:12:29.846 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.846 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d3415824:::obj-1-39:head# 2020-11-16T21:12:29.847 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.847 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d424f07a:::obj-1-547:head# 2020-11-16T21:12:29.847 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.847 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d44139f2:::obj-1-289:head# 2020-11-16T21:12:29.847 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.847 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d46015ab:::obj-1-546:head# 2020-11-16T21:12:29.847 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.847 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d4e503db:::obj-1-525:head# 2020-11-16T21:12:29.848 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.848 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d4e97d18:::obj-1-85:head# 2020-11-16T21:12:29.848 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.848 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d511492e:::obj-1-388:head# 2020-11-16T21:12:29.848 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.848 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d526a7fa:::obj-1-12:head# 2020-11-16T21:12:29.848 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.849 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d5667b59:::obj-1-369:head# 2020-11-16T21:12:29.849 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.849 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d57f2261:::obj-1-419:head# 2020-11-16T21:12:29.849 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.849 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d57f281b:::obj-1-250:head# 2020-11-16T21:12:29.849 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.849 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d6e05894:::obj-1-166:head# 2020-11-16T21:12:29.849 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.850 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d7689481:::obj-1-424:head# 2020-11-16T21:12:29.850 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.850 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d7d27743:::obj-1-129:head# 2020-11-16T21:12:29.850 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.850 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d7e4d3e9:::obj-1-127:head# 2020-11-16T21:12:29.850 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.850 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d7f1422a:::obj-1-594:head# 2020-11-16T21:12:29.851 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.851 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d8469fd2:::obj-1-153:head# 2020-11-16T21:12:29.851 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.851 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d84efd9e:::obj-1-435:head# 2020-11-16T21:12:29.851 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.851 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d883004b:::obj-1-162:head# 2020-11-16T21:12:29.851 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.851 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d8b60854:::obj-1-257:head# 2020-11-16T21:12:29.852 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.852 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d91aaae6:::obj-1-1:head# 2020-11-16T21:12:29.852 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.852 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d98695ca:::obj-1-524:head# 2020-11-16T21:12:29.852 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.852 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:d9ae97c8:::obj-1-177:head# 2020-11-16T21:12:29.852 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.852 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:da1713bd:::obj-1-309:head# 2020-11-16T21:12:29.853 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.853 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:da37824c:::obj-1-2:head# 2020-11-16T21:12:29.853 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.853 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:dab221b0:::obj-1-526:head# 2020-11-16T21:12:29.853 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.853 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:db27d885:::obj-1-363:head# 2020-11-16T21:12:29.853 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.854 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:dc001d76:::obj-1-329:head# 2020-11-16T21:12:29.854 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.854 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:dc4f4119:::obj-1-122:head# 2020-11-16T21:12:29.854 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.854 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:dcc36eb7:::obj-1-283:head# 2020-11-16T21:12:29.854 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.854 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:df7da06c:::obj-1-124:head# 2020-11-16T21:12:29.854 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.855 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:df7fcacb:::obj-1-330:head# 2020-11-16T21:12:29.855 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.855 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e05cbe31:::obj-1-413:head# 2020-11-16T21:12:29.855 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.855 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e1060a54:::obj-1-384:head# 2020-11-16T21:12:29.855 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.855 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e15b839b:::obj-1-316:head# 2020-11-16T21:12:29.856 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.856 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e161b6e5:::obj-1-103:head# 2020-11-16T21:12:29.856 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.856 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e1d1d79c:::obj-1-512:head# 2020-11-16T21:12:29.856 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.856 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e224b482:::obj-1-501:head# 2020-11-16T21:12:29.856 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.856 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e22d6f76:::obj-1-70:head# 2020-11-16T21:12:29.857 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.857 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e2bd801c:::obj-1-523:head# 2020-11-16T21:12:29.857 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.857 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e3d9f395:::obj-1-397:head# 2020-11-16T21:12:29.857 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.857 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e474ea25:::obj-1-457:head# 2020-11-16T21:12:29.857 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.857 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e49e1ff3:::obj-1-130:head# 2020-11-16T21:12:29.858 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.858 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e56c67a0:::obj-1-191:head# 2020-11-16T21:12:29.858 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.858 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e576e921:::obj-1-376:head# 2020-11-16T21:12:29.858 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.858 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e619fd26:::obj-1-395:head# 2020-11-16T21:12:29.858 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.859 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e625bdc5:::obj-1-522:head# 2020-11-16T21:12:29.859 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.859 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e68a0fa6:::obj-1-394:head# 2020-11-16T21:12:29.859 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.859 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e6cf260f:::obj-1-77:head# 2020-11-16T21:12:29.859 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.859 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e6d6255a:::obj-1-108:head# 2020-11-16T21:12:29.859 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.860 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e7a35d2f:::obj-1-116:head# 2020-11-16T21:12:29.860 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.860 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e7b07de5:::obj-1-238:head# 2020-11-16T21:12:29.860 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.860 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e8037aba:::obj-1-404:head# 2020-11-16T21:12:29.860 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.860 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e8e8b34f:::obj-1-291:head# 2020-11-16T21:12:29.861 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.861 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:e987cd7a:::obj-1-123:head# 2020-11-16T21:12:29.861 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.861 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ea534db1:::obj-1-311:head# 2020-11-16T21:12:29.861 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.861 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:eabcc188:::obj-1-325:head# 2020-11-16T21:12:29.861 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.861 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:eb086660:::obj-1-595:head# 2020-11-16T21:12:29.862 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.862 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:eb2235d3:::obj-1-596:head# 2020-11-16T21:12:29.862 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.862 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:eb35ae40:::obj-1-516:head# 2020-11-16T21:12:29.862 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.862 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:eb360a13:::obj-1-224:head# 2020-11-16T21:12:29.862 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.863 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:eb3a3ea1:::obj-1-120:head# 2020-11-16T21:12:29.863 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.863 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ebca9f6e:::obj-1-290:head# 2020-11-16T21:12:29.863 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.863 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ec81db71:::obj-1-478:head# 2020-11-16T21:12:29.863 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.863 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:edeafa4c:::obj-1-362:head# 2020-11-16T21:12:29.863 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.864 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:eeb5261b:::obj-1-539:head# 2020-11-16T21:12:29.864 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.864 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:eec00eee:::obj-1-567:head# 2020-11-16T21:12:29.864 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.864 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ef075e10:::obj-1-463:head# 2020-11-16T21:12:29.864 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.864 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ef8e37d9:::obj-1-94:head# 2020-11-16T21:12:29.865 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.865 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:efad7f67:::obj-1-197:head# 2020-11-16T21:12:29.865 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.865 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:efe09ea7:::obj-1-537:head# 2020-11-16T21:12:29.865 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.865 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f0445019:::obj-1-480:head# 2020-11-16T21:12:29.865 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.865 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f08a432c:::obj-1-489:head# 2020-11-16T21:12:29.866 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.866 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f0bd9006:::obj-1-487:head# 2020-11-16T21:12:29.866 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.866 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f11a115d:::obj-1-591:head# 2020-11-16T21:12:29.866 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.866 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f232e0af:::obj-1-146:head# 2020-11-16T21:12:29.866 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.867 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f258db54:::obj-1-521:head# 2020-11-16T21:12:29.867 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.867 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f29cc204:::obj-1-566:head# 2020-11-16T21:12:29.867 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.867 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f2ae2451:::obj-1-66:head# 2020-11-16T21:12:29.867 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.867 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f318810a:::obj-1-160:head# 2020-11-16T21:12:29.867 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.868 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f3c35a0d:::obj-1-67:head# 2020-11-16T21:12:29.868 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.868 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f3ce9c90:::obj-1-88:head# 2020-11-16T21:12:29.868 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.868 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f3e3e81f:::obj-1-304:head# 2020-11-16T21:12:29.868 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.868 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f3f65b0a:::obj-1-367:head# 2020-11-16T21:12:29.869 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.869 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f4ea014b:::obj-1-149:head# 2020-11-16T21:12:29.869 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.869 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f52d115b:::obj-1-383:head# 2020-11-16T21:12:29.869 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.869 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f590311e:::obj-1-76:head# 2020-11-16T21:12:29.869 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.869 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f5a37734:::obj-1-51:head# 2020-11-16T21:12:29.870 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.870 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f5e2e77c:::obj-1-179:head# 2020-11-16T21:12:29.870 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.870 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f5e381bc:::obj-1-173:head# 2020-11-16T21:12:29.870 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.870 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f6c0ae0b:::obj-1-247:head# 2020-11-16T21:12:29.870 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.871 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f6db8a82:::obj-1-548:head# 2020-11-16T21:12:29.871 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.871 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f6e2dc46:::obj-1-36:head# 2020-11-16T21:12:29.871 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.871 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f76ee569:::obj-1-587:head# 2020-11-16T21:12:29.871 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.871 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f773b9ed:::obj-1-550:head# 2020-11-16T21:12:29.871 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.872 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f7e1adad:::obj-1-350:head# 2020-11-16T21:12:29.872 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.872 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f7eebf7a:::obj-1-559:head# 2020-11-16T21:12:29.872 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.872 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f877f025:::obj-1-187:head# 2020-11-16T21:12:29.872 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.872 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f992f9e9:::obj-1-402:head# 2020-11-16T21:12:29.873 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.873 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:f9fe960a:::obj-1-418:head# 2020-11-16T21:12:29.873 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.873 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:fa9b882f:::obj-1-165:head# 2020-11-16T21:12:29.873 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.873 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:fbd6461c:::obj-1-339:head# 2020-11-16T21:12:29.873 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.873 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:fbeab28d:::obj-1-431:head# 2020-11-16T21:12:29.874 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.874 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:fdea4138:::obj-1-185:head# 2020-11-16T21:12:29.874 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.874 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:fdfa7c9b:::obj-1-183:head# 2020-11-16T21:12:29.874 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.874 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:fe1ba57e:::obj-1-90:head# 2020-11-16T21:12:29.874 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.875 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:feb7adea:::obj-1-560:head# 2020-11-16T21:12:29.875 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.875 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ffab306e:::obj-1-97:head# 2020-11-16T21:12:29.875 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.875 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ffdfd397:::obj-1-433:head# 2020-11-16T21:12:29.875 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.875 INFO:tasks.workunit.client.0.smithi189.stdout:Write #2:ffe8cf04:::obj-1-303:head# 2020-11-16T21:12:29.875 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.876 INFO:tasks.workunit.client.0.smithi189.stdout:write_pg epoch 40 info 2.0( v 40'600 (40'500,40'600] local-lis/les=38/39 n=600 ec=26/26 lis/c=38/38 les/c/f=39/39/0 sis=38) 2020-11-16T21:12:29.876 INFO:tasks.workunit.client.0.smithi189.stdout:Import successful 2020-11-16T21:12:29.876 INFO:tasks.workunit.client.0.smithi189.stdout:get_pg_num_history pg_num_history pg_num_history(e44 pg_nums {1={21=1},2={26=1},3={31=1}} deleted_pools ) 2020-11-16T21:12:29.876 INFO:tasks.workunit.client.0.smithi189.stdout:Importing pgid 1.0 2020-11-16T21:12:29.876 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0002d87b:::obj-fill-318:head# 2020-11-16T21:12:29.876 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.876 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:00676868:::obj-fill-583:head# 2020-11-16T21:12:29.877 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.877 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:00f4514e:::obj-fill-547:head# 2020-11-16T21:12:29.877 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.877 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:01596a3c:::obj-fill-533:head# 2020-11-16T21:12:29.877 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.877 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0285e89c:::obj-fill-190:head# 2020-11-16T21:12:29.877 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.877 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:02bd591e:::obj-fill-463:head# 2020-11-16T21:12:29.878 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.878 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:02c01ffe:::obj-fill-28:head# 2020-11-16T21:12:29.878 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.878 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:03ada31e:::obj-fill-285:head# 2020-11-16T21:12:29.878 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.878 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:03b07c3f:::obj-fill-568:head# 2020-11-16T21:12:29.878 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.879 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:03b85770:::obj-fill-573:head# 2020-11-16T21:12:29.879 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.879 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:03dcbe89:::obj-fill-375:head# 2020-11-16T21:12:29.879 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.879 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:04231826:::obj-fill-594:head# 2020-11-16T21:12:29.879 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.879 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:04d48684:::obj-fill-421:head# 2020-11-16T21:12:29.879 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.880 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:05385fcc:::obj-fill-299:head# 2020-11-16T21:12:29.880 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.880 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:05785d38:::obj-fill-159:head# 2020-11-16T21:12:29.880 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.880 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:057ec231:::obj-fill-138:head# 2020-11-16T21:12:29.880 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.880 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:06058b5e:::obj-fill-569:head# 2020-11-16T21:12:29.880 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.881 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:06d2a3d9:::obj-fill-284:head# 2020-11-16T21:12:29.881 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.881 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:07597b63:::obj-fill-459:head# 2020-11-16T21:12:29.881 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.881 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:075f4069:::obj-fill-550:head# 2020-11-16T21:12:29.881 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.881 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:08234052:::obj-fill-17:head# 2020-11-16T21:12:29.882 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.882 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:091ce2ec:::obj-fill-472:head# 2020-11-16T21:12:29.882 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.882 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:095fd509:::obj-fill-210:head# 2020-11-16T21:12:29.882 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.882 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0a52a594:::obj-fill-116:head# 2020-11-16T21:12:29.882 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.882 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0a63716b:::obj-fill-225:head# 2020-11-16T21:12:29.883 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.883 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0b0fa6fb:::obj-fill-541:head# 2020-11-16T21:12:29.883 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.883 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0b673f72:::obj-fill-232:head# 2020-11-16T21:12:29.883 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.883 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0bbcd1b2:::obj-fill-582:head# 2020-11-16T21:12:29.883 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.884 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0caf8ce3:::obj-fill-214:head# 2020-11-16T21:12:29.884 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.884 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0d94b9fb:::obj-fill-415:head# 2020-11-16T21:12:29.884 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.884 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0de4e6f7:::obj-fill-94:head# 2020-11-16T21:12:29.884 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.884 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0ef75768:::obj-fill-489:head# 2020-11-16T21:12:29.884 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.885 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0f67284f:::obj-fill-213:head# 2020-11-16T21:12:29.885 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.885 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:0f823df2:::obj-fill-234:head# 2020-11-16T21:12:29.885 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.885 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1005c08d:::obj-fill-490:head# 2020-11-16T21:12:29.885 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.885 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1039525f:::obj-fill-369:head# 2020-11-16T21:12:29.885 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.886 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:105fb36c:::obj-fill-226:head# 2020-11-16T21:12:29.886 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.886 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:111abdeb:::obj-fill-554:head# 2020-11-16T21:12:29.886 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.886 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:11392f3d:::obj-fill-453:head# 2020-11-16T21:12:29.886 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.886 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1169976c:::obj-fill-417:head# 2020-11-16T21:12:29.887 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.887 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:11840a2a:::obj-fill-477:head# 2020-11-16T21:12:29.887 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.887 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:11a4832e:::obj-fill-279:head# 2020-11-16T21:12:29.887 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.887 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:11a4d2f6:::obj-fill-256:head# 2020-11-16T21:12:29.887 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.888 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:12d52a31:::obj-fill-22:head# 2020-11-16T21:12:29.888 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.888 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:133c45db:::obj-fill-257:head# 2020-11-16T21:12:29.888 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.888 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:13563256:::obj-fill-150:head# 2020-11-16T21:12:29.888 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.889 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:13c09e00:::obj-fill-71:head# 2020-11-16T21:12:29.889 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.889 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:13e9fbed:::obj-fill-242:head# 2020-11-16T21:12:29.889 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.889 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:13fae6c2:::obj-fill-163:head# 2020-11-16T21:12:29.889 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.889 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:145025e0:::obj-fill-543:head# 2020-11-16T21:12:29.890 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.890 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:14517c84:::obj-fill-13:head# 2020-11-16T21:12:29.890 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.890 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:14802e9c:::obj-fill-127:head# 2020-11-16T21:12:29.890 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.890 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:150f1c96:::obj-fill-62:head# 2020-11-16T21:12:29.890 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.891 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:155e90a2:::obj-fill-537:head# 2020-11-16T21:12:29.891 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.891 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1632c776:::obj-fill-262:head# 2020-11-16T21:12:29.891 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.891 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:163bde28:::obj-fill-523:head# 2020-11-16T21:12:29.891 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.891 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:169c4ad0:::obj-fill-85:head# 2020-11-16T21:12:29.891 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.892 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:16c4da1d:::obj-fill-436:head# 2020-11-16T21:12:29.892 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.892 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:17b28c32:::obj-fill-345:head# 2020-11-16T21:12:29.892 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.892 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:17b34255:::obj-fill-577:head# 2020-11-16T21:12:29.892 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.892 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:17d2541f:::obj-fill-426:head# 2020-11-16T21:12:29.893 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.893 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:17ea19f2:::obj-fill-542:head# 2020-11-16T21:12:29.893 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.893 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:189155ea:::obj-fill-496:head# 2020-11-16T21:12:29.893 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.893 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:191a6d81:::obj-fill-548:head# 2020-11-16T21:12:29.893 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.893 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1a964993:::obj-fill-576:head# 2020-11-16T21:12:29.894 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.894 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1aa623dc:::obj-fill-356:head# 2020-11-16T21:12:29.894 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.894 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1abe5c38:::obj-fill-561:head# 2020-11-16T21:12:29.894 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.894 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1bddc06b:::obj-fill-7:head# 2020-11-16T21:12:29.894 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.895 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1be86b71:::obj-fill-295:head# 2020-11-16T21:12:29.895 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.895 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1c499bca:::obj-fill-240:head# 2020-11-16T21:12:29.895 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.895 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1c545e27:::obj-fill-509:head# 2020-11-16T21:12:29.895 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.895 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1c63c8f4:::obj-fill-4:head# 2020-11-16T21:12:29.895 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.896 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1d5a8f59:::obj-fill-377:head# 2020-11-16T21:12:29.896 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.896 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1d85b5df:::obj-fill-201:head# 2020-11-16T21:12:29.896 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.896 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1d9a0d5b:::obj-fill-581:head# 2020-11-16T21:12:29.896 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.896 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1dcf5667:::obj-fill-488:head# 2020-11-16T21:12:29.897 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.897 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1fa3090d:::obj-fill-458:head# 2020-11-16T21:12:29.897 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.897 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1fe937f6:::obj-fill-337:head# 2020-11-16T21:12:29.897 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.897 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:1ff54542:::obj-fill-349:head# 2020-11-16T21:12:29.897 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.898 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:20d9e0ca:::obj-fill-367:head# 2020-11-16T21:12:29.898 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.898 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:21b8e3bb:::obj-fill-538:head# 2020-11-16T21:12:29.898 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.898 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:227183d4:::obj-fill-235:head# 2020-11-16T21:12:29.898 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.898 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:229e5e5d:::obj-fill-522:head# 2020-11-16T21:12:29.898 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.899 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:234f8861:::obj-fill-476:head# 2020-11-16T21:12:29.899 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.899 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:23ac2c9d:::obj-fill-471:head# 2020-11-16T21:12:29.899 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.899 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:25db1742:::obj-fill-200:head# 2020-11-16T21:12:29.899 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.899 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:264f8545:::obj-fill-230:head# 2020-11-16T21:12:29.900 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.900 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:267ba97f:::obj-fill-493:head# 2020-11-16T21:12:29.900 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.900 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:26b067e8:::obj-fill-208:head# 2020-11-16T21:12:29.900 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.900 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:26f521cb:::obj-fill-300:head# 2020-11-16T21:12:29.900 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.900 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:27e70a11:::obj-fill-368:head# 2020-11-16T21:12:29.901 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.901 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:29357d4a:::obj-fill-456:head# 2020-11-16T21:12:29.901 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.901 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:29f2567e:::obj-fill-414:head# 2020-11-16T21:12:29.901 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.901 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2a10cbb6:::obj-fill-228:head# 2020-11-16T21:12:29.901 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.901 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2a1519a6:::obj-fill-148:head# 2020-11-16T21:12:29.902 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.902 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2a686ca6:::obj-fill-411:head# 2020-11-16T21:12:29.902 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.902 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2ab3f384:::obj-fill-77:head# 2020-11-16T21:12:29.902 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.902 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2ae6a1d0:::obj-fill-187:head# 2020-11-16T21:12:29.902 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.903 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2b890959:::obj-fill-222:head# 2020-11-16T21:12:29.903 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.903 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2bcc26c4:::obj-fill-114:head# 2020-11-16T21:12:29.903 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.903 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2bd3a3a2:::obj-fill-281:head# 2020-11-16T21:12:29.903 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.903 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2c1a21d7:::obj-fill-505:head# 2020-11-16T21:12:29.903 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.904 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2c255347:::obj-fill-95:head# 2020-11-16T21:12:29.904 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.904 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2c2a3f62:::obj-fill-131:head# 2020-11-16T21:12:29.904 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.904 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2c7d92ce:::obj-fill-442:head# 2020-11-16T21:12:29.904 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.905 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2d1ad117:::obj-fill-443:head# 2020-11-16T21:12:29.905 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.905 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2d57b2f7:::obj-fill-192:head# 2020-11-16T21:12:29.905 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.905 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2d7b53ea:::obj-fill-79:head# 2020-11-16T21:12:29.905 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.905 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2db25266:::obj-fill-121:head# 2020-11-16T21:12:29.905 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.906 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2e27853e:::obj-fill-473:head# 2020-11-16T21:12:29.906 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.906 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2f12cb21:::obj-fill-386:head# 2020-11-16T21:12:29.906 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.906 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2f957332:::obj-fill-422:head# 2020-11-16T21:12:29.906 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.906 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:2feae493:::obj-fill-301:head# 2020-11-16T21:12:29.907 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.907 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:302c95d2:::obj-fill-110:head# 2020-11-16T21:12:29.907 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.907 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:302d6b76:::obj-fill-374:head# 2020-11-16T21:12:29.907 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.907 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:30581f34:::obj-fill-313:head# 2020-11-16T21:12:29.907 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.908 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:307e86ac:::obj-fill-381:head# 2020-11-16T21:12:29.908 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.908 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:30ab77bf:::obj-fill-275:head# 2020-11-16T21:12:29.908 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.909 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:31127e6d:::obj-fill-250:head# 2020-11-16T21:12:29.909 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.909 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:32c93500:::obj-fill-155:head# 2020-11-16T21:12:29.909 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.910 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:339880b7:::obj-fill-227:head# 2020-11-16T21:12:29.910 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.910 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:34069e1e:::obj-fill-292:head# 2020-11-16T21:12:29.910 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.910 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:346dff31:::obj-fill-157:head# 2020-11-16T21:12:29.911 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.911 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:34ade0f1:::obj-fill-575:head# 2020-11-16T21:12:29.911 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.911 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:34bbae43:::obj-fill-90:head# 2020-11-16T21:12:29.911 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.911 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:34e97cf8:::obj-fill-334:head# 2020-11-16T21:12:29.912 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.912 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:35d7ccc2:::obj-fill-510:head# 2020-11-16T21:12:29.912 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.912 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:35f519b6:::obj-fill-513:head# 2020-11-16T21:12:29.912 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.912 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:364727e8:::obj-fill-130:head# 2020-11-16T21:12:29.913 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.913 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:364c0361:::obj-fill-586:head# 2020-11-16T21:12:29.913 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.913 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:368e7baf:::obj-fill-546:head# 2020-11-16T21:12:29.913 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.913 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:36ffb224:::obj-fill-82:head# 2020-11-16T21:12:29.913 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.914 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:371cb800:::obj-fill-330:head# 2020-11-16T21:12:29.914 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.914 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:37d5378d:::obj-fill-175:head# 2020-11-16T21:12:29.914 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.914 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:385593f3:::obj-fill-325:head# 2020-11-16T21:12:29.914 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.914 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:388d5073:::obj-fill-317:head# 2020-11-16T21:12:29.915 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.915 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:394af079:::obj-fill-59:head# 2020-11-16T21:12:29.915 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.915 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3a41ad37:::obj-fill-409:head# 2020-11-16T21:12:29.915 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.915 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3b3c63d2:::obj-fill-136:head# 2020-11-16T21:12:29.915 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.916 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3bb50131:::obj-fill-512:head# 2020-11-16T21:12:29.916 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.916 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3bb52bb5:::obj-fill-188:head# 2020-11-16T21:12:29.916 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.916 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3c0683d1:::obj-fill-372:head# 2020-11-16T21:12:29.916 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.916 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3c67d3f4:::obj-fill-249:head# 2020-11-16T21:12:29.916 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.917 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3c6a4a4c:::obj-fill-495:head# 2020-11-16T21:12:29.917 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.917 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3c7fc8c9:::obj-fill-324:head# 2020-11-16T21:12:29.917 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.917 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3d940c50:::obj-fill-591:head# 2020-11-16T21:12:29.917 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.917 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3ed5508e:::obj-fill-558:head# 2020-11-16T21:12:29.918 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.918 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:3f598316:::obj-fill-389:head# 2020-11-16T21:12:29.918 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.918 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:401cd0ce:::obj-fill-251:head# 2020-11-16T21:12:29.918 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.918 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:40fae006:::obj-fill-14:head# 2020-11-16T21:12:29.918 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.919 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4120b10f:::obj-fill-165:head# 2020-11-16T21:12:29.919 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.919 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:41b382c2:::obj-fill-289:head# 2020-11-16T21:12:29.919 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.919 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4277cdea:::obj-fill-319:head# 2020-11-16T21:12:29.919 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.919 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:42978c3c:::obj-fill-174:head# 2020-11-16T21:12:29.920 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.920 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:42b08dc2:::obj-fill-553:head# 2020-11-16T21:12:29.920 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.920 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:42ec4bf6:::obj-fill-209:head# 2020-11-16T21:12:29.920 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.920 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:430c129d:::obj-fill-448:head# 2020-11-16T21:12:29.920 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.921 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4335d553:::obj-fill-600:head# 2020-11-16T21:12:29.921 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.921 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:44804f62:::obj-fill-532:head# 2020-11-16T21:12:29.921 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.921 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:44954d54:::obj-fill-2:head# 2020-11-16T21:12:29.921 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.921 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4502f904:::obj-fill-429:head# 2020-11-16T21:12:29.922 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.922 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4568ddcb:::obj-fill-382:head# 2020-11-16T21:12:29.922 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.922 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:45939974:::obj-fill-563:head# 2020-11-16T21:12:29.923 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.923 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:462a5bbd:::obj-fill-567:head# 2020-11-16T21:12:29.923 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.923 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:47773ae4:::obj-fill-307:head# 2020-11-16T21:12:29.923 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.923 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4802b263:::obj-fill-387:head# 2020-11-16T21:12:29.923 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.923 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:483d938e:::obj-fill-128:head# 2020-11-16T21:12:29.924 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.924 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:48a7ad29:::obj-fill-147:head# 2020-11-16T21:12:29.924 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.924 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:493b61ce:::obj-fill-454:head# 2020-11-16T21:12:29.924 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.924 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4afb70fe:::obj-fill-186:head# 2020-11-16T21:12:29.924 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.925 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4bff08e3:::obj-fill-352:head# 2020-11-16T21:12:29.925 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.925 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4c01eabe:::obj-fill-370:head# 2020-11-16T21:12:29.925 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.925 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4c131560:::obj-fill-6:head# 2020-11-16T21:12:29.925 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.925 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4c4c5e14:::obj-fill-350:head# 2020-11-16T21:12:29.925 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.926 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4c80dc2c:::obj-fill-357:head# 2020-11-16T21:12:29.926 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.926 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4c9dee9b:::obj-fill-540:head# 2020-11-16T21:12:29.926 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.926 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4ce40c18:::obj-fill-280:head# 2020-11-16T21:12:29.926 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.926 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4d854fc3:::obj-fill-585:head# 2020-11-16T21:12:29.927 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.927 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:4f615b53:::obj-fill-101:head# 2020-11-16T21:12:29.927 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.927 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5042b30f:::obj-fill-84:head# 2020-11-16T21:12:29.927 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.928 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:507dc08b:::obj-fill-336:head# 2020-11-16T21:12:29.928 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.928 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:512fcf5a:::obj-fill-169:head# 2020-11-16T21:12:29.928 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.928 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:51a0c8f8:::obj-fill-578:head# 2020-11-16T21:12:29.928 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.928 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:51fd05a2:::obj-fill-425:head# 2020-11-16T21:12:29.929 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.929 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:52400ebc:::obj-fill-81:head# 2020-11-16T21:12:29.929 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.929 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5288d40c:::obj-fill-16:head# 2020-11-16T21:12:29.929 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.929 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5295b0ee:::obj-fill-224:head# 2020-11-16T21:12:29.930 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.930 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:531e4c60:::obj-fill-103:head# 2020-11-16T21:12:29.930 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.930 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:54259f28:::obj-fill-584:head# 2020-11-16T21:12:29.930 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.931 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:542d619c:::obj-fill-528:head# 2020-11-16T21:12:29.931 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.931 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5503ca3e:::obj-fill-494:head# 2020-11-16T21:12:29.931 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.931 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5555c48b:::obj-fill-403:head# 2020-11-16T21:12:29.931 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.931 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:558d1ec7:::obj-fill-447:head# 2020-11-16T21:12:29.931 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.932 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:55aa2233:::obj-fill-21:head# 2020-11-16T21:12:29.932 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.932 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:55aaf8c4:::obj-fill-29:head# 2020-11-16T21:12:29.932 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.932 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:55e246e1:::obj-fill-65:head# 2020-11-16T21:12:29.932 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.932 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:563b14a8:::obj-fill-120:head# 2020-11-16T21:12:29.933 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.933 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:56e1df25:::obj-fill-305:head# 2020-11-16T21:12:29.933 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.933 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5702979c:::obj-fill-73:head# 2020-11-16T21:12:29.933 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.933 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:570722ba:::obj-fill-219:head# 2020-11-16T21:12:29.934 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.934 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:57f7a09b:::obj-fill-571:head# 2020-11-16T21:12:29.934 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.934 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:585309ba:::obj-fill-144:head# 2020-11-16T21:12:29.934 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.934 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:585c7c9c:::obj-fill-437:head# 2020-11-16T21:12:29.934 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.935 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:587c63db:::obj-fill-182:head# 2020-11-16T21:12:29.935 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.935 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:58f34d6b:::obj-fill-598:head# 2020-11-16T21:12:29.935 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.935 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:58f40088:::obj-fill-298:head# 2020-11-16T21:12:29.935 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.935 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:591e150c:::obj-fill-32:head# 2020-11-16T21:12:29.936 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.936 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:59971ddb:::obj-fill-559:head# 2020-11-16T21:12:29.936 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.936 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5adafca4:::obj-fill-56:head# 2020-11-16T21:12:29.936 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.936 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5b32b04f:::obj-fill-484:head# 2020-11-16T21:12:29.936 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.937 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5bdaeb16:::obj-fill-99:head# 2020-11-16T21:12:29.937 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.937 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5c1d6f4b:::obj-fill-133:head# 2020-11-16T21:12:29.937 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.937 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5c92b8f1:::obj-fill-255:head# 2020-11-16T21:12:29.937 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.937 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5ca4850f:::obj-fill-238:head# 2020-11-16T21:12:29.937 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.938 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5d928a5b:::obj-fill-395:head# 2020-11-16T21:12:29.938 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.938 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5dd49320:::obj-fill-423:head# 2020-11-16T21:12:29.938 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.938 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5e7bc368:::obj-fill-506:head# 2020-11-16T21:12:29.939 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.939 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5ebd5ee3:::obj-fill-322:head# 2020-11-16T21:12:29.939 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.939 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5edeedf1:::obj-fill-557:head# 2020-11-16T21:12:29.939 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.939 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:5facf8e2:::obj-fill-15:head# 2020-11-16T21:12:29.940 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.940 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:603a2c76:::obj-fill-439:head# 2020-11-16T21:12:29.940 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.940 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:60b30714:::obj-fill-53:head# 2020-11-16T21:12:29.940 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.940 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:612b4a8a:::obj-fill-412:head# 2020-11-16T21:12:29.941 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.941 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:61a111b6:::obj-fill-87:head# 2020-11-16T21:12:29.941 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.941 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:63a0e8a2:::obj-fill-266:head# 2020-11-16T21:12:29.941 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.941 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:63aa4d60:::obj-fill-68:head# 2020-11-16T21:12:29.941 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.942 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:63ac5b95:::obj-fill-405:head# 2020-11-16T21:12:29.942 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.942 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6415a9e3:::obj-fill-400:head# 2020-11-16T21:12:29.942 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.942 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:64ba03ae:::obj-fill-20:head# 2020-11-16T21:12:29.942 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.942 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6512f015:::obj-fill-597:head# 2020-11-16T21:12:29.943 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.943 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:65433104:::obj-fill-504:head# 2020-11-16T21:12:29.943 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.943 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:65950c58:::obj-fill-86:head# 2020-11-16T21:12:29.943 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.943 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:65b6b2d7:::obj-fill-39:head# 2020-11-16T21:12:29.943 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.944 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:661c8afd:::obj-fill-267:head# 2020-11-16T21:12:29.944 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.944 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:66f87c94:::obj-fill-229:head# 2020-11-16T21:12:29.944 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.944 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:67292cef:::obj-fill-93:head# 2020-11-16T21:12:29.944 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.944 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:67f0e12c:::obj-fill-363:head# 2020-11-16T21:12:29.944 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.945 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:696eb83b:::obj-fill-185:head# 2020-11-16T21:12:29.945 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.945 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6b33981c:::obj-fill-259:head# 2020-11-16T21:12:29.945 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.945 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6b5a527c:::obj-fill-485:head# 2020-11-16T21:12:29.945 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.945 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6b6f1852:::obj-fill-390:head# 2020-11-16T21:12:29.946 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.946 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6ba5c5f8:::obj-fill-304:head# 2020-11-16T21:12:29.946 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.946 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6bce929d:::obj-fill-518:head# 2020-11-16T21:12:29.946 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.946 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6c8ffc04:::obj-fill-61:head# 2020-11-16T21:12:29.946 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.946 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6ca7e551:::obj-fill-40:head# 2020-11-16T21:12:29.947 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.947 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6caff2cd:::obj-fill-332:head# 2020-11-16T21:12:29.947 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.947 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6ce4adc7:::obj-fill-574:head# 2020-11-16T21:12:29.947 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.947 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6d09d770:::obj-fill-74:head# 2020-11-16T21:12:29.948 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.948 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6e039822:::obj-fill-278:head# 2020-11-16T21:12:29.948 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.948 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6f2cd109:::obj-fill-297:head# 2020-11-16T21:12:29.948 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.948 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6f3754c4:::obj-fill-88:head# 2020-11-16T21:12:29.949 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.949 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6f86cfd2:::obj-fill-511:head# 2020-11-16T21:12:29.949 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.949 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:6fe3f3f7:::obj-fill-193:head# 2020-11-16T21:12:29.949 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.949 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7002c57d:::obj-fill-50:head# 2020-11-16T21:12:29.949 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.950 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7030b2a3:::obj-fill-217:head# 2020-11-16T21:12:29.950 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.950 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:70665087:::obj-fill-51:head# 2020-11-16T21:12:29.950 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.950 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:70a0f5e0:::obj-fill-11:head# 2020-11-16T21:12:29.950 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.950 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:71f4643d:::obj-fill-236:head# 2020-11-16T21:12:29.950 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.951 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:722fbd0f:::obj-fill-302:head# 2020-11-16T21:12:29.951 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.951 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:72584da9:::obj-fill-445:head# 2020-11-16T21:12:29.951 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.951 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:72d91d24:::obj-fill-218:head# 2020-11-16T21:12:29.951 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.951 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:72f7268b:::obj-fill-106:head# 2020-11-16T21:12:29.952 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.952 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:74077abf:::obj-fill-435:head# 2020-11-16T21:12:29.952 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.952 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:740ca050:::obj-fill-112:head# 2020-11-16T21:12:29.952 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.952 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7435e881:::obj-fill-78:head# 2020-11-16T21:12:29.952 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.953 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:74ba0bb7:::obj-fill-478:head# 2020-11-16T21:12:29.953 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.953 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:74d82091:::obj-fill-96:head# 2020-11-16T21:12:29.953 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.953 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7515f56c:::obj-fill-483:head# 2020-11-16T21:12:29.953 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.953 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7550d6fe:::obj-fill-286:head# 2020-11-16T21:12:29.953 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.954 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:75807edc:::obj-fill-223:head# 2020-11-16T21:12:29.954 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.954 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:762ba324:::obj-fill-172:head# 2020-11-16T21:12:29.954 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.954 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:76369bda:::obj-fill-570:head# 2020-11-16T21:12:29.954 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.954 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7667a5b0:::obj-fill-196:head# 2020-11-16T21:12:29.955 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.955 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:77736c0c:::obj-fill-566:head# 2020-11-16T21:12:29.955 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.955 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:77fb47a4:::obj-fill-314:head# 2020-11-16T21:12:29.955 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.955 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:785ad894:::obj-fill-80:head# 2020-11-16T21:12:29.955 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.955 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:785e3d14:::obj-fill-530:head# 2020-11-16T21:12:29.956 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.956 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:788efef4:::obj-fill-595:head# 2020-11-16T21:12:29.956 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.956 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:789ec5ab:::obj-fill-44:head# 2020-11-16T21:12:29.956 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.956 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:798a3fa8:::obj-fill-517:head# 2020-11-16T21:12:29.957 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.957 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:79a105ef:::obj-fill-315:head# 2020-11-16T21:12:29.957 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.957 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:79d02f04:::obj-fill-243:head# 2020-11-16T21:12:29.957 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.957 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7a00dfcb:::obj-fill-402:head# 2020-11-16T21:12:29.957 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.958 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7b982684:::obj-fill-168:head# 2020-11-16T21:12:29.958 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.958 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7bb36851:::obj-fill-440:head# 2020-11-16T21:12:29.958 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.958 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7bb49d8c:::obj-fill-55:head# 2020-11-16T21:12:29.958 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.959 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7c77b783:::obj-fill-580:head# 2020-11-16T21:12:29.959 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.959 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7c89d75b:::obj-fill-555:head# 2020-11-16T21:12:29.959 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.959 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7d12fb67:::obj-fill-399:head# 2020-11-16T21:12:29.959 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.959 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7d139961:::obj-fill-92:head# 2020-11-16T21:12:29.960 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.960 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7d39ffd0:::obj-fill-135:head# 2020-11-16T21:12:29.960 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.960 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7e534cc4:::obj-fill-282:head# 2020-11-16T21:12:29.960 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.960 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7e7a08e0:::obj-fill-108:head# 2020-11-16T21:12:29.960 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.960 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7e8e2b67:::obj-fill-244:head# 2020-11-16T21:12:29.961 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.961 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7eea215e:::obj-fill-139:head# 2020-11-16T21:12:29.961 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.961 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7efa3c6e:::obj-fill-408:head# 2020-11-16T21:12:29.961 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.961 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7f03a24b:::obj-fill-45:head# 2020-11-16T21:12:29.961 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.962 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7f6a2e2b:::obj-fill-427:head# 2020-11-16T21:12:29.962 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.962 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7fb0abd6:::obj-fill-479:head# 2020-11-16T21:12:29.962 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.962 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:7fb77de1:::obj-fill-167:head# 2020-11-16T21:12:29.962 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.962 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:80012a03:::obj-fill-105:head# 2020-11-16T21:12:29.962 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.963 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:80075ae5:::obj-fill-376:head# 2020-11-16T21:12:29.963 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.963 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8089baa5:::obj-fill-97:head# 2020-11-16T21:12:29.963 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.963 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:80d0b4c5:::obj-fill-123:head# 2020-11-16T21:12:29.963 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.963 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:810f319f:::obj-fill-552:head# 2020-11-16T21:12:29.964 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.964 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:81427ef2:::obj-fill-474:head# 2020-11-16T21:12:29.964 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.964 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:818d780c:::obj-fill-33:head# 2020-11-16T21:12:29.964 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.964 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:819f31b9:::obj-fill-366:head# 2020-11-16T21:12:29.964 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.965 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8205549d:::obj-fill-125:head# 2020-11-16T21:12:29.965 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.965 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:821f56e5:::obj-fill-339:head# 2020-11-16T21:12:29.965 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.965 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:822807c6:::obj-fill-69:head# 2020-11-16T21:12:29.965 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.965 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:822fcf61:::obj-fill-560:head# 2020-11-16T21:12:29.966 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.966 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:82ad696b:::obj-fill-149:head# 2020-11-16T21:12:29.966 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.966 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:835cc771:::obj-fill-446:head# 2020-11-16T21:12:29.966 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.966 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8386bee1:::obj-fill-137:head# 2020-11-16T21:12:29.966 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.967 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:83c1303a:::obj-fill-311:head# 2020-11-16T21:12:29.967 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.967 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:83cf1c2a:::obj-fill-360:head# 2020-11-16T21:12:29.967 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.967 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:83d70321:::obj-fill-351:head# 2020-11-16T21:12:29.967 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.968 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:83ef162d:::obj-fill-331:head# 2020-11-16T21:12:29.968 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.968 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:840f8f20:::obj-fill-398:head# 2020-11-16T21:12:29.968 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.968 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:84d2e456:::obj-fill-207:head# 2020-11-16T21:12:29.968 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.968 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:856697ff:::obj-fill-333:head# 2020-11-16T21:12:29.968 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.969 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:858be937:::obj-fill-418:head# 2020-11-16T21:12:29.969 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.969 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:85a0b302:::obj-fill-438:head# 2020-11-16T21:12:29.969 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.969 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:85fa942e:::obj-fill-272:head# 2020-11-16T21:12:29.969 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.969 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:863eb6f8:::obj-fill-320:head# 2020-11-16T21:12:29.970 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.970 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:868d4588:::obj-fill-19:head# 2020-11-16T21:12:29.970 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.970 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8759d023:::obj-fill-466:head# 2020-11-16T21:12:29.970 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.970 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:88072111:::obj-fill-312:head# 2020-11-16T21:12:29.970 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.970 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:886edcef:::obj-fill-380:head# 2020-11-16T21:12:29.971 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.971 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:88ce753e:::obj-fill-43:head# 2020-11-16T21:12:29.971 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.971 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:890e1d83:::obj-fill-590:head# 2020-11-16T21:12:29.971 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.971 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:891566a3:::obj-fill-341:head# 2020-11-16T21:12:29.971 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.972 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:891ca235:::obj-fill-177:head# 2020-11-16T21:12:29.972 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.972 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:897fa9d0:::obj-fill-283:head# 2020-11-16T21:12:29.972 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.972 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:89cd304b:::obj-fill-592:head# 2020-11-16T21:12:29.972 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.972 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:89d04c4a:::obj-fill-388:head# 2020-11-16T21:12:29.972 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.973 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:89db45d9:::obj-fill-247:head# 2020-11-16T21:12:29.973 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.973 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8ad879c5:::obj-fill-492:head# 2020-11-16T21:12:29.973 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.973 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8af4b5de:::obj-fill-37:head# 2020-11-16T21:12:29.973 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.974 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8b132e86:::obj-fill-35:head# 2020-11-16T21:12:29.974 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.974 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8b80a60a:::obj-fill-100:head# 2020-11-16T21:12:29.974 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.974 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8bb64fa6:::obj-fill-599:head# 2020-11-16T21:12:29.974 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.974 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8bed03f5:::obj-fill-5:head# 2020-11-16T21:12:29.975 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.975 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8c60e135:::obj-fill-470:head# 2020-11-16T21:12:29.975 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.975 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8c774623:::obj-fill-38:head# 2020-11-16T21:12:29.975 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.975 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8cc528ae:::obj-fill-519:head# 2020-11-16T21:12:29.975 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.976 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8cf73826:::obj-fill-460:head# 2020-11-16T21:12:29.976 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.976 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8d3564b3:::obj-fill-452:head# 2020-11-16T21:12:29.976 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.976 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8dfe1ca2:::obj-fill-469:head# 2020-11-16T21:12:29.976 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.976 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8e527672:::obj-fill-246:head# 2020-11-16T21:12:29.977 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.977 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8e96103f:::obj-fill-444:head# 2020-11-16T21:12:29.977 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.977 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8ed6417f:::obj-fill-146:head# 2020-11-16T21:12:29.977 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.977 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8eda2ffc:::obj-fill-539:head# 2020-11-16T21:12:29.977 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.978 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8f08dd48:::obj-fill-26:head# 2020-11-16T21:12:29.978 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.978 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8f111b25:::obj-fill-18:head# 2020-11-16T21:12:29.978 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.978 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:8fde63f8:::obj-fill-338:head# 2020-11-16T21:12:29.978 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.978 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9030746d:::obj-fill-468:head# 2020-11-16T21:12:29.979 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.979 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9115d8a1:::obj-fill-260:head# 2020-11-16T21:12:29.979 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.979 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:91649909:::obj-fill-140:head# 2020-11-16T21:12:29.979 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.979 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:91c1081d:::obj-fill-404:head# 2020-11-16T21:12:29.979 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.980 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:91f42503:::obj-fill-216:head# 2020-11-16T21:12:29.980 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.980 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:927b34a0:::obj-fill-63:head# 2020-11-16T21:12:29.980 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.980 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:92a00a91:::obj-fill-467:head# 2020-11-16T21:12:29.980 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.980 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:930a7268:::obj-fill-450:head# 2020-11-16T21:12:29.980 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.981 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:93c3d0bf:::obj-fill-118:head# 2020-11-16T21:12:29.981 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.981 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:93ca7c9e:::obj-fill-132:head# 2020-11-16T21:12:29.981 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.981 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:942435d5:::obj-fill-378:head# 2020-11-16T21:12:29.981 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.981 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9445f8c5:::obj-fill-245:head# 2020-11-16T21:12:29.982 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.982 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9457c8d9:::obj-fill-535:head# 2020-11-16T21:12:29.982 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.982 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:950dc56e:::obj-fill-348:head# 2020-11-16T21:12:29.982 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.982 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9559ee1d:::obj-fill-545:head# 2020-11-16T21:12:29.982 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.983 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:957c5cc4:::obj-fill-212:head# 2020-11-16T21:12:29.983 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.983 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:95bdac02:::obj-fill-588:head# 2020-11-16T21:12:29.983 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.983 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:95f202af:::obj-fill-126:head# 2020-11-16T21:12:29.983 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.983 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:96789003:::obj-fill-83:head# 2020-11-16T21:12:29.984 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.984 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:96d494a8:::obj-fill-316:head# 2020-11-16T21:12:29.984 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.984 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:97adb85b:::obj-fill-233:head# 2020-11-16T21:12:29.984 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.984 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:980d998c:::obj-fill-197:head# 2020-11-16T21:12:29.984 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.985 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:98235c44:::obj-fill-396:head# 2020-11-16T21:12:29.985 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.985 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:987d3b87:::obj-fill-327:head# 2020-11-16T21:12:29.985 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.985 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:990a5e24:::obj-fill-515:head# 2020-11-16T21:12:29.985 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.986 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:995fec05:::obj-fill-191:head# 2020-11-16T21:12:29.986 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.986 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9a769b75:::obj-fill-587:head# 2020-11-16T21:12:29.986 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.986 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9b141990:::obj-fill-394:head# 2020-11-16T21:12:29.986 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.986 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9bb108fb:::obj-fill-579:head# 2020-11-16T21:12:29.987 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.987 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9ce19d92:::obj-fill-361:head# 2020-11-16T21:12:29.987 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.987 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9d8c3f95:::obj-fill-572:head# 2020-11-16T21:12:29.987 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.987 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9e283e00:::obj-fill-254:head# 2020-11-16T21:12:29.988 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.988 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9e3a1a61:::obj-fill-58:head# 2020-11-16T21:12:29.988 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.988 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9e8f6029:::obj-fill-41:head# 2020-11-16T21:12:29.988 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.989 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9ef117aa:::obj-fill-520:head# 2020-11-16T21:12:29.989 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.989 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9f3bdfc2:::obj-fill-184:head# 2020-11-16T21:12:29.989 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.989 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9fd89320:::obj-fill-364:head# 2020-11-16T21:12:29.990 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.990 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:9feb7062:::obj-fill-419:head# 2020-11-16T21:12:29.990 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.990 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a07504c4:::obj-fill-516:head# 2020-11-16T21:12:29.990 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.991 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a1897e34:::obj-fill-153:head# 2020-11-16T21:12:29.991 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.991 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a19a9d76:::obj-fill-30:head# 2020-11-16T21:12:29.991 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.991 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a1a49c98:::obj-fill-98:head# 2020-11-16T21:12:29.992 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.992 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a1ea911a:::obj-fill-36:head# 2020-11-16T21:12:29.992 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.992 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a24bad98:::obj-fill-335:head# 2020-11-16T21:12:29.992 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.993 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a263b16f:::obj-fill-326:head# 2020-11-16T21:12:29.993 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.993 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a27d661e:::obj-fill-152:head# 2020-11-16T21:12:29.993 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.993 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a282a129:::obj-fill-451:head# 2020-11-16T21:12:29.993 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.994 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a28f05ba:::obj-fill-241:head# 2020-11-16T21:12:29.994 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.994 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a2cdc5cc:::obj-fill-180:head# 2020-11-16T21:12:29.994 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.995 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a30777ea:::obj-fill-486:head# 2020-11-16T21:12:29.995 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.995 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a3a65661:::obj-fill-151:head# 2020-11-16T21:12:29.995 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.995 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a3cb2967:::obj-fill-195:head# 2020-11-16T21:12:29.996 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.996 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a3ffc098:::obj-fill-293:head# 2020-11-16T21:12:29.996 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.996 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a4a31ccc:::obj-fill-413:head# 2020-11-16T21:12:29.996 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.997 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a51c3677:::obj-fill-384:head# 2020-11-16T21:12:29.997 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.997 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a5e941a7:::obj-fill-141:head# 2020-11-16T21:12:29.997 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.997 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a6cdd8b2:::obj-fill-342:head# 2020-11-16T21:12:29.998 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.998 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a6e20b86:::obj-fill-593:head# 2020-11-16T21:12:29.998 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.998 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a7a9ccf9:::obj-fill-129:head# 2020-11-16T21:12:29.998 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.999 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a7b702d3:::obj-fill-49:head# 2020-11-16T21:12:29.999 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.999 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a8daf8bd:::obj-fill-391:head# 2020-11-16T21:12:29.999 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:29.999 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:a99c3cef:::obj-fill-171:head# 2020-11-16T21:12:30.000 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.000 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:aabb5718:::obj-fill-521:head# 2020-11-16T21:12:30.000 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.000 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ab0cb851:::obj-fill-183:head# 2020-11-16T21:12:30.000 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.001 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ab7a05e4:::obj-fill-203:head# 2020-11-16T21:12:30.001 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.001 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ab7dda83:::obj-fill-113:head# 2020-11-16T21:12:30.001 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.001 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:abb3502c:::obj-fill-461:head# 2020-11-16T21:12:30.001 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.002 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ac1c41be:::obj-fill-162:head# 2020-11-16T21:12:30.002 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.002 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ac2114ba:::obj-fill-239:head# 2020-11-16T21:12:30.002 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.002 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ad355fca:::obj-fill-271:head# 2020-11-16T21:12:30.003 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.003 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ad4634a0:::obj-fill-487:head# 2020-11-16T21:12:30.003 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.003 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:add2bb13:::obj-fill-31:head# 2020-11-16T21:12:30.003 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.004 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:aeb6258d:::obj-fill-124:head# 2020-11-16T21:12:30.004 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.004 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:af17f7d7:::obj-fill-464:head# 2020-11-16T21:12:30.004 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.004 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:af1e1cf0:::obj-fill-237:head# 2020-11-16T21:12:30.004 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.005 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:afc439ce:::obj-fill-248:head# 2020-11-16T21:12:30.005 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.005 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:afceab4d:::obj-fill-527:head# 2020-11-16T21:12:30.005 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.005 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b03af40c:::obj-fill-143:head# 2020-11-16T21:12:30.005 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.005 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b04cf0db:::obj-fill-507:head# 2020-11-16T21:12:30.006 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.006 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b07896f0:::obj-fill-269:head# 2020-11-16T21:12:30.006 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.006 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b0b954d5:::obj-fill-524:head# 2020-11-16T21:12:30.007 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.007 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b1cbfcb7:::obj-fill-134:head# 2020-11-16T21:12:30.007 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.007 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b2978fd3:::obj-fill-27:head# 2020-11-16T21:12:30.007 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.008 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b394798f:::obj-fill-156:head# 2020-11-16T21:12:30.008 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.008 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b3c6f36d:::obj-fill-401:head# 2020-11-16T21:12:30.008 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.008 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b3f8e5ac:::obj-fill-290:head# 2020-11-16T21:12:30.009 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.009 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b41ed5f8:::obj-fill-76:head# 2020-11-16T21:12:30.009 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.010 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b4878e76:::obj-fill-253:head# 2020-11-16T21:12:30.010 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.010 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b4e19741:::obj-fill-503:head# 2020-11-16T21:12:30.010 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.011 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b53a3690:::obj-fill-264:head# 2020-11-16T21:12:30.011 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.011 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b545ab4f:::obj-fill-24:head# 2020-11-16T21:12:30.011 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.012 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b55af5e6:::obj-fill-491:head# 2020-11-16T21:12:30.012 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.012 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b57edf82:::obj-fill-526:head# 2020-11-16T21:12:30.012 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.012 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b589d2be:::obj-fill-323:head# 2020-11-16T21:12:30.013 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.013 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b6207f32:::obj-fill-344:head# 2020-11-16T21:12:30.013 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.013 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b757ee2e:::obj-fill-433:head# 2020-11-16T21:12:30.013 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.015 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b75cc479:::obj-fill-34:head# 2020-11-16T21:12:30.016 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.016 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b7b03117:::obj-fill-42:head# 2020-11-16T21:12:30.016 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.016 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b7e6205d:::obj-fill-501:head# 2020-11-16T21:12:30.017 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.017 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b8d18b7a:::obj-fill-353:head# 2020-11-16T21:12:30.017 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.017 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b9189c29:::obj-fill-328:head# 2020-11-16T21:12:30.018 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.018 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b98e07aa:::obj-fill-54:head# 2020-11-16T21:12:30.018 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.018 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:b9cd1e9a:::obj-fill-371:head# 2020-11-16T21:12:30.018 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.018 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:bc3322bb:::obj-fill-340:head# 2020-11-16T21:12:30.019 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.019 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:bcf43161:::obj-fill-294:head# 2020-11-16T21:12:30.019 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.019 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:bd07fc4f:::obj-fill-549:head# 2020-11-16T21:12:30.020 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.020 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:bdf48e47:::obj-fill-430:head# 2020-11-16T21:12:30.020 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.020 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:be01c111:::obj-fill-1:head# 2020-11-16T21:12:30.021 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.021 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:be3a4bc7:::obj-fill-102:head# 2020-11-16T21:12:30.021 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.022 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:be8e1fcd:::obj-fill-321:head# 2020-11-16T21:12:30.022 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.022 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:bf5fcbac:::obj-fill-287:head# 2020-11-16T21:12:30.022 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.023 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:bf806d14:::obj-fill-115:head# 2020-11-16T21:12:30.023 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.023 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:bfb28823:::obj-fill-72:head# 2020-11-16T21:12:30.024 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.024 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:bfef805e:::obj-fill-119:head# 2020-11-16T21:12:30.024 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.024 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c02f8e10:::obj-fill-497:head# 2020-11-16T21:12:30.025 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.025 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c050cd6e:::obj-fill-8:head# 2020-11-16T21:12:30.025 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.025 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c218e297:::obj-fill-173:head# 2020-11-16T21:12:30.026 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.026 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c2bc66fe:::obj-fill-252:head# 2020-11-16T21:12:30.027 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.027 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c3b55903:::obj-fill-428:head# 2020-11-16T21:12:30.027 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.027 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c51907b0:::obj-fill-457:head# 2020-11-16T21:12:30.028 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.028 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c561dc83:::obj-fill-75:head# 2020-11-16T21:12:30.028 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.028 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c5d1e1a9:::obj-fill-268:head# 2020-11-16T21:12:30.029 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.029 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c60ab0d0:::obj-fill-303:head# 2020-11-16T21:12:30.029 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.029 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c644ce52:::obj-fill-514:head# 2020-11-16T21:12:30.030 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.030 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c6ab3a11:::obj-fill-407:head# 2020-11-16T21:12:30.030 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.030 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c74377bf:::obj-fill-432:head# 2020-11-16T21:12:30.030 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.031 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c767fff6:::obj-fill-531:head# 2020-11-16T21:12:30.031 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.031 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c7a9adad:::obj-fill-475:head# 2020-11-16T21:12:30.031 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.032 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c7e3a9c2:::obj-fill-465:head# 2020-11-16T21:12:30.032 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.032 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c850aed7:::obj-fill-424:head# 2020-11-16T21:12:30.032 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.032 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c8a0088a:::obj-fill-329:head# 2020-11-16T21:12:30.032 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.033 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:c9d419f9:::obj-fill-176:head# 2020-11-16T21:12:30.033 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.033 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:caf25ec2:::obj-fill-274:head# 2020-11-16T21:12:30.033 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.033 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cb625830:::obj-fill-70:head# 2020-11-16T21:12:30.034 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.035 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cbb65125:::obj-fill-529:head# 2020-11-16T21:12:30.035 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.035 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cbe2544c:::obj-fill-3:head# 2020-11-16T21:12:30.035 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.035 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cbfce4ee:::obj-fill-455:head# 2020-11-16T21:12:30.035 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.036 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cc62419c:::obj-fill-498:head# 2020-11-16T21:12:30.036 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.036 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cce831fa:::obj-fill-160:head# 2020-11-16T21:12:30.036 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.036 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cda71af4:::obj-fill-122:head# 2020-11-16T21:12:30.037 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.037 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cea7a728:::obj-fill-265:head# 2020-11-16T21:12:30.037 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.037 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cee19f0a:::obj-fill-142:head# 2020-11-16T21:12:30.037 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.038 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cf22117e:::obj-fill-499:head# 2020-11-16T21:12:30.038 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.038 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cfbfa66d:::obj-fill-66:head# 2020-11-16T21:12:30.038 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.038 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cfc2552c:::obj-fill-211:head# 2020-11-16T21:12:30.039 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.039 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:cfc563e8:::obj-fill-392:head# 2020-11-16T21:12:30.039 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.039 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d02d8880:::obj-fill-362:head# 2020-11-16T21:12:30.040 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.040 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d054947e:::obj-fill-67:head# 2020-11-16T21:12:30.040 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.040 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d061c977:::obj-fill-164:head# 2020-11-16T21:12:30.040 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.041 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d06a5f57:::obj-fill-296:head# 2020-11-16T21:12:30.041 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.041 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d110062a:::obj-fill-343:head# 2020-11-16T21:12:30.042 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.042 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d1b8ad47:::obj-fill-383:head# 2020-11-16T21:12:30.042 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.042 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d1e80db6:::obj-fill-117:head# 2020-11-16T21:12:30.042 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.043 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d210027e:::obj-fill-231:head# 2020-11-16T21:12:30.043 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.043 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d24428ed:::obj-fill-199:head# 2020-11-16T21:12:30.043 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.043 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d27c3c4b:::obj-fill-154:head# 2020-11-16T21:12:30.044 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.044 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d28957e3:::obj-fill-263:head# 2020-11-16T21:12:30.044 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.044 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d2d44da7:::obj-fill-544:head# 2020-11-16T21:12:30.044 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.045 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d31004a4:::obj-fill-434:head# 2020-11-16T21:12:30.045 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.045 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d32487c3:::obj-fill-385:head# 2020-11-16T21:12:30.045 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.045 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d340891c:::obj-fill-12:head# 2020-11-16T21:12:30.045 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.046 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d4135728:::obj-fill-310:head# 2020-11-16T21:12:30.046 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.046 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d5a43972:::obj-fill-9:head# 2020-11-16T21:12:30.046 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.047 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d5efeb15:::obj-fill-347:head# 2020-11-16T21:12:30.047 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.047 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d6463f70:::obj-fill-536:head# 2020-11-16T21:12:30.047 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.047 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d666d9b7:::obj-fill-420:head# 2020-11-16T21:12:30.047 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.048 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d6e86688:::obj-fill-166:head# 2020-11-16T21:12:30.048 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.048 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d730370d:::obj-fill-206:head# 2020-11-16T21:12:30.048 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.048 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d783551c:::obj-fill-109:head# 2020-11-16T21:12:30.048 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.049 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d7f77e5f:::obj-fill-551:head# 2020-11-16T21:12:30.049 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.049 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d843056c:::obj-fill-291:head# 2020-11-16T21:12:30.049 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.049 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d84eb173:::obj-fill-221:head# 2020-11-16T21:12:30.050 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.050 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d8dd069c:::obj-fill-60:head# 2020-11-16T21:12:30.050 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.050 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d8f2f750:::obj-fill-562:head# 2020-11-16T21:12:30.050 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.050 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d8f80ac2:::obj-fill-179:head# 2020-11-16T21:12:30.050 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.051 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:d9f904cf:::obj-fill-308:head# 2020-11-16T21:12:30.051 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.051 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:db0a534b:::obj-fill-410:head# 2020-11-16T21:12:30.051 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.051 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:dc4403f4:::obj-fill-416:head# 2020-11-16T21:12:30.051 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.051 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:dc66c3ca:::obj-fill-220:head# 2020-11-16T21:12:30.052 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.052 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:dc8198dc:::obj-fill-596:head# 2020-11-16T21:12:30.052 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.052 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:dcbade2b:::obj-fill-10:head# 2020-11-16T21:12:30.052 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.052 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ddb787ea:::obj-fill-556:head# 2020-11-16T21:12:30.053 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.053 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:de58065b:::obj-fill-306:head# 2020-11-16T21:12:30.053 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.053 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:deaa6fab:::obj-fill-194:head# 2020-11-16T21:12:30.053 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.053 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:dfee7714:::obj-fill-202:head# 2020-11-16T21:12:30.053 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.054 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:dff5df4a:::obj-fill-565:head# 2020-11-16T21:12:30.054 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.054 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e2053442:::obj-fill-107:head# 2020-11-16T21:12:30.054 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.054 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e32055b8:::obj-fill-57:head# 2020-11-16T21:12:30.054 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.054 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e37fac1e:::obj-fill-161:head# 2020-11-16T21:12:30.054 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.055 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e388f5aa:::obj-fill-273:head# 2020-11-16T21:12:30.055 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.055 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e38b1f4d:::obj-fill-393:head# 2020-11-16T21:12:30.055 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.055 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e4792474:::obj-fill-158:head# 2020-11-16T21:12:30.056 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.056 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e4899cb0:::obj-fill-270:head# 2020-11-16T21:12:30.056 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.056 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e62ae9f1:::obj-fill-288:head# 2020-11-16T21:12:30.056 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.056 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e64fd95a:::obj-fill-449:head# 2020-11-16T21:12:30.056 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.057 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e6682cfc:::obj-fill-373:head# 2020-11-16T21:12:30.057 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.057 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e716717b:::obj-fill-178:head# 2020-11-16T21:12:30.057 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.057 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e80226af:::obj-fill-354:head# 2020-11-16T21:12:30.057 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.058 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e83a5940:::obj-fill-564:head# 2020-11-16T21:12:30.058 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.058 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e854bfbe:::obj-fill-589:head# 2020-11-16T21:12:30.058 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.058 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e8b197bc:::obj-fill-89:head# 2020-11-16T21:12:30.058 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.058 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e92fca88:::obj-fill-397:head# 2020-11-16T21:12:30.059 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.059 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e94d8f7b:::obj-fill-261:head# 2020-11-16T21:12:30.059 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.059 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:e98cc0a4:::obj-fill-525:head# 2020-11-16T21:12:30.059 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.059 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ea5a9459:::obj-fill-431:head# 2020-11-16T21:12:30.059 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.059 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ebc54d47:::obj-fill-258:head# 2020-11-16T21:12:30.060 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.060 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ec0002dd:::obj-fill-205:head# 2020-11-16T21:12:30.060 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.060 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ec740b5c:::obj-fill-46:head# 2020-11-16T21:12:30.060 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.060 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ed35dda4:::obj-fill-170:head# 2020-11-16T21:12:30.061 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.061 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ed51b8f7:::obj-fill-91:head# 2020-11-16T21:12:30.061 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.061 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:edd96161:::obj-fill-48:head# 2020-11-16T21:12:30.061 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.061 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ee3f0d4b:::obj-fill-47:head# 2020-11-16T21:12:30.061 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.062 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ee4784dd:::obj-fill-480:head# 2020-11-16T21:12:30.062 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.062 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ee5a8271:::obj-fill-198:head# 2020-11-16T21:12:30.062 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.062 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ef745b66:::obj-fill-145:head# 2020-11-16T21:12:30.062 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.062 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:efa5309d:::obj-fill-215:head# 2020-11-16T21:12:30.063 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.063 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:efbdd3a1:::obj-fill-276:head# 2020-11-16T21:12:30.063 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.063 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f059d4c0:::obj-fill-482:head# 2020-11-16T21:12:30.063 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.063 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f0f9fb7a:::obj-fill-379:head# 2020-11-16T21:12:30.063 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.063 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f16a3656:::obj-fill-462:head# 2020-11-16T21:12:30.064 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.064 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f1b395ae:::obj-fill-25:head# 2020-11-16T21:12:30.064 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.064 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f5876dad:::obj-fill-359:head# 2020-11-16T21:12:30.064 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.064 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f5a283cd:::obj-fill-534:head# 2020-11-16T21:12:30.064 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.064 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f5cd905a:::obj-fill-358:head# 2020-11-16T21:12:30.065 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.065 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f5e48302:::obj-fill-406:head# 2020-11-16T21:12:30.065 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.065 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f5ea65fc:::obj-fill-52:head# 2020-11-16T21:12:30.065 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.065 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f642de6b:::obj-fill-204:head# 2020-11-16T21:12:30.065 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.066 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f716eafd:::obj-fill-502:head# 2020-11-16T21:12:30.066 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.066 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f752e9cb:::obj-fill-23:head# 2020-11-16T21:12:30.066 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.066 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f85fe42c:::obj-fill-346:head# 2020-11-16T21:12:30.066 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.067 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:f882fd3b:::obj-fill-355:head# 2020-11-16T21:12:30.067 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.067 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:fa133332:::obj-fill-309:head# 2020-11-16T21:12:30.067 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.067 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:fa875e52:::obj-fill-64:head# 2020-11-16T21:12:30.067 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.067 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:fad91a22:::obj-fill-481:head# 2020-11-16T21:12:30.067 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.068 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:fbbc3122:::obj-fill-111:head# 2020-11-16T21:12:30.068 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.068 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:fc539ddb:::obj-fill-181:head# 2020-11-16T21:12:30.068 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.068 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:fc54128d:::obj-fill-104:head# 2020-11-16T21:12:30.068 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.068 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:fe00aded:::obj-fill-365:head# 2020-11-16T21:12:30.069 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.069 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:fe1e9971:::obj-fill-189:head# 2020-11-16T21:12:30.069 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.069 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ff0e0563:::obj-fill-508:head# 2020-11-16T21:12:30.069 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.069 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ff4e156c:::obj-fill-441:head# 2020-11-16T21:12:30.070 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.070 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:ffa355f7:::obj-fill-500:head# 2020-11-16T21:12:30.070 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.070 INFO:tasks.workunit.client.0.smithi189.stdout:Write #1:fff6c93b:::obj-fill-277:head# 2020-11-16T21:12:30.070 INFO:tasks.workunit.client.0.smithi189.stdout:snapset 0=[]:{} 2020-11-16T21:12:30.070 INFO:tasks.workunit.client.0.smithi189.stdout:write_pg epoch 36 info 1.0( v 36'600 (36'500,36'600] local-lis/les=23/24 n=600 ec=21/21 lis/c=23/23 les/c/f=24/24/0 sis=23) 2020-11-16T21:12:30.070 INFO:tasks.workunit.client.0.smithi189.stdout:Import successful 2020-11-16T21:12:30.071 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:12:30.071 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:45:00.178836+0000 0'0 45:33 [2,0] 2 [2,0] 2 0'0 2020-11-16T20:41:56.209471+0000 0'0 2020-11-16T20:41:56.209471+0000 0 2020-11-16T21:12:30.071 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 566 0 0 0 0 1477826 0 0 66 66 stale+active+clean 2020-11-16T20:41:53.479578+0000 36'566 36:589 [1] 1 [1] 1 0'0 2020-11-16T20:41:51.380521+0000 0'0 2020-11-16T20:41:51.380521+0000 0 2020-11-16T21:12:30.071 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 600 0 0 0 0 614400 0 0 100 100 active+clean 2020-11-16T20:45:00.008078+0000 40'600 45:645 [2,0] 2 [2,0] 2 0'0 2020-11-16T20:41:53.771402+0000 0'0 2020-11-16T20:41:53.771402+0000 0 2020-11-16T21:12:30.071 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:12:30.071 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:12:30.071 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:12:30.072 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T20:45:00.178836+0000 0'0 45:33 [2,0] 2 [2,0] 2 0'0 2020-11-16T20:41:56.209471+0000 0'0 2020-11-16T20:41:56.209471+0000 0 2020-11-16T21:12:30.072 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 566 0 0 0 0 1477826 0 0 66 66 stale+active+clean 2020-11-16T20:41:53.479578+0000 36'566 36:589 [1] 1 [1] 1 0'0 2020-11-16T20:41:51.380521+0000 0'0 2020-11-16T20:41:51.380521+0000 0 2020-11-16T21:12:30.072 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 600 0 0 0 0 614400 0 0 100 100 active+clean 2020-11-16T20:45:00.008078+0000 40'600 45:645 [2,0] 2 [2,0] 2 0'0 2020-11-16T20:41:53.771402+0000 0'0 2020-11-16T20:41:53.771402+0000 0 2020-11-16T21:12:30.072 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:12:30.072 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:12:30.072 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:12:30.072 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.072 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.073 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up out weight 0 up_from 49 up_thru 34 down_at 45 last_clean_interval [40,44) [v2:127.0.0.1:6810/225542,v1:127.0.0.1:6811/225542] [v2:127.0.0.1:6812/225542,v1:127.0.0.1:6813/225542] exists,up 929f76af-e446-4277-aece-f953facd3c63 2020-11-16T21:12:30.073 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836579 2020-11-16T21:12:30.073 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 210453397513 2020-11-16T21:12:30.073 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836594 2020-11-16T21:12:30.073 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 210453397528 2020-11-16T21:12:30.073 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836597 2020-11-16T21:12:30.073 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 210453397531 2020-11-16T21:12:30.074 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:12:30.074 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 600 0 600 0 0 1566600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2020-11-16T20:50:26.341827+0000 45'600 56:668 [1,0] 1 [0] 0 0'0 2020-11-16T20:41:56.209471+0000 0'0 2020-11-16T20:41:56.209471+0000 0 2020-11-16T21:12:30.074 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 600 0 0 0 0 1566600 0 0 100 100 active+clean 2020-11-16T20:48:55.521630+0000 36'600 55:645 [1] 1 [1] 1 0'0 2020-11-16T20:41:51.380521+0000 0'0 2020-11-16T20:41:51.380521+0000 0 2020-11-16T21:12:30.074 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 600 0 0 0 0 1566600 0 0 10 10 active+clean 2020-11-16T20:50:26.433857+0000 45'1200 56:2476 [1,0] 1 [1,0] 1 0'0 2020-11-16T20:41:53.771402+0000 0'0 2020-11-16T20:41:53.771402+0000 0 2020-11-16T21:12:30.074 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:12:30.074 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:12:30.075 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 7ee53c15-f82b-4104-add3-8389cc45af48 2020-11-16T21:12:30.075 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.075 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:12:30.075 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:12:30.075 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.075 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.075 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/227967,v1:127.0.0.1:6803/227967] [v2:127.0.0.1:6804/227967,v1:127.0.0.1:6805/227967] exists,up 7ee53c15-f82b-4104-add3-8389cc45af48 2020-11-16T21:12:30.076 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 ba099e1a-b7ef-4d93-ad31-b8bbd7f0d1ad 2020-11-16T21:12:30.076 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.076 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:12:30.076 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:12:30.076 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.076 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.076 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/228961,v1:127.0.0.1:6811/228961] [v2:127.0.0.1:6812/228961,v1:127.0.0.1:6813/228961] exists,up ba099e1a-b7ef-4d93-ad31-b8bbd7f0d1ad 2020-11-16T21:12:30.077 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 377dd9af-483f-486e-9c3c-126cfe3237b6 2020-11-16T21:12:30.077 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:12:30.077 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:12:30.077 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:12:30.077 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.077 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.077 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/229955,v1:127.0.0.1:6819/229955] [v2:127.0.0.1:6820/229955,v1:127.0.0.1:6821/229955] exists,up 377dd9af-483f-486e-9c3c-126cfe3237b6 2020-11-16T21:12:30.078 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 05ac7116-ae31-4b04-b82c-40e5dfeb90af 2020-11-16T21:12:30.078 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:12:30.078 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:12:30.078 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:12:30.078 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.078 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.079 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/230949,v1:127.0.0.1:6827/230949] [v2:127.0.0.1:6828/230949,v1:127.0.0.1:6829/230949] exists,up 05ac7116-ae31-4b04-b82c-40e5dfeb90af 2020-11-16T21:12:30.079 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 a0bf3665-d2ef-4edc-8b39-65d7deaf24ea 2020-11-16T21:12:30.079 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:12:30.079 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:12:30.079 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:12:30.079 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.079 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.079 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/231943,v1:127.0.0.1:6835/231943] [v2:127.0.0.1:6836/231943,v1:127.0.0.1:6837/231943] exists,up a0bf3665-d2ef-4edc-8b39-65d7deaf24ea 2020-11-16T21:12:30.080 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 d4e12b60-777c-4297-91cf-53e605d2d67f 2020-11-16T21:12:30.080 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T21:12:30.080 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T21:12:30.080 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T21:12:30.080 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.080 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.080 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/232937,v1:127.0.0.1:6843/232937] [v2:127.0.0.1:6844/232937,v1:127.0.0.1:6845/232937] exists,up d4e12b60-777c-4297-91cf-53e605d2d67f 2020-11-16T21:12:30.081 INFO:tasks.workunit.client.0.smithi189.stdout:add osd6 5152a9e7-7900-4d0d-818a-0323f6092dea 2020-11-16T21:12:30.081 INFO:tasks.workunit.client.0.smithi189.stdout:6 2020-11-16T21:12:30.081 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd6 key to auth repository 2020-11-16T21:12:30.081 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.6 2020-11-16T21:12:30.081 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.081 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.081 INFO:tasks.workunit.client.0.smithi189.stdout:osd.6 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/233931,v1:127.0.0.1:6851/233931] [v2:127.0.0.1:6852/233931,v1:127.0.0.1:6853/233931] exists,up 5152a9e7-7900-4d0d-818a-0323f6092dea 2020-11-16T21:12:30.082 INFO:tasks.workunit.client.0.smithi189.stdout:add osd7 7051b692-ea86-4f5b-abaa-d8fb1fbe02be 2020-11-16T21:12:30.082 INFO:tasks.workunit.client.0.smithi189.stdout:7 2020-11-16T21:12:30.082 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd7 key to auth repository 2020-11-16T21:12:30.082 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.7 2020-11-16T21:12:30.082 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.082 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.082 INFO:tasks.workunit.client.0.smithi189.stdout:osd.7 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/234926,v1:127.0.0.1:6859/234926] [v2:127.0.0.1:6860/234926,v1:127.0.0.1:6861/234926] exists,up 7051b692-ea86-4f5b-abaa-d8fb1fbe02be 2020-11-16T21:12:30.082 INFO:tasks.workunit.client.0.smithi189.stdout:add osd8 e22a2b02-f4e5-42a0-8724-23fd0b5b7904 2020-11-16T21:12:30.083 INFO:tasks.workunit.client.0.smithi189.stdout:8 2020-11-16T21:12:30.083 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd8 key to auth repository 2020-11-16T21:12:30.083 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.8 2020-11-16T21:12:30.083 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.083 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.083 INFO:tasks.workunit.client.0.smithi189.stdout:osd.8 up in weight 1 up_from 43 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/235920,v1:127.0.0.1:6867/235920] [v2:127.0.0.1:6868/235920,v1:127.0.0.1:6869/235920] exists,up e22a2b02-f4e5-42a0-8724-23fd0b5b7904 2020-11-16T21:12:30.083 INFO:tasks.workunit.client.0.smithi189.stdout:add osd9 74a825de-57e6-4fee-9e1a-ea3010a9016b 2020-11-16T21:12:30.084 INFO:tasks.workunit.client.0.smithi189.stdout:9 2020-11-16T21:12:30.084 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd9 key to auth repository 2020-11-16T21:12:30.084 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.9 2020-11-16T21:12:30.084 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:30.084 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:30.084 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:12:30.084 INFO:tasks.workunit.client.0.smithi189.stdout:osd.9 up in weight 1 up_from 48 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/236919,v1:127.0.0.1:6875/236919] [v2:127.0.0.1:6876/236919,v1:127.0.0.1:6877/236919] exists,up 74a825de-57e6-4fee-9e1a-ea3010a9016b 2020-11-16T21:12:30.085 INFO:tasks.workunit.client.0.smithi189.stdout:add osd10 e368b490-ada9-4609-a55a-1d53fd53353e 2020-11-16T21:12:30.085 INFO:tasks.workunit.client.0.smithi189.stdout:10 2020-11-16T21:12:30.085 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd10 key to auth repository 2020-11-16T21:12:34.266 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.10 2020-11-16T21:12:34.266 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:34.267 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:34.267 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:12:34.267 INFO:tasks.workunit.client.0.smithi189.stdout:osd.10 up in weight 1 up_from 53 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/237939,v1:127.0.0.1:6883/237939] [v2:127.0.0.1:6884/237939,v1:127.0.0.1:6885/237939] exists,up e368b490-ada9-4609-a55a-1d53fd53353e 2020-11-16T21:12:34.267 INFO:tasks.workunit.client.0.smithi189.stdout:add osd11 005d10e5-1349-480c-a878-cef198c556d6 2020-11-16T21:12:34.267 INFO:tasks.workunit.client.0.smithi189.stdout:11 2020-11-16T21:12:34.267 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd11 key to auth repository 2020-11-16T21:12:34.268 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.11 2020-11-16T21:12:34.268 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:34.268 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:34.268 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:12:34.268 INFO:tasks.workunit.client.0.smithi189.stdout:osd.11 up in weight 1 up_from 58 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6890/238959,v1:127.0.0.1:6891/238959] [v2:127.0.0.1:6892/238959,v1:127.0.0.1:6893/238959] exists,up 005d10e5-1349-480c-a878-cef198c556d6 2020-11-16T21:12:34.268 INFO:tasks.workunit.client.0.smithi189.stdout:add osd12 9aa9ef44-2ca1-452e-8fa3-f3d180643e79 2020-11-16T21:12:34.268 INFO:tasks.workunit.client.0.smithi189.stdout:12 2020-11-16T21:12:34.268 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd12 key to auth repository 2020-11-16T21:12:34.269 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.12 2020-11-16T21:12:34.269 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:34.269 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:34.269 INFO:tasks.workunit.client.0.smithi189.stdout:osd.12 up in weight 1 up_from 63 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6898/239979,v1:127.0.0.1:6899/239979] [v2:127.0.0.1:6900/239979,v1:127.0.0.1:6901/239979] exists,up 9aa9ef44-2ca1-452e-8fa3-f3d180643e79 2020-11-16T21:12:34.269 INFO:tasks.workunit.client.0.smithi189.stdout:add osd13 d1e0d617-c34d-435f-bddb-7808c3ca712e 2020-11-16T21:12:34.269 INFO:tasks.workunit.client.0.smithi189.stdout:13 2020-11-16T21:12:34.269 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd13 key to auth repository 2020-11-16T21:12:34.270 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.13 2020-11-16T21:12:34.270 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:34.270 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:34.270 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:12:34.270 INFO:tasks.workunit.client.0.smithi189.stdout:osd.13 up in weight 1 up_from 68 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6906/240973,v1:127.0.0.1:6907/240973] [v2:127.0.0.1:6908/240973,v1:127.0.0.1:6909/240973] exists,up d1e0d617-c34d-435f-bddb-7808c3ca712e 2020-11-16T21:12:34.270 INFO:tasks.workunit.client.0.smithi189.stdout:add osd14 301430e9-8f0f-4858-b138-5dc5dfec9c70 2020-11-16T21:12:34.270 INFO:tasks.workunit.client.0.smithi189.stdout:14 2020-11-16T21:12:34.271 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd14 key to auth repository 2020-11-16T21:12:34.271 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.14 2020-11-16T21:12:34.271 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:34.271 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:34.271 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:12:34.271 INFO:tasks.workunit.client.0.smithi189.stdout:osd.14 up in weight 1 up_from 73 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6914/241993,v1:127.0.0.1:6915/241993] [v2:127.0.0.1:6916/241993,v1:127.0.0.1:6917/241993] exists,up 301430e9-8f0f-4858-b138-5dc5dfec9c70 2020-11-16T21:12:34.271 INFO:tasks.workunit.client.0.smithi189.stdout:add osd15 9b59bd85-b530-48f0-9ac0-38da75971adc 2020-11-16T21:12:34.272 INFO:tasks.workunit.client.0.smithi189.stdout:15 2020-11-16T21:12:34.272 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd15 key to auth repository 2020-11-16T21:12:34.272 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.15 2020-11-16T21:12:34.272 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:34.272 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:34.272 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:12:34.273 INFO:tasks.workunit.client.0.smithi189.stdout:osd.15 up in weight 1 up_from 78 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6922/243013,v1:127.0.0.1:6923/243013] [v2:127.0.0.1:6924/243013,v1:127.0.0.1:6925/243013] exists,up 9b59bd85-b530-48f0-9ac0-38da75971adc 2020-11-16T21:12:34.273 INFO:tasks.workunit.client.0.smithi189.stdout:add osd16 808f55de-c834-4dda-ba03-66260ea3da9b 2020-11-16T21:12:34.273 INFO:tasks.workunit.client.0.smithi189.stdout:16 2020-11-16T21:12:34.273 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd16 key to auth repository 2020-11-16T21:12:34.273 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.16 2020-11-16T21:12:34.273 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:34.273 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:34.273 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:12:34.274 INFO:tasks.workunit.client.0.smithi189.stdout:osd.16 up in weight 1 up_from 83 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6930/244033,v1:127.0.0.1:6931/244033] [v2:127.0.0.1:6932/244033,v1:127.0.0.1:6933/244033] exists,up 808f55de-c834-4dda-ba03-66260ea3da9b 2020-11-16T21:12:34.274 INFO:tasks.workunit.client.0.smithi189.stdout:add osd17 11997534-baef-4fbe-b8b3-3d02488918ee 2020-11-16T21:12:34.274 INFO:tasks.workunit.client.0.smithi189.stdout:17 2020-11-16T21:12:34.274 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd17 key to auth repository 2020-11-16T21:12:34.274 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.17 2020-11-16T21:12:34.274 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:34.274 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:34.275 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:12:34.275 INFO:tasks.workunit.client.0.smithi189.stdout:osd.17 up in weight 1 up_from 88 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6938/245053,v1:127.0.0.1:6939/245053] [v2:127.0.0.1:6940/245053,v1:127.0.0.1:6941/245053] exists,up 11997534-baef-4fbe-b8b3-3d02488918ee 2020-11-16T21:12:34.275 INFO:tasks.workunit.client.0.smithi189.stdout:add osd18 6c5aed7d-0a46-4541-a1a4-345da649cb11 2020-11-16T21:12:34.275 INFO:tasks.workunit.client.0.smithi189.stdout:18 2020-11-16T21:12:34.275 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd18 key to auth repository 2020-11-16T21:12:34.275 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.18 2020-11-16T21:12:34.275 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:34.276 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:34.276 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:12:34.276 INFO:tasks.workunit.client.0.smithi189.stdout:osd.18 up in weight 1 up_from 93 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6946/246074,v1:127.0.0.1:6947/246074] [v2:127.0.0.1:6948/246074,v1:127.0.0.1:6949/246074] exists,up 6c5aed7d-0a46-4541-a1a4-345da649cb11 2020-11-16T21:12:34.276 INFO:tasks.workunit.client.0.smithi189.stdout:add osd19 456df5ff-f0a4-4a3d-add6-4a1ac301bc08 2020-11-16T21:12:34.276 INFO:tasks.workunit.client.0.smithi189.stdout:19 2020-11-16T21:12:34.276 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd19 key to auth repository 2020-11-16T21:12:34.276 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.19 2020-11-16T21:12:34.277 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:12:34.277 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:12:34.277 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:12:34.277 INFO:tasks.workunit.client.0.smithi189.stdout:osd.19 up in weight 1 up_from 98 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6954/247098,v1:127.0.0.1:6955/247098] [v2:127.0.0.1:6956/247098,v1:127.0.0.1:6957/247098] exists,up 456df5ff-f0a4-4a3d-add6-4a1ac301bc08 2020-11-16T21:12:34.277 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836521 2020-11-16T21:12:34.277 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705705 2020-11-16T21:12:34.277 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542182 2020-11-16T21:12:34.277 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378662 2020-11-16T21:12:34.278 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247843 2020-11-16T21:12:34.278 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 120259084322 2020-11-16T21:12:34.278 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.6 seq 141733920800 2020-11-16T21:12:34.278 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.7 seq 163208757279 2020-11-16T21:12:34.278 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.8 seq 184683593757 2020-11-16T21:12:34.278 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.9 seq 206158430235 2020-11-16T21:12:34.278 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.10 seq 227633266713 2020-11-16T21:12:34.279 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.11 seq 249108103193 2020-11-16T21:12:34.279 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.12 seq 270582939671 2020-11-16T21:12:34.279 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.13 seq 292057776148 2020-11-16T21:12:34.279 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.14 seq 313532612627 2020-11-16T21:12:34.279 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.15 seq 335007449104 2020-11-16T21:12:34.279 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.16 seq 356482285582 2020-11-16T21:12:34.279 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.17 seq 377957122060 2020-11-16T21:12:34.279 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.18 seq 399431958538 2020-11-16T21:12:34.280 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.19 seq 420906795016 2020-11-16T21:12:34.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:12:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:12:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:12:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:12:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:12:34.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:12:34.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:12:34.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:12:34.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.317 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:12:34.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:12:34.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:12:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:12:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:12:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:12:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:12:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:12:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:12:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:12:34.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:12:34.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:12:34.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:12:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:12:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:12:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:12:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:12:34.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:12:34.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:12:34.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:12:34.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:12:34.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:12:34.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:12:34.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:12:34.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:12:34.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:12:34.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:12:34.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:12:34.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:12:34.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:12:34.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:12:34.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:12:34.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:12:34.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:12:34.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:12:34.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:12:34.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:12:34.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:12:34.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:12:34.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:12:34.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:12:34.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:12:34.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:12:34.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:12:34.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:12:34.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:12:34.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:12:34.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:12:34.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:12:34.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:12:34.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:12:34.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:12:34.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:12:34.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:12:34.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:12:34.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:12:34.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:12:34.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:12:34.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:12:34.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:12:34.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:12:34.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:12:34.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:12:34.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:12:34.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:12:34.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:12:34.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:12:34.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:12:34.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:12:34.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:12:34.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:12:34.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:12:34.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:12:34.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:12:34.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:12:34.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:12:34.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:12:34.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:12:34.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:12:34.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:12:34.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:12:34.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:12:34.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:12:34.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:12:34.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:12:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:12:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:12:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:12:34.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:12:34.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:12:34.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:12:34.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:12:34.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:12:34.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:12:34.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:12:34.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:12:34.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:12:34.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:12:34.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:12:34.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:12:34.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:12:34.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:12:34.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:12:34.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:12:34.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:12:34.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:12:34.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:12:34.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:12:34.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:12:34.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:12:34.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:12:34.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:12:34.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:12:34.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:12:34.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:12:34.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:12:34.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:12:34.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:12:34.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:12:34.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:12:34.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:12:34.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:12:34.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:12:34.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:12:34.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:12:34.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:12:34.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:12:34.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:12:34.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:12:34.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:12:34.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:12:34.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:12:34.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:12:34.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:12:34.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:12:34.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:12:34.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:12:34.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:12:34.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:12:34.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:12:34.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:12:34.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:12:34.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:12:34.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:12:34.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:12:34.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:12:34.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:12:34.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:12:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:12:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:12:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:12:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:12:34.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:12:34.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:12:34.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:12:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:12:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:12:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:12:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:12:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:12:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:12:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:12:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:12:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:12:34.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:12:34.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:12:34.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:12:34.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:12:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:12:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:12:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:12:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:12:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:12:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:12:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:12:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:12:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:12:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:12:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:12:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:12:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:12:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:12:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:12:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:12:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:12:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:12:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:12:34.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:12:34.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:12:34.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:12:34.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:12:34.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:12:34.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:12:34.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:12:34.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:12:34.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:12:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:12:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:12:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:12:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:12:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:12:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:12:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:12:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:12:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:12:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:12:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:12:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:12:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:12:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:12:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:12:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:12:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:12:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:12:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:12:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:12:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:12:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:12:34.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:12:34.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:12:34.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:12:34.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:12:34.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:12:34.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:12:34.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:12:34.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:12:34.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:12:34.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:12:34.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:12:34.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:12:34.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:12:34.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:12:34.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:12:34.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:12:34.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:12:34.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:12:34.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:12:34.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:12:34.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:12:34.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:12:34.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:12:34.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:12:34.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:12:34.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:12:34.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:12:34.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:12:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:12:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:12:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:12:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:12:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:12:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:12:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:12:34.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:12:34.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:12:34.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:12:34.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:12:34.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:12:34.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:12:34.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:12:34.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:12:34.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:12:34.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:12:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:12:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:12:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:12:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:12:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:12:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:12:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:12:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:12:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:12:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:12:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:12:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:12:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:12:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:12:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:12:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:12:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:12:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:12:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:12:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:12:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:12:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:12:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:12:34.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:12:34.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:12:34.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:12:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:12:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:12:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:12:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:12:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:12:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:12:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:12:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:12:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:12:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:12:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:12:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:12:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:12:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:12:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:12:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:12:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:12:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:12:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:12:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:12:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:12:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:12:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:12:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:12:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:12:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:12:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:12:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:12:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:12:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:12:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:12:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:12:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:12:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:12:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:12:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:12:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:12:34.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:12:34.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:12:34.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:12:34.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:12:34.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:12:34.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:12:34.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:12:34.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:12:34.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:12:34.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:12:34.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:12:34.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:12:34.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:12:34.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:12:34.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:12:34.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:12:34.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:12:34.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:12:34.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:12:34.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:12:34.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:12:34.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:12:34.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:12:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:12:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:12:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:12:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:12:34.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:12:34.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:12:34.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:12:34.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:12:34.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:12:34.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:12:34.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:12:34.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:12:34.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:12:34.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:12:34.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:12:34.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:12:34.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:12:34.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:12:34.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:12:34.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:12:34.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:12:34.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:12:34.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:12:34.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:12:34.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:12:34.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:12:34.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:12:34.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:12:34.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:12:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:12:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:12:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:12:34.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:12:34.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:12:34.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:12:34.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:12:34.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:12:34.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:12:34.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:12:34.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:12:34.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:12:34.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:12:34.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:12:34.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:12:34.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:12:34.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:12:34.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:12:34.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:12:34.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:12:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:12:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:12:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:12:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:12:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:12:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:12:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:12:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:12:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:12:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:12:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:12:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:12:34.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:12:34.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:12:34.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:12:34.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:12:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:12:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:12:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:12:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:12:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:12:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:12:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:12:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:12:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:12:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:12:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:12:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:12:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:12:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:12:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:12:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:12:34.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:12:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:12:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:12:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:12:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:12:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:12:34.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:12:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:12:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:12:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:12:34.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:12:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:12:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:12:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:12:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:12:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:12:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:12:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:12:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:12:34.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:12:34.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:12:34.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:12:34.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:12:34.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:12:34.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:12:34.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:12:34.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:12:34.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:12:34.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:12:34.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:12:34.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:12:34.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:12:34.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:12:34.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:12:34.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:34.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:34.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:12:34.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:12:34.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:12:34.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:12:34.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:12:34.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:12:34.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:12:34.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:34.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:12:35.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:35.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:35.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:12:39.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:12:39.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:12:39.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:12:39.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:12:39.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:12:39.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:12:39.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:12:39.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:39.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:39.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:12:39.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:12:39.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:12:39.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:12:39.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:12:39.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:12:39.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:12:39.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:12:39.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:39.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:39.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:12:39.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:12:39.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:12:39.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:12:39.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:12:39.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:12:39.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:12:39.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:12:39.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:39.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:39.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:12:39.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:12:39.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:12:39.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:12:39.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:12:39.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:12:39.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:12:39.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:12:39.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:39.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:39.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:12:39.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:12:39.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:12:39.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:12:39.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:12:39.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:12:39.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:12:39.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:12:39.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:39.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:39.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:12:39.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:12:39.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:12:39.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:12:39.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:12:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:12:39.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:12:39.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:12:39.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:39.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:39.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:12:39.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:12:39.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:39.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:12:44.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:12:44.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:12:44.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:12:44.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:12:44.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:12:44.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:44.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:44.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:12:44.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:12:44.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:12:44.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:12:44.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:12:44.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:12:44.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:12:44.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:12:44.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:44.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:44.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:12:44.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:12:44.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:12:44.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:12:44.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:12:44.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:12:44.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:12:44.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:12:44.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:44.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:44.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:12:44.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:12:44.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:12:44.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:12:44.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:12:44.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:12:44.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:12:44.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:12:44.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:44.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:44.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:12:44.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:12:44.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:12:44.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:12:44.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:12:44.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:12:44.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:12:44.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:12:44.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:44.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:44.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:12:44.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:12:44.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:12:44.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:12:44.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:12:44.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:12:44.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:12:44.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:12:44.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:44.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:44.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:12:44.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:12:44.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:12:44.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:12:44.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:12:44.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:44.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:12:45.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:45.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:12:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:12:45.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:45.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:45.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:12:45.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:45.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:12:45.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:45.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:12:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:12:45.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:45.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:12:45.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:45.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:12:45.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:45.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:12:45.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:45.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:12:46.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:46.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:46.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:46.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:12:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:12:46.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:46.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:12:46.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:46.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:12:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:12:46.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:46.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:12:46.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:46.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:12:47.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:47.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:12:47.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:47.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:47.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:47.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:12:47.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:47.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:12:47.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:47.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:12:47.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:47.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:12:47.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:47.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:12:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:12:47.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:47.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:12:47.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:47.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:12:48.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:48.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:48.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:48.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:12:48.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:12:48.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:48.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:12:48.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:48.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:12:48.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:48.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:12:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:12:48.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:48.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:12:48.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:48.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:12:48.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:48.789 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:48.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:48.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:12:48.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:48.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:12:48.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:48.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:12:49.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:49.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:12:49.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:49.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:12:49.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:49.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:12:49.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:49.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:12:49.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:49.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:12:49.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:49.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:49.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:49.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:12:49.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:49.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:12:49.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:49.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:12:49.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:49.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:12:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:12:50.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:50.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:12:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:12:50.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:50.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:12:54.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:54.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:54.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:12:54.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:12:54.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:12:54.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:12:54.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:12:54.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:12:54.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:12:54.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:12:54.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:54.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:54.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:12:54.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:12:54.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:12:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:12:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:12:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:12:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:12:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:12:54.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:54.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:12:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:12:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:12:54.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:12:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:12:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:12:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:12:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:12:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:54.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:12:54.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:12:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:12:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:12:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:12:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:12:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:12:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:12:54.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:54.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:12:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:12:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:12:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:12:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:12:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:12:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:12:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:12:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:54.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:12:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:12:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:12:54.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:12:54.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:12:54.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:12:54.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:12:54.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:12:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:54.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:54.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:54.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:12:55.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:55.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:12:55.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:55.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:12:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:55.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:12:55.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:55.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:12:55.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:55.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:12:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:12:55.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:55.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:12:55.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:55.782 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:55.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:55.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:12:55.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:55.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:12:55.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:55.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:12:56.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:56.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:12:56.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:56.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:12:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:12:56.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:56.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:12:56.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:56.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:12:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:56.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:12:56.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:56.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:12:56.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:56.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:12:56.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:56.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:12:57.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:57.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:12:57.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:57.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:12:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:12:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:12:57.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:57.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:57.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:12:57.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:57.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:12:57.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:57.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:12:57.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:57.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:12:57.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:57.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:12:57.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:57.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:12:57.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:57.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:12:58.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:58.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:12:58.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:58.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:58.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:58.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:12:58.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:58.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:12:58.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:58.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:12:58.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:58.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:12:58.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:58.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:12:58.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:58.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:12:58.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:58.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:12:58.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:58.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:12:58.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:58.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:58.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:58.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:12:59.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:59.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:12:59.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:59.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:12:59.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:59.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:12:59.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:59.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:12:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:59.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:12:59.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:59.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:12:59.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:59.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:12:59.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:12:59.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:12:59.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:59.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:12:59.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:59.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:12:59.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:12:59.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:13:00.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:00.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:13:04.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:13:04.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:13:04.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:13:04.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:13:04.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:04.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:04.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:13:04.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:13:04.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:13:04.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:13:04.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:13:04.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:13:04.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:13:04.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:13:04.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:04.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:04.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:13:04.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:13:04.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:13:04.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:13:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:13:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:13:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:13:04.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:13:04.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:04.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:04.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:13:04.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:13:04.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:13:04.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:13:04.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:13:04.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:13:04.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:13:04.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:13:04.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:04.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:04.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:13:04.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:13:04.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:13:04.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:13:04.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:13:04.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:13:04.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:13:04.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:13:04.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:04.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:04.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:13:04.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:13:04.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:13:04.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:13:04.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:13:04.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:13:04.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:13:04.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:13:04.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:04.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:04.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:13:04.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:13:04.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:13:04.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:13:04.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:13:04.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:04.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:13:05.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:13:09.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:13:09.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:09.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:09.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:13:09.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:13:09.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:13:09.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:13:09.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:13:09.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:13:09.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:13:09.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:13:09.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:09.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:09.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:13:09.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:13:09.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:13:09.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:13:09.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:13:09.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:13:09.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:13:09.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:13:09.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:09.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:09.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:13:09.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:13:09.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:13:09.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:13:09.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:13:09.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:13:09.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:13:09.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:13:09.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:09.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:09.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:13:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:13:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:13:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:13:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:13:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:13:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:13:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:13:09.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:09.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:09.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:13:09.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:13:09.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:13:09.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:13:09.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:13:09.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:13:09.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:13:09.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:13:09.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:09.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:09.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:13:09.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:13:09.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:13:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:13:09.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:13:09.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:13:09.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:13:09.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:13:09.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:09.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:09.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:09.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:13:10.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:10.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:13:10.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:10.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:13:10.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:10.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:13:10.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:10.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:13:10.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:10.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:13:10.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:10.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:13:10.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:10.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:13:10.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:10.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:10.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:10.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:13:10.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:10.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:13:10.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:10.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:13:11.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:11.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:13:11.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:11.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:13:11.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:11.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:13:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:11.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:13:11.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:11.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:13:11.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:11.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:11.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:11.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:13:11.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:11.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:13:11.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:11.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:13:11.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:11.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:13:12.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:12.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:13:12.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:12.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:13:12.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:12.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:13:12.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:12.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:13:12.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:12.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:12.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:12.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:13:12.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:12.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:13:12.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:12.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:13:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:13:12.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:12.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:13:12.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:12.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:13:12.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:12.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:13:13.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:13.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:13:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:13.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:13:13.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:13.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:13:13.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:13.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:13:13.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:13.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:13:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:13:13.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:13.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:13:13.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:13.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:13:13.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:13.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:13:14.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:14.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:14.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:14.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:13:14.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:14.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:13:14.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:14.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:13:14.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:14.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:13:14.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:14.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:13:14.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:14.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:13:14.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:14.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:13:14.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:14.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:13:14.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:14.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:14.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:14.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:13:14.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:14.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:13:14.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:14.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:13:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:15.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:13:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:13:15.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:15.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:13:15.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:15.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:13:15.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:15.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:13:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:15.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:13:15.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:15.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:13:15.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:15.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:13:15.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:15.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:13:15.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:15.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:13:16.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:16.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:13:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:13:16.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:16.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:13:16.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:16.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:16.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:16.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:13:16.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:16.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:13:16.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:16.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:13:16.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:16.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:13:16.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:16.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:13:16.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:16.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:13:17.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:17.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:13:17.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:17.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:13:17.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:17.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:17.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:17.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:13:17.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:17.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:13:17.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:17.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:13:17.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:17.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:13:17.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:17.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:13:17.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:17.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:13:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:13:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:13:18.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:18.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:18.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:13:18.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:18.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:13:18.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:18.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:13:18.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:18.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:13:18.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:18.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:13:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:13:18.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:18.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:13:18.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:18.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:13:18.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:18.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:18.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:18.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:13:19.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:19.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:13:19.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:19.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:13:19.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:19.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:13:19.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:19.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:13:19.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:19.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:13:19.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:19.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:13:19.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:19.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:13:19.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:19.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:19.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:19.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:13:19.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:19.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:13:19.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:19.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:13:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:13:20.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:20.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:13:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:13:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:13:20.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:20.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:13:20.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:20.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:20.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:20.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:13:20.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:20.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:13:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:13:20.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:20.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:13:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:13:21.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:21.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:13:21.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:21.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:13:21.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:21.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:13:21.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:21.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:21.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:21.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:13:21.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:21.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:13:21.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:21.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:13:21.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:21.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:13:21.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:21.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:13:21.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:21.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:13:21.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:21.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:13:22.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:22.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:13:22.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:22.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:22.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:22.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:13:22.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:22.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:13:22.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:22.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:13:22.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:22.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:13:22.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:22.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:13:22.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:22.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:13:22.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:22.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:13:22.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:22.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:13:22.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:22.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:22.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:22.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:13:22.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:22.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:13:23.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:23.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:13:23.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:23.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:13:23.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:23.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:13:23.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:23.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:13:23.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:23.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:13:23.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:23.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:13:23.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:23.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:23.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:23.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:13:23.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:23.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:13:23.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:23.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:13:23.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:23.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:13:24.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:24.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:13:24.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:24.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:13:24.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:24.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:13:24.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:24.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:13:24.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:24.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:24.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:24.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:13:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:13:24.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:24.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:13:24.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:24.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:13:24.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:24.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:13:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:13:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:13:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:13:25.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:25.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:13:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:13:25.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:25.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:13:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:25.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:13:25.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:25.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:13:25.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:25.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:13:25.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:25.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:13:25.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:25.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:13:26.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:26.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:26.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:26.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:13:26.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:26.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:13:26.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:26.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:13:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:13:26.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:26.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:13:26.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:26.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:13:26.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:26.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:13:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:13:26.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:26.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:26.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:26.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:13:26.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:26.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:13:26.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:26.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:13:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:13:27.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:27.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:13:27.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:27.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:13:27.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:27.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:13:27.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:27.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:13:27.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:27.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:27.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:27.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:13:27.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:27.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:13:27.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:27.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:13:27.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:27.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:13:28.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:28.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:13:28.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:28.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:13:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:13:28.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:28.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:13:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:28.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:13:28.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:28.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:13:28.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:28.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:13:28.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:28.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:13:28.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:28.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:13:28.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:28.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:13:28.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:28.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:13:29.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:29.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:13:29.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:29.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:29.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:29.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:13:29.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:29.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:13:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:13:29.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:13:29.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:29.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:13:29.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:29.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:13:29.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:29.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:13:29.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:29.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:13:29.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:29.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:29.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:29.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:13:30.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:30.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:13:30.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:30.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:13:30.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:30.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:13:30.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:30.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:13:30.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:30.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:13:30.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:13:30.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:30.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:13:30.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:30.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:30.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:30.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:13:30.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:30.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:13:30.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:30.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:13:31.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:31.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:13:31.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:31.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:13:31.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:31.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:13:31.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:31.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:13:31.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:31.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:13:31.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:31.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:31.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:31.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:13:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:13:31.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:31.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:13:31.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:31.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:13:32.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:32.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:13:32.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:32.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:13:32.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:32.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:13:32.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:32.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:13:32.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:32.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:32.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:32.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:13:32.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:32.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:13:36.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:13:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:13:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:13:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:13:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:13:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:13:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:36.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:13:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:13:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:13:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:13:36.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:13:36.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:13:36.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:13:36.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:13:36.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:36.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:36.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:13:36.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:13:36.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:13:36.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:13:36.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:13:36.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:13:36.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:13:36.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:13:36.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:36.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:36.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:13:36.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:13:36.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:13:36.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:13:36.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:13:36.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:13:36.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:13:36.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:13:36.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:36.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:36.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:13:36.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:13:36.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:13:36.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:13:36.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:13:36.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:13:36.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:13:36.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:13:36.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:36.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:36.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:13:36.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:13:36.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:13:36.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:13:36.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:13:36.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:13:36.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:13:36.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:36.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:13:37.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:37.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:37.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:37.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:13:37.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:37.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:13:37.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:37.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:13:37.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:37.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:13:37.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:37.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:13:37.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:37.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:13:37.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:37.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:13:37.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:37.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:13:37.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:37.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:37.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:37.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:13:37.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:37.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:13:38.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:38.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:13:38.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:38.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:13:38.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:38.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:13:38.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:38.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:13:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:13:38.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:38.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:13:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:38.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:38.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:38.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:13:38.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:38.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:13:38.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:38.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:13:38.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:38.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:13:38.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:38.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:13:39.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:39.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:13:39.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:39.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:13:39.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:39.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:13:39.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:39.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:39.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:39.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:13:39.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:39.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:13:39.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:39.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:13:39.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:39.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:13:39.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:39.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:13:39.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:39.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:13:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:39.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:13:40.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:13:40.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:40.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:40.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:40.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:13:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:40.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:13:40.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:40.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:13:40.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:40.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:13:40.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:40.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:13:40.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:40.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:13:40.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:40.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:13:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:13:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:40.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:13:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:40.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:13:41.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:41.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:13:41.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:41.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:13:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:13:41.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:41.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:13:41.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:41.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:13:41.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:41.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:13:41.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:41.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:41.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:41.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:13:41.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:41.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:13:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:13:41.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:41.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:13:42.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:42.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:13:42.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:42.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:13:42.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:42.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:13:46.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:13:46.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:46.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:46.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:13:46.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:13:46.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:13:46.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:13:46.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:13:46.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:13:46.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:13:46.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:13:46.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:46.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:46.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:13:46.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:13:46.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:13:46.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:13:46.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:13:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:13:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:13:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:13:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:46.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:46.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:13:46.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:13:46.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:13:46.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:13:46.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:13:46.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:13:46.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:13:46.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:13:46.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:46.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:46.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:13:46.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:13:46.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:13:46.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:13:46.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:13:46.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:13:46.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:13:46.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:13:46.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:46.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:46.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:13:46.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:13:46.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:13:46.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:13:46.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:13:46.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:13:46.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:13:46.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:13:46.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:46.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:46.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:13:46.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:13:46.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:13:46.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:13:46.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:13:46.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:13:47.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:47.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:13:47.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:47.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:13:47.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:47.221 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:47.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:47.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:13:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:13:47.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:47.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:13:47.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:47.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:13:48.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:48.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:13:48.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:48.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:13:48.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:48.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:13:48.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:48.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:13:48.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:48.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:48.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:48.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:13:48.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:48.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:13:48.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:48.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:13:48.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:48.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:13:48.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:48.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:13:48.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:48.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:13:48.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:48.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:13:49.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:49.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:13:49.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:49.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:49.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:49.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:13:49.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:49.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:13:49.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:49.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:13:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:49.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:13:49.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:49.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:13:49.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:49.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:13:49.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:49.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:13:49.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:49.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:13:49.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:49.926 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:49.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:49.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:13:50.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:50.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:13:50.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:50.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:13:50.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:50.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:13:50.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:50.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:13:50.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:50.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:13:50.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:50.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:13:50.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:50.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:13:50.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:50.679 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:50.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:13:50.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:50.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:13:50.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:50.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:13:50.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:50.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:13:51.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:51.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:13:51.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:51.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:13:51.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:51.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:13:51.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:51.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:13:51.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:51.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:51.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:51.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:13:51.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:51.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:13:51.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:51.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:13:51.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:51.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:13:51.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:51.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:13:51.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:51.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:13:52.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:52.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:13:52.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:52.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:13:52.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:52.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:13:52.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:52.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:13:52.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:52.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:13:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:13:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:52.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:13:52.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:52.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:13:52.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:52.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:13:52.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:52.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:13:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:53.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:13:53.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:53.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:13:53.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:53.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:13:53.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:53.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:13:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:13:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:13:53.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:53.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:13:53.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:53.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:13:53.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:53.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:53.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:53.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:13:53.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:53.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:13:54.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:54.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:13:54.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:54.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:13:54.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:54.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:13:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:54.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:13:54.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:54.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:13:54.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:54.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:13:54.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:54.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:54.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:54.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:13:54.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:54.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:13:54.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:54.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:13:54.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:54.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:13:54.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:54.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:13:55.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:55.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:13:55.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:55.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:13:55.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:55.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:13:55.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:55.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:55.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:55.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:13:55.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:55.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:13:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:13:55.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:55.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:13:55.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:55.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:13:55.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:55.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:13:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:13:56.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:56.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:13:56.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:56.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:13:56.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:56.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:13:56.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:56.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:13:56.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:56.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:13:56.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:56.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:13:56.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:56.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:13:56.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:56.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:13:56.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:56.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:13:56.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:56.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:56.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:56.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:13:57.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:57.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:13:57.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:57.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:13:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:57.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:13:57.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:57.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:13:57.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:57.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:13:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:57.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:13:57.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:57.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:13:57.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:57.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:57.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:57.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:13:57.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:57.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:13:57.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:57.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:13:58.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:58.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:13:58.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:58.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:13:58.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:58.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:13:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:13:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:13:58.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:58.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:13:58.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:58.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:13:58.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:58.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:13:58.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:58.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:13:58.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:58.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:13:59.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:59.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:13:59.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:59.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:13:59.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:59.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:13:59.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:13:59.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:13:59.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:59.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:13:59.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:59.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:13:59.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:59.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:13:59.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:59.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:13:59.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:59.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:13:59.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:59.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:13:59.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:13:59.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:14:00.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:00.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:14:00.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:00.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:00.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:00.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:14:00.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:00.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:14:00.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:00.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:14:00.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:00.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:14:00.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:00.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:14:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:14:00.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:00.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:14:00.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:00.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:14:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:00.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:00.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:00.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:14:00.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:00.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:14:01.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:01.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:14:01.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:01.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:14:01.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:01.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:14:01.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:01.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:14:01.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:01.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:14:01.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:01.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:14:01.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:01.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:01.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:01.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:14:01.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:01.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:14:01.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:01.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:14:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:14:02.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:02.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:14:02.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:02.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:14:02.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:02.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:14:02.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:02.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:14:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:02.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:14:02.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:02.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:14:02.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:02.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:14:02.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:02.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:14:02.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:02.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:14:02.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:02.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:14:03.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:03.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:14:03.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:03.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:14:03.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:03.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:03.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:03.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:14:03.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:03.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:14:03.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:03.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:14:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:14:03.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:03.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:14:03.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:03.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:14:03.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:03.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:14:03.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:03.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:14:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:04.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:14:04.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:04.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:14:04.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:04.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:14:04.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:04.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:14:04.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:04.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:14:04.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:04.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:14:04.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:04.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:14:04.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:04.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:14:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:04.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:14:04.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:04.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:14:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:14:05.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:05.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:14:05.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:05.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:14:05.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:05.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:14:05.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:05.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:14:05.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:05.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:14:05.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:05.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:05.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:14:05.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:05.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:14:05.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:05.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:14:05.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:05.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:14:05.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:05.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:14:06.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:06.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:14:06.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:06.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:14:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:14:06.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:06.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:06.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:06.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:14:06.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:06.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:14:06.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:06.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:14:06.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:06.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:14:06.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:06.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:14:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:06.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:14:06.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:06.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:14:07.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:07.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:14:07.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:07.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:14:07.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:07.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:14:07.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:07.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:14:07.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:07.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:14:07.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:07.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:14:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:07.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:14:07.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:07.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:14:07.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:07.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:14:07.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:07.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:07.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:07.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:14:08.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:08.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:14:12.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:14:12.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:14:12.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:14:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:14:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:14:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:14:12.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:12.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:14:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:14:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:14:12.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:14:12.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:14:12.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:14:12.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:14:12.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:14:12.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:12.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:12.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:14:12.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:14:12.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:14:12.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:14:12.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:14:12.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:14:12.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:14:12.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:14:12.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:12.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:12.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:14:12.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:14:12.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:14:12.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:14:12.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:14:12.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:14:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:14:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:14:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:12.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:14:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:14:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:14:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:14:12.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:14:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:14:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:14:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:14:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:12.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:14:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:14:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:14:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:14:12.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:14:12.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:14:12.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:14:12.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:14:12.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:12.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:14:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:14:12.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:14:12.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:14:12.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:12.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:14:13.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:13.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:14:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:14:13.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:13.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:14:13.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:13.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:13.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:13.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:14:13.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:13.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:14:13.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:13.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:14:13.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:13.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:14:13.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:13.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:14:13.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:13.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:14:14.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:14:14.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:14.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:14:14.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:14.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:14.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:14.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:14:14.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:14.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:14:14.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:14.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:14:14.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:14.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:14:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:14:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:14:14.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:14.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:14:14.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:14.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:14:14.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:14.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:14.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:14.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:14:15.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:15.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:14:15.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:15.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:14:15.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:15.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:14:15.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:15.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:14:15.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:15.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:14:15.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:15.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:14:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:14:15.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:15.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:15.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:15.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:14:15.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:15.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:14:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:14:16.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:16.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:14:16.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:16.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:14:16.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:16.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:14:16.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:16.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:14:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:14:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:16.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:16.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:16.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:14:16.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:16.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:14:16.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:16.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:14:16.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:16.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:14:16.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:16.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:14:17.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:17.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:14:17.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:17.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:14:17.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:17.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:14:17.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:17.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:17.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:17.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:14:17.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:17.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:14:17.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:17.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:14:17.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:17.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:14:18.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:18.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:14:18.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:18.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:14:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:14:18.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:18.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:14:18.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:18.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:18.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:18.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:14:18.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:18.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:14:18.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:18.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:14:19.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:19.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:14:19.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:19.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:14:19.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:19.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:14:19.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:19.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:14:19.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:19.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:14:19.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:19.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:19.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:19.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:14:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:14:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:14:19.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:19.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:14:19.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:19.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:14:20.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:20.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:14:20.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:20.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:14:20.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:20.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:14:20.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:20.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:20.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:20.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:14:20.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:20.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:14:20.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:20.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:14:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:20.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:14:20.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:20.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:14:20.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:20.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:14:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:14:21.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:21.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:14:21.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:21.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:21.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:21.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:14:21.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:21.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:14:21.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:21.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:14:21.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:21.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:14:21.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:21.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:14:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:21.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:14:21.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:21.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:14:21.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:21.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:14:21.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:21.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:21.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:21.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:14:22.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:22.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:14:22.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:22.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:14:22.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:22.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:14:22.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:22.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:14:22.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:22.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:14:22.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:22.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:14:22.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:22.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:14:22.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:22.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:14:26.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:14:26.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:14:26.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:14:26.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:14:26.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:14:26.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:14:26.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:14:26.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:26.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:26.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:14:26.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:14:26.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:14:26.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:14:26.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:14:26.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:14:26.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:14:26.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:14:26.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:26.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:26.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:14:26.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:14:26.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:14:26.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:14:26.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:14:26.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:14:26.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:14:26.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:14:26.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:26.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:26.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:14:26.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:14:26.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:14:26.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:14:26.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:14:26.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:14:26.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:14:26.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:14:26.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:26.906 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:26.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:14:26.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:14:26.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:14:26.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:14:26.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:14:26.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:14:26.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:14:26.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:14:26.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:26.908 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:26.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:14:26.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:14:26.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:14:26.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:26.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:14:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:14:27.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:27.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:14:27.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:27.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:14:27.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:27.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:14:27.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:27.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:27.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:27.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:14:27.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:27.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:14:27.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:27.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:14:27.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:27.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:14:27.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:27.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:14:27.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:27.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:14:28.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:28.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:14:28.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:28.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:14:28.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:28.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:28.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:28.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:14:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:14:28.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:28.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:14:28.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:28.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:14:28.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:28.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:14:28.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:28.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:14:28.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:28.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:14:28.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:28.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:14:28.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:28.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:29.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:29.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:14:29.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:29.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:14:29.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:29.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:14:29.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:29.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:14:29.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:29.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:14:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:29.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:14:29.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:29.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:14:29.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:29.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:14:29.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:29.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:29.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:29.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:14:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:14:29.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:29.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:14:30.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:30.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:14:30.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:30.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:14:30.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:30.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:14:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:14:30.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:30.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:14:30.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:30.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:30.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:30.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:14:30.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:30.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:14:30.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:30.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:14:30.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:30.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:14:30.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:30.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:14:30.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:30.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:14:31.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:31.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:14:31.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:31.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:14:31.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:31.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:31.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:31.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:14:31.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:31.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:14:31.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:31.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:14:31.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:31.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:14:31.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:31.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:14:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:31.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:14:31.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:31.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:14:31.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:31.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:14:32.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:32.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:32.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:32.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:14:32.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:32.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:14:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:14:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:32.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:14:32.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:32.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:14:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:14:36.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:14:36.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:14:36.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:36.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:36.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:14:36.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:14:36.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:14:36.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:14:36.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:14:36.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:14:36.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:14:36.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:14:36.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:36.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:36.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:14:36.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:14:36.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:14:36.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:14:36.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:14:36.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:14:36.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:14:36.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:14:36.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:36.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:36.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:14:36.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:14:36.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:14:36.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:14:36.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:14:36.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:14:36.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:14:36.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:14:36.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:36.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:36.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:14:36.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:14:36.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:14:36.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:14:36.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:14:36.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:14:36.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:14:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:14:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:36.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:14:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:14:36.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:14:36.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:14:36.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:14:36.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:14:36.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:14:36.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:14:36.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:36.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:36.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:14:36.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:14:36.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:36.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:14:37.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:37.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:14:37.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:37.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:14:37.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:37.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:14:37.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:37.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:14:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:14:37.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:37.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:37.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:37.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:14:37.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:37.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:14:37.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:37.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:14:37.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:37.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:14:37.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:37.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:14:38.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:38.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:14:38.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:38.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:14:38.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:38.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:14:38.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:38.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:38.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:38.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:14:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:14:38.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:38.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:14:38.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:38.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:14:38.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:38.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:14:38.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:38.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:14:38.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:38.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:14:39.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:39.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:14:39.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:39.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:39.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:39.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:14:39.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:39.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:14:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:14:39.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:39.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:14:39.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:39.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:14:39.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:39.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:14:39.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:39.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:14:39.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:39.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:14:39.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:39.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:39.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:39.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:14:39.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:39.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:14:40.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:40.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:14:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:40.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:14:40.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:40.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:14:40.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:40.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:14:40.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:40.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:14:40.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:40.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:14:40.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:40.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:40.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:40.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:14:40.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:40.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:14:40.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:40.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:14:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:40.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:14:41.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:41.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:14:41.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:41.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:14:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:41.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:14:41.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:41.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:14:41.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:41.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:41.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:41.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:14:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:14:41.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:41.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:14:41.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:41.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:14:41.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:41.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:14:41.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:41.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:14:41.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:41.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:14:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:14:42.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:42.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:42.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:42.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:14:42.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:42.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:14:42.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:42.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:14:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:14:42.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:42.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:14:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:14:42.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:42.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:14:42.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:42.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:14:42.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:42.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:42.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:14:42.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:42.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:14:43.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:43.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:14:43.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:43.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:14:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:43.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:14:43.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:43.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:14:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:14:43.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:43.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:14:43.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:43.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:43.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:43.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:14:43.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:43.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:14:43.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:43.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:14:44.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:44.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:14:44.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:44.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:14:44.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:44.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:14:44.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:44.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:14:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:14:44.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:44.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:44.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:44.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:14:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:14:44.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:44.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:14:44.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:44.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:14:44.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:14:45.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:45.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:14:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:14:45.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:45.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:14:45.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:45.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:45.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:45.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:14:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:14:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:14:45.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:45.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:14:45.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:14:45.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:45.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:14:45.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:45.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:14:46.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:46.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:14:46.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:46.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:46.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:46.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:14:46.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:46.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:14:46.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:46.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:14:46.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:14:54.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:14:54.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:14:54.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:14:54.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:14:54.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:54.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:54.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:14:54.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:14:54.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:14:54.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:14:54.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:14:54.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:14:54.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:14:54.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:14:54.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:54.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:54.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:14:54.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:14:54.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:14:54.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:14:54.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:14:54.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:14:54.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:14:54.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:14:54.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:54.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:54.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:14:54.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:14:54.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:14:54.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:14:54.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:14:54.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:14:54.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:14:54.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:14:54.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:54.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:54.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:14:54.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:14:54.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:14:54.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:14:54.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:14:54.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:14:54.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:14:54.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:14:54.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:54.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:54.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:14:54.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:14:54.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:14:54.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:14:54.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:14:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:14:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:14:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:14:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:54.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:54.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:14:54.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:14:54.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:14:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:14:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:14:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:14:54.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:14:54.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:14:54.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:54.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:54.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:14:54.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:14:54.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:14:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:14:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:14:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:14:54.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:14:54.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:14:54.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:54.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:54.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:14:54.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:14:54.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:14:54.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:14:54.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:14:54.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:14:54.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:14:54.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:14:54.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:54.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:54.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:14:54.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:14:54.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:14:54.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:14:54.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:14:54.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:14:54.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:14:54.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:14:54.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:54.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:54.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:14:54.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:14:54.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:14:54.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:14:54.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:14:54.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:14:54.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:14:54.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:14:54.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:54.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:54.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:14:54.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:54.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:14:55.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:55.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:14:55.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:55.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:14:55.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:55.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:14:55.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:55.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:14:55.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:55.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:14:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:14:55.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:55.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:55.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:55.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:14:55.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:55.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:14:55.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:55.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:14:55.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:55.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:14:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:14:56.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:56.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:14:56.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:56.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:14:56.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:56.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:14:56.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:56.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:56.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:14:56.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:56.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:14:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:14:56.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:56.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:14:56.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:56.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:14:56.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:56.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:14:56.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:56.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:14:57.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:57.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:14:57.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:57.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:57.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:57.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:14:57.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:57.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:14:57.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:57.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:14:57.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:57.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:14:57.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:57.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:14:57.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:57.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:14:57.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:57.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:14:57.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:57.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:14:57.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:57.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:57.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:57.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:14:58.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:58.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:14:58.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:58.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:14:58.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:58.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:14:58.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:58.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:14:58.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:58.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:14:58.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:58.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:14:58.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:58.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:14:58.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:58.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:58.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:14:58.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:58.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:14:58.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:58.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:14:58.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:58.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:14:59.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:59.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:14:59.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:59.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:14:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:14:59.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:14:59.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:14:59.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:14:59.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:59.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:14:59.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:59.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:14:59.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:59.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:14:59.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:59.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:14:59.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:59.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:14:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:14:59.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:15:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:15:00.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:00.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:15:00.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:00.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:00.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:00.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:15:00.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:00.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:15:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:00.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:15:00.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:00.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:15:00.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:00.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:15:00.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:00.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:15:00.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:00.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:15:00.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:00.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:15:01.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:01.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:01.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:01.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:15:01.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:15:01.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:01.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:15:01.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:01.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:15:01.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:01.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:15:01.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:01.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:15:01.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:01.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:15:01.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:01.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:15:01.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:01.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:01.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:01.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:15:01.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:01.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:15:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:15:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:02.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:15:02.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:02.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:15:02.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:02.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:15:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:02.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:15:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:15:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:02.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:15:02.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:02.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:15:02.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:02.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:15:02.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:02.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:15:02.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:02.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:15:03.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:03.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:15:03.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:03.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:15:03.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:03.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:15:03.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:03.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:03.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:03.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:15:03.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:03.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:15:03.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:03.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:15:03.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:03.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:15:03.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:03.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:15:03.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:03.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:15:03.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:03.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:15:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:15:04.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:04.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:04.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:04.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:15:04.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:04.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:15:04.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:04.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:15:04.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:04.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:15:04.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:04.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:15:04.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:04.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:15:04.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:04.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:15:04.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:04.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:15:04.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:04.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:04.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:04.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:15:04.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:04.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:15:05.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:05.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:15:05.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:05.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:15:05.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:05.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:15:05.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:05.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:15:05.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:05.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:15:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:05.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:15:05.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:05.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:05.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:05.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:15:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:05.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:15:05.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:05.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:15:05.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:05.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:15:06.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:06.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:15:06.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:06.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:15:06.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:06.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:15:06.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:06.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:15:06.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:06.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:06.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:06.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:15:06.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:06.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:15:06.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:06.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:15:06.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:06.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:15:06.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:06.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:15:06.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:06.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:15:07.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:07.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:15:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:15:07.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:07.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:07.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:07.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:15:07.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:07.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:15:07.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:07.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:15:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:07.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:15:07.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:07.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:15:07.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:07.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:15:07.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:07.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:15:07.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:07.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:15:08.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:08.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:08.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:08.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:15:08.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:08.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:15:08.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:08.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:15:08.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:08.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:15:08.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:08.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:15:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:08.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:15:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:15:08.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:08.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:15:08.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:08.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:08.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:08.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:15:08.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:08.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:15:08.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:08.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:15:09.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:09.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:15:09.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:09.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:15:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:15:09.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:09.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:15:09.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:09.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:15:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:09.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:09.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:09.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:15:09.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:09.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:15:09.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:09.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:15:09.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:09.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:15:09.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:09.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:15:10.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:10.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:15:10.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:15:14.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:15:14.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:14.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:14.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:15:14.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:15:14.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:15:14.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:15:14.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:15:14.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:15:14.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:15:14.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:15:14.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:14.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:14.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:15:14.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:15:14.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:15:14.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:15:14.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:15:14.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:15:14.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:15:14.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:15:14.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:14.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:14.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:15:14.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:15:14.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:15:14.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:15:14.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:15:14.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:15:14.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:15:14.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:15:14.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:14.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:14.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:15:14.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:15:14.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:15:14.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:15:14.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:15:14.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:15:14.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:15:14.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:15:14.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:14.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:14.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:15:14.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:15:14.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:15:14.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:15:14.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:15:14.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:15:14.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:15:14.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:15:14.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:14.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:14.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:15:14.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:15:14.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:15:14.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:15:14.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:15:14.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:15:14.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:15:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:15:15.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:15.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:19.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:15:19.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:15:19.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:15:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:15:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:15:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:15:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:15:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:15:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:19.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:15:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:15:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:15:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:15:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:15:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:15:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:15:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:15:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:19.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:15:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:15:19.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:15:19.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:15:19.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:15:19.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:15:19.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:15:19.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:15:19.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:19.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:19.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:15:19.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:15:19.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:15:19.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:15:19.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:15:19.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:15:19.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:15:19.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:15:19.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:19.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:19.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:15:19.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:15:19.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:15:19.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:15:19.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:15:19.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:15:19.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:15:19.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:15:19.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:19.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:19.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:15:19.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:15:19.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:15:19.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:15:19.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:15:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:15:19.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:15:19.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:15:19.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:19.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:19.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:19.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:15:20.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:20.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:15:20.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:20.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:15:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:15:20.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:20.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:15:20.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:20.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:15:20.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:20.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:15:20.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:20.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:15:20.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:20.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:20.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:20.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:15:20.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:20.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:15:20.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:20.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:15:21.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:21.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:15:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:21.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:15:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:15:21.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:21.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:15:21.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:21.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:15:21.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:21.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:21.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:21.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:15:21.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:21.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:15:21.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:21.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:15:21.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:21.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:15:21.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:21.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:15:22.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:22.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:15:22.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:22.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:15:22.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:22.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:15:22.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:22.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:22.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:22.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:15:22.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:22.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:15:22.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:22.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:15:22.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:22.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:15:22.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:22.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:15:22.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:22.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:15:22.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:22.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:15:22.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:22.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:15:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:23.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:15:23.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:23.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:15:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:15:23.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:23.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:15:23.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:23.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:15:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:23.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:15:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:15:23.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:23.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:15:23.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:23.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:23.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:23.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:15:23.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:23.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:15:24.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:24.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:15:24.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:24.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:15:24.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:15:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:15:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:15:32.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:15:32.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:32.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:32.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:15:32.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:15:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:15:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:15:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:15:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:15:32.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:15:32.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:15:32.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:32.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:32.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:15:32.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:15:32.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:15:32.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:15:32.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:15:32.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:15:32.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:15:32.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:15:32.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:32.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:32.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:15:32.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:15:32.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:15:32.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:15:32.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:15:32.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:15:32.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:15:32.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:15:32.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:32.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:32.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:15:32.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:15:32.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:15:32.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:15:32.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:15:32.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:15:32.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:15:32.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:15:32.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:32.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:32.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:15:32.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:15:32.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:15:32.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:15:32.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:15:32.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:15:32.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:15:32.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:15:32.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:32.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:32.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:15:32.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:15:32.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:15:32.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:15:32.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:15:32.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:15:32.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:15:32.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:15:32.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:32.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:32.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:15:32.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:15:32.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:15:32.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:15:32.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:15:32.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:15:32.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:15:32.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:15:32.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2020-11-16T21:15:32.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:32.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:15:32.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:15:32.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:15:32.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:15:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:15:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:15:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:15:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:15:32.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:223: TEST_backfill_test_multi: ceph pg dump pgs 2020-11-16T21:15:32.651 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:15:32.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:15:32.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test1 size 2 2020-11-16T21:15:32.651 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T21:15:32.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test2 size 2 2020-11-16T21:15:32.652 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 2 2020-11-16T21:15:32.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:32.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test3 size 2 2020-11-16T21:15:32.652 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:15:32.652 INFO:tasks.workunit.client.0.smithi189.stdout:8.0 591 0 0 0 0 2420736 0 0 91 91 active+clean 2020-11-16T20:54:05.167704+0000 146'591 146:611 [3] 3 [3] 3 0'0 2020-11-16T20:54:01.389262+0000 0'0 2020-11-16T20:54:01.389262+0000 0 2020-11-16T21:15:32.653 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 592 0 0 0 0 2424832 0 0 92 92 active+clean 2020-11-16T20:54:01.450453+0000 146'592 146:610 [5] 5 [5] 5 0'0 2020-11-16T20:53:58.234741+0000 0'0 2020-11-16T20:53:58.234741+0000 0 2020-11-16T21:15:32.653 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 598 0 0 0 0 2449408 0 0 98 98 active+clean 2020-11-16T20:53:47.953472+0000 146'598 146:634 [17] 17 [17] 17 0'0 2020-11-16T20:53:44.302494+0000 0'0 2020-11-16T20:53:44.302494+0000 0 2020-11-16T21:15:32.653 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 598 0 0 0 0 2449408 0 0 98 98 active+clean 2020-11-16T20:53:45.416870+0000 146'598 146:640 [8] 8 [8] 8 0'0 2020-11-16T20:53:41.661066+0000 0'0 2020-11-16T20:53:41.661066+0000 0 2020-11-16T21:15:32.653 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 598 0 0 0 0 2449408 0 0 98 98 active+clean 2020-11-16T20:53:50.073816+0000 146'598 146:638 [1] 1 [1] 1 0'0 2020-11-16T20:53:46.927735+0000 0'0 2020-11-16T20:53:46.927735+0000 0 2020-11-16T21:15:32.653 INFO:tasks.workunit.client.0.smithi189.stdout:4.0 595 0 0 0 0 2437120 0 0 95 95 active+clean 2020-11-16T20:53:53.215263+0000 146'595 146:621 [4] 4 [4] 4 0'0 2020-11-16T20:53:49.973123+0000 0'0 2020-11-16T20:53:49.973123+0000 0 2020-11-16T21:15:32.653 INFO:tasks.workunit.client.0.smithi189.stdout:5.0 594 0 0 0 0 2433024 0 0 94 94 active+clean 2020-11-16T20:53:55.708989+0000 146'594 146:622 [12] 12 [12] 12 0'0 2020-11-16T20:53:52.540182+0000 0'0 2020-11-16T20:53:52.540182+0000 0 2020-11-16T21:15:32.653 INFO:tasks.workunit.client.0.smithi189.stdout:6.0 594 0 0 0 0 2433024 0 0 94 94 active+clean 2020-11-16T20:53:59.697500+0000 146'594 146:618 [7] 7 [7] 7 0'0 2020-11-16T20:53:55.565820+0000 0'0 2020-11-16T20:53:55.565820+0000 0 2020-11-16T21:15:32.654 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:15:32.654 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:15:33.557 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 2 2020-11-16T21:15:33.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:33.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test4 size 2 2020-11-16T21:15:35.541 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 2 2020-11-16T21:15:35.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:35.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test5 size 2 2020-11-16T21:15:37.372 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 5 size to 2 2020-11-16T21:15:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test6 size 2 2020-11-16T21:15:39.314 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 6 size to 2 2020-11-16T21:15:39.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:39.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test7 size 2 2020-11-16T21:15:40.593 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 7 size to 2 2020-11-16T21:15:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2020-11-16T21:15:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test8 size 2 2020-11-16T21:15:42.583 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 8 size to 2 2020-11-16T21:15:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:229: TEST_backfill_test_multi: sleep 30 2020-11-16T21:16:12.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:231: TEST_backfill_test_multi: wait_for_not_backfilling 240 2020-11-16T21:16:12.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2020-11-16T21:16:12.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2020-11-16T21:16:12.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2020-11-16T21:16:12.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:16:12.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:16:12.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:16:12.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2020-11-16T21:16:12.598 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:16:12.598 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:16:12.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:16:12.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:16:12.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:16:12.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:16:12.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:16:12.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:16:12.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:16:12.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:16:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:16:13.272 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:16:13.272 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:16:13.272 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:16:13.272 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:16:13.272 INFO:tasks.workunit.client.0.smithi189.stderr:5 2020-11-16T21:16:13.272 INFO:tasks.workunit.client.0.smithi189.stderr:6 2020-11-16T21:16:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:7 2020-11-16T21:16:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:8 2020-11-16T21:16:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:9 2020-11-16T21:16:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:10 2020-11-16T21:16:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:11 2020-11-16T21:16:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:12 2020-11-16T21:16:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:13 2020-11-16T21:16:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:14 2020-11-16T21:16:13.274 INFO:tasks.workunit.client.0.smithi189.stderr:15 2020-11-16T21:16:13.274 INFO:tasks.workunit.client.0.smithi189.stderr:16 2020-11-16T21:16:13.274 INFO:tasks.workunit.client.0.smithi189.stderr:17 2020-11-16T21:16:13.274 INFO:tasks.workunit.client.0.smithi189.stderr:18 2020-11-16T21:16:13.274 INFO:tasks.workunit.client.0.smithi189.stderr:19' 2020-11-16T21:16:13.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:16:13.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:13.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:16:13.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836789 2020-11-16T21:16:13.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836789 2020-11-16T21:16:13.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789' 2020-11-16T21:16:13.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:13.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:16:13.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705972 2020-11-16T21:16:13.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705972 2020-11-16T21:16:13.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972' 2020-11-16T21:16:13.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:13.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:16:13.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542450 2020-11-16T21:16:13.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542450 2020-11-16T21:16:13.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450' 2020-11-16T21:16:13.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:13.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:16:13.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378929 2020-11-16T21:16:13.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378929 2020-11-16T21:16:13.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929' 2020-11-16T21:16:13.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:13.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:16:14.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784248111 2020-11-16T21:16:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784248111 2020-11-16T21:16:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111' 2020-11-16T21:16:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:14.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:16:14.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084589 2020-11-16T21:16:14.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084589 2020-11-16T21:16:14.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589' 2020-11-16T21:16:14.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:14.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-11-16T21:16:14.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733921068 2020-11-16T21:16:14.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733921068 2020-11-16T21:16:14.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068' 2020-11-16T21:16:14.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:14.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-11-16T21:16:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757546 2020-11-16T21:16:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757546 2020-11-16T21:16:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546' 2020-11-16T21:16:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:14.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-11-16T21:16:14.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=184683594025 2020-11-16T21:16:14.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 184683594025 2020-11-16T21:16:14.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025' 2020-11-16T21:16:14.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:14.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-11-16T21:16:15.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=206158430503 2020-11-16T21:16:15.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 206158430503 2020-11-16T21:16:15.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025 9-206158430503' 2020-11-16T21:16:15.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:15.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-11-16T21:16:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=227633266981 2020-11-16T21:16:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 227633266981 2020-11-16T21:16:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025 9-206158430503 10-227633266981' 2020-11-16T21:16:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:15.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.11 flush_pg_stats 2020-11-16T21:16:15.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=249108103460 2020-11-16T21:16:15.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 249108103460 2020-11-16T21:16:15.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025 9-206158430503 10-227633266981 11-249108103460' 2020-11-16T21:16:15.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:15.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.12 flush_pg_stats 2020-11-16T21:16:15.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=270582939938 2020-11-16T21:16:15.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 270582939938 2020-11-16T21:16:15.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025 9-206158430503 10-227633266981 11-249108103460 12-270582939938' 2020-11-16T21:16:15.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:15.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.13 flush_pg_stats 2020-11-16T21:16:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=292057776416 2020-11-16T21:16:15.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 292057776416 2020-11-16T21:16:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025 9-206158430503 10-227633266981 11-249108103460 12-270582939938 13-292057776416' 2020-11-16T21:16:15.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:15.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.14 flush_pg_stats 2020-11-16T21:16:15.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=313532612895 2020-11-16T21:16:15.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 313532612895 2020-11-16T21:16:15.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025 9-206158430503 10-227633266981 11-249108103460 12-270582939938 13-292057776416 14-313532612895' 2020-11-16T21:16:15.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:15.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.15 flush_pg_stats 2020-11-16T21:16:16.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=335007449372 2020-11-16T21:16:16.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 335007449372 2020-11-16T21:16:16.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025 9-206158430503 10-227633266981 11-249108103460 12-270582939938 13-292057776416 14-313532612895 15-335007449372' 2020-11-16T21:16:16.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:16.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.16 flush_pg_stats 2020-11-16T21:16:16.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=356482285850 2020-11-16T21:16:16.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 356482285850 2020-11-16T21:16:16.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025 9-206158430503 10-227633266981 11-249108103460 12-270582939938 13-292057776416 14-313532612895 15-335007449372 16-356482285850' 2020-11-16T21:16:16.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:16.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.17 flush_pg_stats 2020-11-16T21:16:16.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=377957122328 2020-11-16T21:16:16.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 377957122328 2020-11-16T21:16:16.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025 9-206158430503 10-227633266981 11-249108103460 12-270582939938 13-292057776416 14-313532612895 15-335007449372 16-356482285850 17-377957122328' 2020-11-16T21:16:16.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:16.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.18 flush_pg_stats 2020-11-16T21:16:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=399431958806 2020-11-16T21:16:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 399431958806 2020-11-16T21:16:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025 9-206158430503 10-227633266981 11-249108103460 12-270582939938 13-292057776416 14-313532612895 15-335007449372 16-356482285850 17-377957122328 18-399431958806' 2020-11-16T21:16:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:16:16.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.19 flush_pg_stats 2020-11-16T21:16:16.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=420906795284 2020-11-16T21:16:16.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 420906795284 2020-11-16T21:16:16.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836789 1-38654705972 2-60129542450 3-81604378929 4-98784248111 5-120259084589 6-141733921068 7-163208757546 8-184683594025 9-206158430503 10-227633266981 11-249108103460 12-270582939938 13-292057776416 14-313532612895 15-335007449372 16-356482285850 17-377957122328 18-399431958806 19-420906795284' 2020-11-16T21:16:16.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:16.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836789 2020-11-16T21:16:16.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:16.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:16:16.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:16.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836789 2020-11-16T21:16:16.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836789 2020-11-16T21:16:16.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836789' 2020-11-16T21:16:16.724 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836789 2020-11-16T21:16:16.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:16:17.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836789 -lt 21474836789 2020-11-16T21:16:17.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:17.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705972 2020-11-16T21:16:17.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:17.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:16:17.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705972 2020-11-16T21:16:17.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:17.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705972 2020-11-16T21:16:17.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705972' 2020-11-16T21:16:17.147 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705972 2020-11-16T21:16:17.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:16:17.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705973 -lt 38654705972 2020-11-16T21:16:17.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:17.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542450 2020-11-16T21:16:17.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:17.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:16:17.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542450 2020-11-16T21:16:17.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542450 2020-11-16T21:16:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542450' 2020-11-16T21:16:17.563 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542450 2020-11-16T21:16:17.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:16:17.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542451 -lt 60129542450 2020-11-16T21:16:17.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:17.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378929 2020-11-16T21:16:17.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:17.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:16:17.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378929 2020-11-16T21:16:17.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:17.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378929 2020-11-16T21:16:17.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378929' 2020-11-16T21:16:17.970 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378929 2020-11-16T21:16:17.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:16:18.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378930 -lt 81604378929 2020-11-16T21:16:18.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:18.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784248111 2020-11-16T21:16:18.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:18.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:16:18.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784248111 2020-11-16T21:16:18.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:18.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784248111 2020-11-16T21:16:18.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784248111' 2020-11-16T21:16:18.392 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784248111 2020-11-16T21:16:18.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:16:18.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784248112 -lt 98784248111 2020-11-16T21:16:18.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:18.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084589 2020-11-16T21:16:18.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:18.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:16:18.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084589 2020-11-16T21:16:18.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:18.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084589 2020-11-16T21:16:18.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084589' 2020-11-16T21:16:18.792 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 120259084589 2020-11-16T21:16:18.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:16:19.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084590 -lt 120259084589 2020-11-16T21:16:19.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:19.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-141733921068 2020-11-16T21:16:19.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:19.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-11-16T21:16:19.185 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-141733921068 2020-11-16T21:16:19.185 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:19.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733921068 2020-11-16T21:16:19.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 141733921068' 2020-11-16T21:16:19.187 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.6 seq 141733921068 2020-11-16T21:16:19.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-11-16T21:16:19.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733921069 -lt 141733921068 2020-11-16T21:16:19.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:19.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-163208757546 2020-11-16T21:16:19.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-11-16T21:16:19.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-163208757546 2020-11-16T21:16:19.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:19.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757546 2020-11-16T21:16:19.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 163208757546' 2020-11-16T21:16:19.683 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.7 seq 163208757546 2020-11-16T21:16:19.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-11-16T21:16:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757547 -lt 163208757546 2020-11-16T21:16:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:20.078 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-184683594025 2020-11-16T21:16:20.078 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:20.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-11-16T21:16:20.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:20.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-184683594025 2020-11-16T21:16:20.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=184683594025 2020-11-16T21:16:20.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 184683594025' 2020-11-16T21:16:20.082 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.8 seq 184683594025 2020-11-16T21:16:20.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-11-16T21:16:20.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 184683594026 -lt 184683594025 2020-11-16T21:16:20.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:20.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-206158430503 2020-11-16T21:16:20.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:20.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-11-16T21:16:20.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-206158430503 2020-11-16T21:16:20.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:20.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=206158430503 2020-11-16T21:16:20.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 206158430503' 2020-11-16T21:16:20.548 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.9 seq 206158430503 2020-11-16T21:16:20.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-11-16T21:16:21.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 206158430504 -lt 206158430503 2020-11-16T21:16:21.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:21.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-227633266981 2020-11-16T21:16:21.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:21.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-11-16T21:16:21.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-227633266981 2020-11-16T21:16:21.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:21.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=227633266981 2020-11-16T21:16:21.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 227633266981' 2020-11-16T21:16:21.027 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.10 seq 227633266981 2020-11-16T21:16:21.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-11-16T21:16:21.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 227633266982 -lt 227633266981 2020-11-16T21:16:21.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:21.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 11-249108103460 2020-11-16T21:16:21.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:21.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=11 2020-11-16T21:16:21.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 11-249108103460 2020-11-16T21:16:21.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:21.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=249108103460 2020-11-16T21:16:21.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.11 seq 249108103460' 2020-11-16T21:16:21.576 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.11 seq 249108103460 2020-11-16T21:16:21.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 11 2020-11-16T21:16:21.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 249108103461 -lt 249108103460 2020-11-16T21:16:21.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:21.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 12-270582939938 2020-11-16T21:16:21.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:21.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=12 2020-11-16T21:16:21.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:21.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 12-270582939938 2020-11-16T21:16:21.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=270582939938 2020-11-16T21:16:21.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.12 seq 270582939938' 2020-11-16T21:16:21.989 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.12 seq 270582939938 2020-11-16T21:16:21.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 12 2020-11-16T21:16:22.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 270582939940 -lt 270582939938 2020-11-16T21:16:22.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:22.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 13-292057776416 2020-11-16T21:16:22.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:22.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=13 2020-11-16T21:16:22.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 13-292057776416 2020-11-16T21:16:22.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:22.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=292057776416 2020-11-16T21:16:22.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.13 seq 292057776416' 2020-11-16T21:16:22.446 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.13 seq 292057776416 2020-11-16T21:16:22.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 13 2020-11-16T21:16:22.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 292057776417 -lt 292057776416 2020-11-16T21:16:22.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:22.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 14-313532612895 2020-11-16T21:16:22.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:22.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=14 2020-11-16T21:16:22.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:22.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 14-313532612895 2020-11-16T21:16:22.833 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.14 seq 313532612895 2020-11-16T21:16:22.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=313532612895 2020-11-16T21:16:22.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.14 seq 313532612895' 2020-11-16T21:16:22.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 14 2020-11-16T21:16:23.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 313532612896 -lt 313532612895 2020-11-16T21:16:23.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:23.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 15-335007449372 2020-11-16T21:16:23.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:23.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=15 2020-11-16T21:16:23.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:23.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 15-335007449372 2020-11-16T21:16:23.246 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.15 seq 335007449372 2020-11-16T21:16:23.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=335007449372 2020-11-16T21:16:23.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.15 seq 335007449372' 2020-11-16T21:16:23.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 15 2020-11-16T21:16:23.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 335007449373 -lt 335007449372 2020-11-16T21:16:23.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:23.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 16-356482285850 2020-11-16T21:16:23.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:23.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=16 2020-11-16T21:16:23.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 16-356482285850 2020-11-16T21:16:23.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=356482285850 2020-11-16T21:16:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.16 seq 356482285850' 2020-11-16T21:16:23.676 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.16 seq 356482285850 2020-11-16T21:16:23.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 16 2020-11-16T21:16:24.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 356482285851 -lt 356482285850 2020-11-16T21:16:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:24.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 17-377957122328 2020-11-16T21:16:24.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:24.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=17 2020-11-16T21:16:24.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:24.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 17-377957122328 2020-11-16T21:16:24.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=377957122328 2020-11-16T21:16:24.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.17 seq 377957122328' 2020-11-16T21:16:24.095 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.17 seq 377957122328 2020-11-16T21:16:24.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 17 2020-11-16T21:16:24.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 377957122329 -lt 377957122328 2020-11-16T21:16:24.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:24.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 18-399431958806 2020-11-16T21:16:24.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:24.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=18 2020-11-16T21:16:24.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 18-399431958806 2020-11-16T21:16:24.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=399431958806 2020-11-16T21:16:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.18 seq 399431958806' 2020-11-16T21:16:24.496 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.18 seq 399431958806 2020-11-16T21:16:24.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 18 2020-11-16T21:16:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 399431958807 -lt 399431958806 2020-11-16T21:16:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:16:24.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 19-420906795284 2020-11-16T21:16:24.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:16:24.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=19 2020-11-16T21:16:24.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 19-420906795284 2020-11-16T21:16:24.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:16:24.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=420906795284 2020-11-16T21:16:24.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.19 seq 420906795284' 2020-11-16T21:16:24.890 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.19 seq 420906795284 2020-11-16T21:16:24.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 19 2020-11-16T21:16:25.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 420906795285 -lt 420906795284 2020-11-16T21:16:25.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:16:25.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:16:25.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:16:25.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 8 == 0 2020-11-16T21:16:25.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:16:25.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:16:25.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:16:25.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:16:25.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:16:25.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:16:25.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:16:26.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2020-11-16T21:16:26.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2020-11-16T21:16:26.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' -1 2020-11-16T21:16:26.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:68: wait_for_not_state: loop=0 2020-11-16T21:16:26.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:69: wait_for_not_state: num_in_state=4 2020-11-16T21:16:26.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 5 2020-11-16T21:16:31.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:16:31.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:16:31.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:16:31.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:16:31.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:16:31.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:16:31.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:16:31.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:16:31.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2020-11-16T21:16:31.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2020-11-16T21:16:31.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2020-11-16T21:16:31.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 1 >= 17 )) 2020-11-16T21:16:31.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 10 2020-11-16T21:16:41.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:16:41.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:16:41.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:16:41.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:16:41.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:16:41.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:16:41.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:16:41.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:16:41.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2020-11-16T21:16:42.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2020-11-16T21:16:42.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2020-11-16T21:16:42.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 2 >= 17 )) 2020-11-16T21:16:42.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:16:57.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:16:57.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:16:57.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:16:57.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:16:57.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:16:57.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:16:57.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:16:57.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:16:57.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2020-11-16T21:16:57.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2020-11-16T21:16:57.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2020-11-16T21:16:57.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 3 >= 17 )) 2020-11-16T21:16:57.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:17:12.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:17:12.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:17:12.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:17:12.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:17:12.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:17:12.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:17:12.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:17:12.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:17:12.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2020-11-16T21:17:12.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2020-11-16T21:17:12.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2020-11-16T21:17:12.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 4 >= 17 )) 2020-11-16T21:17:12.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:17:27.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:17:27.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:17:27.806 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:17:27.806 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:17:27.806 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:17:27.806 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:17:27.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:17:27.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:17:28.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2020-11-16T21:17:28.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2020-11-16T21:17:28.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2020-11-16T21:17:28.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 5 >= 17 )) 2020-11-16T21:17:28.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:17:43.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:17:43.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:17:43.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:17:43.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:17:43.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:17:43.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:17:43.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:17:43.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:17:43.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2020-11-16T21:17:43.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2020-11-16T21:17:43.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2020-11-16T21:17:43.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 6 >= 17 )) 2020-11-16T21:17:43.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:17:58.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:17:58.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:17:58.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:17:58.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:17:58.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:17:58.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:17:58.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:17:58.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:17:59.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2020-11-16T21:17:59.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2020-11-16T21:17:59.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2020-11-16T21:17:59.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 7 >= 17 )) 2020-11-16T21:17:59.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:18:14.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:18:14.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:18:14.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:18:14.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:18:14.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:18:14.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:18:14.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:18:14.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:18:14.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2020-11-16T21:18:14.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2020-11-16T21:18:14.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2020-11-16T21:18:14.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 8 >= 17 )) 2020-11-16T21:18:14.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:18:29.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:18:29.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:18:29.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:18:29.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:18:29.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:18:29.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:18:29.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:18:29.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:18:29.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:18:29.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:18:29.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:18:29.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:18:29.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:232: TEST_backfill_test_multi: wait_for_not_activating 60 2020-11-16T21:18:29.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2020-11-16T21:18:29.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2020-11-16T21:18:29.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2020-11-16T21:18:29.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:18:29.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:18:29.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:18:29.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2020-11-16T21:18:29.807 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:18:29.807 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:18:29.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:18:29.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:18:29.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:18:29.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:18:29.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:18:29.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:18:29.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:18:29.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:18:30.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:18:30.309 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:18:30.309 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:18:30.309 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:18:30.310 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:18:30.310 INFO:tasks.workunit.client.0.smithi189.stderr:5 2020-11-16T21:18:30.310 INFO:tasks.workunit.client.0.smithi189.stderr:6 2020-11-16T21:18:30.310 INFO:tasks.workunit.client.0.smithi189.stderr:7 2020-11-16T21:18:30.310 INFO:tasks.workunit.client.0.smithi189.stderr:8 2020-11-16T21:18:30.310 INFO:tasks.workunit.client.0.smithi189.stderr:9 2020-11-16T21:18:30.311 INFO:tasks.workunit.client.0.smithi189.stderr:10 2020-11-16T21:18:30.311 INFO:tasks.workunit.client.0.smithi189.stderr:11 2020-11-16T21:18:30.311 INFO:tasks.workunit.client.0.smithi189.stderr:12 2020-11-16T21:18:30.311 INFO:tasks.workunit.client.0.smithi189.stderr:13 2020-11-16T21:18:30.311 INFO:tasks.workunit.client.0.smithi189.stderr:14 2020-11-16T21:18:30.311 INFO:tasks.workunit.client.0.smithi189.stderr:15 2020-11-16T21:18:30.311 INFO:tasks.workunit.client.0.smithi189.stderr:16 2020-11-16T21:18:30.311 INFO:tasks.workunit.client.0.smithi189.stderr:17 2020-11-16T21:18:30.312 INFO:tasks.workunit.client.0.smithi189.stderr:18 2020-11-16T21:18:30.312 INFO:tasks.workunit.client.0.smithi189.stderr:19' 2020-11-16T21:18:30.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:18:30.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:30.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:18:30.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836818 2020-11-16T21:18:30.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836818 2020-11-16T21:18:30.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818' 2020-11-16T21:18:30.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:30.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:18:30.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654706002 2020-11-16T21:18:30.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654706002 2020-11-16T21:18:30.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002' 2020-11-16T21:18:30.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:30.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:18:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542479 2020-11-16T21:18:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542479 2020-11-16T21:18:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479' 2020-11-16T21:18:30.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:30.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:18:30.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378959 2020-11-16T21:18:30.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378959 2020-11-16T21:18:30.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959' 2020-11-16T21:18:30.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:30.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:18:31.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784248140 2020-11-16T21:18:31.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784248140 2020-11-16T21:18:31.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140' 2020-11-16T21:18:31.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:31.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:18:31.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084619 2020-11-16T21:18:31.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084619 2020-11-16T21:18:31.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619' 2020-11-16T21:18:31.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:31.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-11-16T21:18:31.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733921097 2020-11-16T21:18:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733921097 2020-11-16T21:18:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097' 2020-11-16T21:18:31.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:31.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-11-16T21:18:31.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757576 2020-11-16T21:18:31.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757576 2020-11-16T21:18:31.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576' 2020-11-16T21:18:31.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:31.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-11-16T21:18:31.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=184683594054 2020-11-16T21:18:31.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 184683594054 2020-11-16T21:18:31.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054' 2020-11-16T21:18:31.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:31.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-11-16T21:18:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=206158430532 2020-11-16T21:18:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 206158430532 2020-11-16T21:18:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054 9-206158430532' 2020-11-16T21:18:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:32.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-11-16T21:18:32.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=227633267010 2020-11-16T21:18:32.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 227633267010 2020-11-16T21:18:32.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054 9-206158430532 10-227633267010' 2020-11-16T21:18:32.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:32.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.11 flush_pg_stats 2020-11-16T21:18:32.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=249108103490 2020-11-16T21:18:32.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 249108103490 2020-11-16T21:18:32.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054 9-206158430532 10-227633267010 11-249108103490' 2020-11-16T21:18:32.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:32.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.12 flush_pg_stats 2020-11-16T21:18:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=270582939968 2020-11-16T21:18:32.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 270582939968 2020-11-16T21:18:32.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054 9-206158430532 10-227633267010 11-249108103490 12-270582939968' 2020-11-16T21:18:32.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:32.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.13 flush_pg_stats 2020-11-16T21:18:32.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=292057776445 2020-11-16T21:18:32.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 292057776445 2020-11-16T21:18:32.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054 9-206158430532 10-227633267010 11-249108103490 12-270582939968 13-292057776445' 2020-11-16T21:18:32.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:32.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.14 flush_pg_stats 2020-11-16T21:18:32.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=313532612924 2020-11-16T21:18:32.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 313532612924 2020-11-16T21:18:32.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054 9-206158430532 10-227633267010 11-249108103490 12-270582939968 13-292057776445 14-313532612924' 2020-11-16T21:18:32.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:32.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.15 flush_pg_stats 2020-11-16T21:18:33.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=335007449401 2020-11-16T21:18:33.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 335007449401 2020-11-16T21:18:33.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054 9-206158430532 10-227633267010 11-249108103490 12-270582939968 13-292057776445 14-313532612924 15-335007449401' 2020-11-16T21:18:33.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:33.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.16 flush_pg_stats 2020-11-16T21:18:33.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=356482285879 2020-11-16T21:18:33.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 356482285879 2020-11-16T21:18:33.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054 9-206158430532 10-227633267010 11-249108103490 12-270582939968 13-292057776445 14-313532612924 15-335007449401 16-356482285879' 2020-11-16T21:18:33.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:33.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.17 flush_pg_stats 2020-11-16T21:18:33.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=377957122357 2020-11-16T21:18:33.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 377957122357 2020-11-16T21:18:33.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054 9-206158430532 10-227633267010 11-249108103490 12-270582939968 13-292057776445 14-313532612924 15-335007449401 16-356482285879 17-377957122357' 2020-11-16T21:18:33.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:33.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.18 flush_pg_stats 2020-11-16T21:18:33.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=399431958835 2020-11-16T21:18:33.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 399431958835 2020-11-16T21:18:33.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054 9-206158430532 10-227633267010 11-249108103490 12-270582939968 13-292057776445 14-313532612924 15-335007449401 16-356482285879 17-377957122357 18-399431958835' 2020-11-16T21:18:33.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:18:33.602 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.19 flush_pg_stats 2020-11-16T21:18:33.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=420906795313 2020-11-16T21:18:33.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 420906795313 2020-11-16T21:18:33.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836818 1-38654706002 2-60129542479 3-81604378959 4-98784248140 5-120259084619 6-141733921097 7-163208757576 8-184683594054 9-206158430532 10-227633267010 11-249108103490 12-270582939968 13-292057776445 14-313532612924 15-335007449401 16-356482285879 17-377957122357 18-399431958835 19-420906795313' 2020-11-16T21:18:33.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:33.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836818 2020-11-16T21:18:33.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:33.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:18:33.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:33.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836818 2020-11-16T21:18:33.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836818 2020-11-16T21:18:33.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836818' 2020-11-16T21:18:33.772 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836818 2020-11-16T21:18:33.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:18:34.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836819 -lt 21474836818 2020-11-16T21:18:34.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:34.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:34.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654706002 2020-11-16T21:18:34.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:18:34.173 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654706002 2020-11-16T21:18:34.173 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:34.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654706002 2020-11-16T21:18:34.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654706002' 2020-11-16T21:18:34.175 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654706002 2020-11-16T21:18:34.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:18:34.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654706002 -lt 38654706002 2020-11-16T21:18:34.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:34.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542479 2020-11-16T21:18:34.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:34.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:18:34.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542479 2020-11-16T21:18:34.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:34.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542479 2020-11-16T21:18:34.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542479' 2020-11-16T21:18:34.573 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542479 2020-11-16T21:18:34.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:18:34.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542480 -lt 60129542479 2020-11-16T21:18:34.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:34.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378959 2020-11-16T21:18:34.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:34.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:18:34.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378959 2020-11-16T21:18:34.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:34.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378959 2020-11-16T21:18:34.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378959' 2020-11-16T21:18:34.987 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378959 2020-11-16T21:18:34.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:18:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378959 -lt 81604378959 2020-11-16T21:18:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:35.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784248140 2020-11-16T21:18:35.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:18:35.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784248140 2020-11-16T21:18:35.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784248140 2020-11-16T21:18:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784248140' 2020-11-16T21:18:35.380 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784248140 2020-11-16T21:18:35.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:18:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784248141 -lt 98784248140 2020-11-16T21:18:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:35.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084619 2020-11-16T21:18:35.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:18:35.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084619 2020-11-16T21:18:35.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:35.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084619 2020-11-16T21:18:35.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084619' 2020-11-16T21:18:35.788 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 120259084619 2020-11-16T21:18:35.788 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:18:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084620 -lt 120259084619 2020-11-16T21:18:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:36.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-141733921097 2020-11-16T21:18:36.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:36.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-11-16T21:18:36.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-141733921097 2020-11-16T21:18:36.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733921097 2020-11-16T21:18:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 141733921097' 2020-11-16T21:18:36.176 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.6 seq 141733921097 2020-11-16T21:18:36.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-11-16T21:18:36.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733921098 -lt 141733921097 2020-11-16T21:18:36.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:36.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-163208757576 2020-11-16T21:18:36.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:36.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-11-16T21:18:36.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-163208757576 2020-11-16T21:18:36.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:36.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757576 2020-11-16T21:18:36.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 163208757576' 2020-11-16T21:18:36.561 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.7 seq 163208757576 2020-11-16T21:18:36.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-11-16T21:18:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757576 -lt 163208757576 2020-11-16T21:18:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:36.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-184683594054 2020-11-16T21:18:36.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:36.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-11-16T21:18:36.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-184683594054 2020-11-16T21:18:36.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:36.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=184683594054 2020-11-16T21:18:36.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 184683594054' 2020-11-16T21:18:36.964 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.8 seq 184683594054 2020-11-16T21:18:36.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-11-16T21:18:37.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 184683594055 -lt 184683594054 2020-11-16T21:18:37.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:37.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-206158430532 2020-11-16T21:18:37.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:37.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-11-16T21:18:37.361 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-206158430532 2020-11-16T21:18:37.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:37.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=206158430532 2020-11-16T21:18:37.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 206158430532' 2020-11-16T21:18:37.363 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.9 seq 206158430532 2020-11-16T21:18:37.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-11-16T21:18:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 206158430533 -lt 206158430532 2020-11-16T21:18:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:37.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-227633267010 2020-11-16T21:18:37.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:37.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-11-16T21:18:37.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-227633267010 2020-11-16T21:18:37.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:37.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=227633267010 2020-11-16T21:18:37.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 227633267010' 2020-11-16T21:18:37.767 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.10 seq 227633267010 2020-11-16T21:18:37.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-11-16T21:18:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 227633267012 -lt 227633267010 2020-11-16T21:18:38.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:38.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 11-249108103490 2020-11-16T21:18:38.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=11 2020-11-16T21:18:38.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 11-249108103490 2020-11-16T21:18:38.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=249108103490 2020-11-16T21:18:38.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.11 seq 249108103490' 2020-11-16T21:18:38.168 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.11 seq 249108103490 2020-11-16T21:18:38.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 11 2020-11-16T21:18:38.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 249108103491 -lt 249108103490 2020-11-16T21:18:38.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:38.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 12-270582939968 2020-11-16T21:18:38.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:38.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=12 2020-11-16T21:18:38.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 12-270582939968 2020-11-16T21:18:38.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:38.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=270582939968 2020-11-16T21:18:38.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.12 seq 270582939968' 2020-11-16T21:18:38.555 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.12 seq 270582939968 2020-11-16T21:18:38.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 12 2020-11-16T21:18:38.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 270582939969 -lt 270582939968 2020-11-16T21:18:38.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:38.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 13-292057776445 2020-11-16T21:18:38.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:38.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=13 2020-11-16T21:18:38.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 13-292057776445 2020-11-16T21:18:38.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:38.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=292057776445 2020-11-16T21:18:38.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.13 seq 292057776445' 2020-11-16T21:18:38.938 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.13 seq 292057776445 2020-11-16T21:18:38.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 13 2020-11-16T21:18:39.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 292057776446 -lt 292057776445 2020-11-16T21:18:39.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:39.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 14-313532612924 2020-11-16T21:18:39.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:39.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=14 2020-11-16T21:18:39.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 14-313532612924 2020-11-16T21:18:39.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:39.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=313532612924 2020-11-16T21:18:39.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.14 seq 313532612924' 2020-11-16T21:18:39.321 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.14 seq 313532612924 2020-11-16T21:18:39.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 14 2020-11-16T21:18:39.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 313532612925 -lt 313532612924 2020-11-16T21:18:39.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:39.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 15-335007449401 2020-11-16T21:18:39.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:39.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=15 2020-11-16T21:18:39.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 15-335007449401 2020-11-16T21:18:39.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:39.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=335007449401 2020-11-16T21:18:39.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.15 seq 335007449401' 2020-11-16T21:18:39.749 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.15 seq 335007449401 2020-11-16T21:18:39.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 15 2020-11-16T21:18:40.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 335007449402 -lt 335007449401 2020-11-16T21:18:40.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:40.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 16-356482285879 2020-11-16T21:18:40.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:40.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=16 2020-11-16T21:18:40.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 16-356482285879 2020-11-16T21:18:40.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:40.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=356482285879 2020-11-16T21:18:40.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.16 seq 356482285879' 2020-11-16T21:18:40.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 16 2020-11-16T21:18:40.160 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.16 seq 356482285879 2020-11-16T21:18:40.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 356482285881 -lt 356482285879 2020-11-16T21:18:40.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:40.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 17-377957122357 2020-11-16T21:18:40.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:40.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=17 2020-11-16T21:18:40.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 17-377957122357 2020-11-16T21:18:40.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:40.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=377957122357 2020-11-16T21:18:40.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.17 seq 377957122357' 2020-11-16T21:18:40.555 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.17 seq 377957122357 2020-11-16T21:18:40.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 17 2020-11-16T21:18:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 377957122359 -lt 377957122357 2020-11-16T21:18:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:40.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 18-399431958835 2020-11-16T21:18:40.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:40.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=18 2020-11-16T21:18:40.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 18-399431958835 2020-11-16T21:18:40.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:40.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=399431958835 2020-11-16T21:18:40.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.18 seq 399431958835' 2020-11-16T21:18:40.952 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.18 seq 399431958835 2020-11-16T21:18:40.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 18 2020-11-16T21:18:41.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 399431958836 -lt 399431958835 2020-11-16T21:18:41.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:18:41.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 19-420906795313 2020-11-16T21:18:41.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:18:41.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=19 2020-11-16T21:18:41.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 19-420906795313 2020-11-16T21:18:41.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:18:41.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=420906795313 2020-11-16T21:18:41.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.19 seq 420906795313' 2020-11-16T21:18:41.335 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.19 seq 420906795313 2020-11-16T21:18:41.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 19 2020-11-16T21:18:41.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 420906795315 -lt 420906795313 2020-11-16T21:18:41.724 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:18:41.724 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:18:41.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:18:42.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 8 == 0 2020-11-16T21:18:42.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:18:42.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2020-11-16T21:18:42.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2020-11-16T21:18:42.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:18:42.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2020-11-16T21:18:42.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:18:42.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2020-11-16T21:18:42.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:18:42.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:18:42.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:18:42.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:18:42.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:234: TEST_backfill_test_multi: ERRORS=0 2020-11-16T21:18:42.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:235: TEST_backfill_test_multi: wc -l 2020-11-16T21:18:42.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:235: TEST_backfill_test_multi: ceph pg dump pgs 2020-11-16T21:18:42.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:235: TEST_backfill_test_multi: grep +backfill_toofull 2020-11-16T21:18:42.913 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:18:42.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:235: TEST_backfill_test_multi: full=4 2020-11-16T21:18:42.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:236: TEST_backfill_test_multi: '[' 4 -lt 1 ']' 2020-11-16T21:18:42.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:242: TEST_backfill_test_multi: expr 8 - 4 2020-11-16T21:18:42.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:242: TEST_backfill_test_multi: expected=4 2020-11-16T21:18:42.926 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:243: TEST_backfill_test_multi: ceph pg dump pgs 2020-11-16T21:18:42.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:243: TEST_backfill_test_multi: grep active+clean 2020-11-16T21:18:42.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:243: TEST_backfill_test_multi: wc -l 2020-11-16T21:18:43.290 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:18:43.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:243: TEST_backfill_test_multi: '[' 4 '!=' 4 ']' 2020-11-16T21:18:43.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:249: TEST_backfill_test_multi: ceph pg dump pgs 2020-11-16T21:18:43.663 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:18:43.664 INFO:tasks.workunit.client.0.smithi189.stdout:8.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2020-11-16T21:18:23.636648+0000 146'600 183:1863 [3,15] 3 [3,15] 3 0'0 2020-11-16T20:54:01.389262+0000 0'0 2020-11-16T20:54:01.389262+0000 0 2020-11-16T21:18:43.664 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2020-11-16T21:18:15.110068+0000 146'600 182:675 [5,1] 5 [5] 5 0'0 2020-11-16T20:53:58.234741+0000 0'0 2020-11-16T20:53:58.234741+0000 0 2020-11-16T21:18:43.664 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2020-11-16T21:18:36.348118+0000 146'600 183:686 [17,5] 17 [17] 17 0'0 2020-11-16T20:53:44.302494+0000 0'0 2020-11-16T20:53:44.302494+0000 0 2020-11-16T21:18:43.664 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2020-11-16T21:18:15.567482+0000 146'600 182:1891 [8,0] 8 [8,0] 8 0'0 2020-11-16T20:53:41.661066+0000 0'0 2020-11-16T20:53:41.661066+0000 0 2020-11-16T21:18:43.664 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2020-11-16T21:18:38.303980+0000 146'600 183:700 [1,7] 1 [1] 1 0'0 2020-11-16T20:53:46.927735+0000 0'0 2020-11-16T20:53:46.927735+0000 0 2020-11-16T21:18:43.665 INFO:tasks.workunit.client.0.smithi189.stdout:4.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2020-11-16T21:18:19.099926+0000 146'600 182:1871 [4,10] 4 [4,10] 4 0'0 2020-11-16T20:53:49.973123+0000 0'0 2020-11-16T20:53:49.973123+0000 0 2020-11-16T21:18:43.665 INFO:tasks.workunit.client.0.smithi189.stdout:5.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2020-11-16T21:18:20.094124+0000 146'600 182:1870 [12,19] 12 [12,19] 12 0'0 2020-11-16T20:53:52.540182+0000 0'0 2020-11-16T20:53:52.540182+0000 0 2020-11-16T21:18:43.665 INFO:tasks.workunit.client.0.smithi189.stdout:6.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2020-11-16T21:18:12.440496+0000 146'600 182:676 [7,4] 7 [7] 7 0'0 2020-11-16T20:53:55.565820+0000 0'0 2020-11-16T20:53:55.565820+0000 0 2020-11-16T21:18:43.665 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:18:43.665 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:18:43.666 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:18:43.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:250: TEST_backfill_test_multi: ceph status 2020-11-16T21:18:44.160 INFO:tasks.workunit.client.0.smithi189.stdout: cluster: 2020-11-16T21:18:44.160 INFO:tasks.workunit.client.0.smithi189.stdout: id: f5b90eb9-f99a-41dc-aff8-50e583582af1 2020-11-16T21:18:44.161 INFO:tasks.workunit.client.0.smithi189.stdout: health: HEALTH_WARN 2020-11-16T21:18:44.161 INFO:tasks.workunit.client.0.smithi189.stdout: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull 2020-11-16T21:18:44.161 INFO:tasks.workunit.client.0.smithi189.stdout: Degraded data redundancy: 2400/9600 objects degraded (25.000%), 4 pgs degraded, 4 pgs undersized 2020-11-16T21:18:44.161 INFO:tasks.workunit.client.0.smithi189.stdout: 8 pool(s) do not have an application enabled 2020-11-16T21:18:44.161 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:18:44.161 INFO:tasks.workunit.client.0.smithi189.stdout: services: 2020-11-16T21:18:44.162 INFO:tasks.workunit.client.0.smithi189.stdout: mon: 1 daemons, quorum a (age 28m) 2020-11-16T21:18:44.162 INFO:tasks.workunit.client.0.smithi189.stdout: mgr: x(active, since 27m) 2020-11-16T21:18:44.162 INFO:tasks.workunit.client.0.smithi189.stdout: osd: 20 osds: 20 up (since 25m), 20 in (since 25m); 4 remapped pgs 2020-11-16T21:18:44.163 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:18:44.163 INFO:tasks.workunit.client.0.smithi189.stdout: data: 2020-11-16T21:18:44.163 INFO:tasks.workunit.client.0.smithi189.stdout: pools: 8 pools, 8 pgs 2020-11-16T21:18:44.164 INFO:tasks.workunit.client.0.smithi189.stdout: objects: 4.80k objects, 19 MiB 2020-11-16T21:18:44.164 INFO:tasks.workunit.client.0.smithi189.stdout: usage: 28 MiB used, 42 MiB / 70 MiB avail 2020-11-16T21:18:44.164 INFO:tasks.workunit.client.0.smithi189.stdout: pgs: 2400/9600 objects degraded (25.000%) 2020-11-16T21:18:44.165 INFO:tasks.workunit.client.0.smithi189.stdout: 4 active+clean 2020-11-16T21:18:44.165 INFO:tasks.workunit.client.0.smithi189.stdout: 4 active+undersized+degraded+remapped+backfill_toofull 2020-11-16T21:18:44.165 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:18:44.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:252: TEST_backfill_test_multi: ceph status --format=json-pretty 2020-11-16T21:18:44.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:254: TEST_backfill_test_multi: jq .health.checks.PG_BACKFILL_FULL.severity td/osd-backfill-space/stat.json 2020-11-16T21:18:44.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:254: TEST_backfill_test_multi: eval 'SEV="HEALTH_WARN"' 2020-11-16T21:18:44.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:254: TEST_backfill_test_multi: SEV=HEALTH_WARN 2020-11-16T21:18:44.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:255: TEST_backfill_test_multi: '[' HEALTH_WARN '!=' HEALTH_WARN ']' 2020-11-16T21:18:44.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:259: TEST_backfill_test_multi: jq .health.checks.PG_BACKFILL_FULL.summary.message td/osd-backfill-space/stat.json 2020-11-16T21:18:44.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:259: TEST_backfill_test_multi: eval 'MSG="Low' space hindering backfill '(add' storage if this 'doesn'\''t' resolve 'itself):' 4 pgs 'backfill_toofull"' 2020-11-16T21:18:44.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:259: TEST_backfill_test_multi: MSG='Low space hindering backfill (add storage if this doesn'\''t resolve itself): 4 pgs backfill_toofull' 2020-11-16T21:18:44.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:260: TEST_backfill_test_multi: '[' 'Low space hindering backfill (add storage if this doesn'\''t resolve itself): 4 pgs backfill_toofull' '!=' 'Low space hindering backfill (add storage if this doesn'\''t resolve itself): 4 pgs backfill_toofull' ']' 2020-11-16T21:18:44.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:264: TEST_backfill_test_multi: rm -f td/osd-backfill-space/stat.json 2020-11-16T21:18:44.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:266: TEST_backfill_test_multi: '[' 0 '!=' 0 ']' 2020-11-16T21:18:44.679 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: seq 1 8 2020-11-16T21:18:44.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2020-11-16T21:18:44.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test1 2020-11-16T21:18:44.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2020-11-16T21:18:44.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-11-16T21:18:45.187 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' does not exist 2020-11-16T21:18:45.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2020-11-16T21:18:45.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test2 2020-11-16T21:18:45.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2020-11-16T21:18:45.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-11-16T21:18:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' does not exist 2020-11-16T21:18:46.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2020-11-16T21:18:46.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test3 2020-11-16T21:18:46.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2020-11-16T21:18:46.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-11-16T21:18:46.630 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' does not exist 2020-11-16T21:18:46.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2020-11-16T21:18:46.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test4 2020-11-16T21:18:46.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test4 2020-11-16T21:18:46.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-11-16T21:18:47.199 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test4' does not exist 2020-11-16T21:18:47.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2020-11-16T21:18:47.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test5 2020-11-16T21:18:47.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test5 2020-11-16T21:18:47.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-11-16T21:18:47.938 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test5' does not exist 2020-11-16T21:18:47.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2020-11-16T21:18:47.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test6 2020-11-16T21:18:47.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test6 2020-11-16T21:18:47.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-11-16T21:18:48.602 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test6' does not exist 2020-11-16T21:18:48.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2020-11-16T21:18:48.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test7 2020-11-16T21:18:48.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test7 2020-11-16T21:18:48.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-11-16T21:18:49.477 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test7' does not exist 2020-11-16T21:18:49.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2020-11-16T21:18:49.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test8 2020-11-16T21:18:49.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test8 2020-11-16T21:18:49.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-11-16T21:18:50.162 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test8' does not exist 2020-11-16T21:18:50.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:276: TEST_backfill_test_multi: kill_daemons td/osd-backfill-space 2020-11-16T21:18:50.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:18:50.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:18:50.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:18:50.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:18:50.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:18:50.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:18:50.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:277: TEST_backfill_test_multi: grep -q 'num_bytes mismatch' td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.10.log td/osd-backfill-space/osd.11.log td/osd-backfill-space/osd.12.log td/osd-backfill-space/osd.13.log td/osd-backfill-space/osd.14.log td/osd-backfill-space/osd.15.log td/osd-backfill-space/osd.16.log td/osd-backfill-space/osd.17.log td/osd-backfill-space/osd.18.log td/osd-backfill-space/osd.19.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log td/osd-backfill-space/osd.3.log td/osd-backfill-space/osd.4.log td/osd-backfill-space/osd.5.log td/osd-backfill-space/osd.6.log td/osd-backfill-space/osd.7.log td/osd-backfill-space/osd.8.log td/osd-backfill-space/osd.9.log 2020-11-16T21:18:50.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2020-11-16T21:18:50.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:18:50.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:18:50.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:18:50.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:18:50.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:18:50.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:18:50.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:18:50.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:18:50.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:18:50.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:18:50.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:18:50.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:18:50.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:18:50.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:18:50.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:18:50.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:18:50.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:18:50.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:18:50.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:18:50.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:18:50.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:18:50.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:18:50.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:18:50.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:18:50.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:18:50.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:18:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:18:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:18:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2020-11-16T21:18:50.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2020-11-16T21:18:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2020-11-16T21:18:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2020-11-16T21:18:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:18:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:18:51.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:18:51.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:18:51.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:18:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:18:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:18:51.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:18:51.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:18:51.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:18:51.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:18:51.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:18:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:18:51.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:18:51.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:18:51.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:18:51.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:18:51.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:18:51.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:18:51.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:18:51.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:18:51.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:18:51.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:18:51.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:18:51.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:18:51.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:18:51.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:18:51.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:18:51.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2020-11-16T21:18:51.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:18:51.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:18:51.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:18:51.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137323 2020-11-16T21:18:51.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:18:51.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:18:51.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:18:51.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2020-11-16T21:18:51.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_backfill_test_sametarget td/osd-backfill-space 2020-11-16T21:18:51.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:294: TEST_backfill_test_sametarget: local dir=td/osd-backfill-space 2020-11-16T21:18:51.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:295: TEST_backfill_test_sametarget: local pools=10 2020-11-16T21:18:51.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:296: TEST_backfill_test_sametarget: local OSDS=5 2020-11-16T21:18:51.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:298: TEST_backfill_test_sametarget: run_mon td/osd-backfill-space a 2020-11-16T21:18:51.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2020-11-16T21:18:51.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:18:51.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:18:51.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:18:51.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2020-11-16T21:18:51.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2020-11-16T21:18:51.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:18:51.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:18:51.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:18:51.328 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:18:51.328 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:18:51.328 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:18:51.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:18:51.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:18:51.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:18:51.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:18:51.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:18:51.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:18:51.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:18:51.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:18:51.535 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:18:51.535 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:18:51.535 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:18:51.536 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:18:51.536 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:18:51.536 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:18:51.536 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:18:51.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:18:51.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get fsid 2020-11-16T21:18:51.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:18:51.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:18:51.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:18:51.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:18:51.651 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:18:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:18:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:18:51.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:18:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:18:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:18:51.652 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:18:51.653 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:18:51.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:18:51.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get mon_host 2020-11-16T21:18:51.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:299: TEST_backfill_test_sametarget: run_mgr td/osd-backfill-space x 2020-11-16T21:18:51.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2020-11-16T21:18:51.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:18:51.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:18:51.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:18:51.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2020-11-16T21:18:51.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:18:52.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:18:52.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:18:52.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:18:52.268 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:18:52.269 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:18:52.269 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:18:52.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:18:52.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:18:52.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:18:52.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:300: TEST_backfill_test_sametarget: export CEPH_ARGS 2020-11-16T21:18:52.304 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: expr 5 - 1 2020-11-16T21:18:52.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: seq 0 4 2020-11-16T21:18:52.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:18:52.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:304: TEST_backfill_test_sametarget: run_osd td/osd-backfill-space 0 2020-11-16T21:18:52.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:18:52.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:18:52.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:18:52.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:18:52.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2020-11-16T21:18:52.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:18:52.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:18:52.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:18:52.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:18:52.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2020-11-16T21:18:52.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2020-11-16T21:18:52.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:18:52.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:18:52.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:18:52.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:18:52.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:18:52.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:18:52.312 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:18:52.312 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:18:52.312 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:18:52.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:18:52.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:18:52.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:18:52.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:18:52.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:18:52.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:18:52.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:18:52.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:18:52.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:18:52.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:18:52.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:18:52.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:18:52.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2020-11-16T21:18:52.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:18:52.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=66e039ca-a305-4a0a-a815-6d27011b9e48 2020-11-16T21:18:52.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 66e039ca-a305-4a0a-a815-6d27011b9e48' 2020-11-16T21:18:52.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:18:52.315 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 66e039ca-a305-4a0a-a815-6d27011b9e48 2020-11-16T21:18:52.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC87LJf4rlsFBAA+yooue4OttMAEYZovjp0mA== 2020-11-16T21:18:52.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC87LJf4rlsFBAA+yooue4OttMAEYZovjp0mA=="}' 2020-11-16T21:18:52.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 66e039ca-a305-4a0a-a815-6d27011b9e48 -i td/osd-backfill-space/0/new.json 2020-11-16T21:18:52.677 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:18:52.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2020-11-16T21:18:52.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC87LJf4rlsFBAA+yooue4OttMAEYZovjp0mA== --osd-uuid 66e039ca-a305-4a0a-a815-6d27011b9e48 2020-11-16T21:18:52.724 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:52.730+0000 7f7615026d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:18:52.727 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:52.730+0000 7f7615026d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:18:52.729 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:52.734+0000 7f7615026d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:18:52.730 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:52.734+0000 7f7615026d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:18:52.730 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:52.734+0000 7f7615026d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:18:52.730 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:52.734+0000 7f7615026d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:18:52.764 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:52.770+0000 7f7615026d80 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2020-11-16T21:18:57.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2020-11-16T21:18:57.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:18:57.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:18:57.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:18:57.368 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:18:57.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:18:57.877 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:18:57.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:18:57.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:18:57.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:18:57.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:18:57.904 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:57.910+0000 7f85e7831d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:18:57.907 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:57.910+0000 7f85e7831d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:18:57.908 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:57.914+0000 7f85e7831d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:18:58.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:18:58.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:18:58.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:18:58.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:18:58.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:18:58.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:18:58.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:18:58.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:18:58.278 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:18:58.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:18:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:58.458+0000 7f85e7831d80 -1 Falling back to public interface 2020-11-16T21:18:58.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:18:59.300 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:18:59.298+0000 7f85e7831d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:18:59.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:18:59.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:18:59.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:18:59.647 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:18:59.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:18:59.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:19:00.090 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/350283,v1:127.0.0.1:6803/350283] [v2:127.0.0.1:6804/350283,v1:127.0.0.1:6805/350283] exists,up 66e039ca-a305-4a0a-a815-6d27011b9e48 2020-11-16T21:19:00.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:19:00.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:19:00.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:19:00.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:19:00.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:304: TEST_backfill_test_sametarget: run_osd td/osd-backfill-space 1 2020-11-16T21:19:00.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:19:00.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:19:00.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:19:00.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:19:00.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2020-11-16T21:19:00.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:19:00.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:19:00.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:19:00.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:19:00.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2020-11-16T21:19:00.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2020-11-16T21:19:00.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:19:00.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:19:00.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:19:00.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:19:00.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:19:00.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:19:00.094 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:19:00.095 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:19:00.095 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:19:00.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:19:00.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:19:00.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:19:00.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:19:00.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:19:00.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:19:00.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:19:00.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:19:00.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:19:00.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:19:00.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:19:00.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:19:00.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2020-11-16T21:19:00.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:19:00.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=277697ce-4974-4db0-8de7-644a6b3efe27 2020-11-16T21:19:00.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 277697ce-4974-4db0-8de7-644a6b3efe27' 2020-11-16T21:19:00.098 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:19:00.098 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 277697ce-4974-4db0-8de7-644a6b3efe27 2020-11-16T21:19:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDE7LJf4dNdBxAASJ9J8xd0ekIqhtozIKq9EA== 2020-11-16T21:19:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDE7LJf4dNdBxAASJ9J8xd0ekIqhtozIKq9EA=="}' 2020-11-16T21:19:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 277697ce-4974-4db0-8de7-644a6b3efe27 -i td/osd-backfill-space/1/new.json 2020-11-16T21:19:00.685 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:19:00.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2020-11-16T21:19:00.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDE7LJf4dNdBxAASJ9J8xd0ekIqhtozIKq9EA== --osd-uuid 277697ce-4974-4db0-8de7-644a6b3efe27 2020-11-16T21:19:00.734 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:00.738+0000 7f049dd44d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:00.737 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:00.742+0000 7f049dd44d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:00.738 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:00.742+0000 7f049dd44d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:00.738 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:00.742+0000 7f049dd44d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:19:00.739 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:00.742+0000 7f049dd44d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:19:00.739 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:00.742+0000 7f049dd44d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:19:00.784 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:00.790+0000 7f049dd44d80 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2020-11-16T21:19:05.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2020-11-16T21:19:05.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:19:05.239 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:19:05.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:19:05.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:19:05.742 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:19:05.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:19:05.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:19:05.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:19:05.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:19:05.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:19:05.774 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:05.778+0000 7f6075e99d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:05.777 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:05.782+0000 7f6075e99d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:05.779 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:05.782+0000 7f6075e99d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:06.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:19:06.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:19:06.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:19:06.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:19:06.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:19:06.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:19:06.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:19:06.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:19:06.136 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:19:06.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:19:06.322 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:06.326+0000 7f6075e99d80 -1 Falling back to public interface 2020-11-16T21:19:06.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:19:07.158 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:07.162+0000 7f6075e99d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:19:07.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:19:07.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:19:07.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:19:07.512 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:19:07.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:19:07.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:19:07.887 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/351277,v1:127.0.0.1:6811/351277] [v2:127.0.0.1:6812/351277,v1:127.0.0.1:6813/351277] exists,up 277697ce-4974-4db0-8de7-644a6b3efe27 2020-11-16T21:19:07.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:19:07.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:19:07.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:19:07.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:19:07.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:304: TEST_backfill_test_sametarget: run_osd td/osd-backfill-space 2 2020-11-16T21:19:07.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:19:07.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:19:07.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:19:07.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:19:07.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2020-11-16T21:19:07.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:19:07.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:19:07.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:19:07.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:19:07.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2020-11-16T21:19:07.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2020-11-16T21:19:07.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:19:07.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:19:07.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:19:07.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:19:07.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:19:07.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:19:07.893 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:19:07.893 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:19:07.894 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:19:07.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:19:07.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:19:07.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:19:07.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:19:07.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:19:07.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:19:07.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:19:07.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:19:07.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:19:07.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:19:07.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:19:07.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:19:07.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2020-11-16T21:19:07.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:19:07.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e219be78-8fa7-4fda-b3b4-cce6bfbf4768 2020-11-16T21:19:07.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 e219be78-8fa7-4fda-b3b4-cce6bfbf4768' 2020-11-16T21:19:07.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:19:07.897 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 e219be78-8fa7-4fda-b3b4-cce6bfbf4768 2020-11-16T21:19:07.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDL7LJf8xkyNxAA8NrT1kwos7ozYh/Lhu9bIQ== 2020-11-16T21:19:07.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDL7LJf8xkyNxAA8NrT1kwos7ozYh/Lhu9bIQ=="}' 2020-11-16T21:19:07.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e219be78-8fa7-4fda-b3b4-cce6bfbf4768 -i td/osd-backfill-space/2/new.json 2020-11-16T21:19:08.568 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:19:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2020-11-16T21:19:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDL7LJf8xkyNxAA8NrT1kwos7ozYh/Lhu9bIQ== --osd-uuid e219be78-8fa7-4fda-b3b4-cce6bfbf4768 2020-11-16T21:19:08.618 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:08.622+0000 7f59e11c7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:08.620 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:08.626+0000 7f59e11c7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:08.622 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:08.626+0000 7f59e11c7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:08.623 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:08.626+0000 7f59e11c7d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:19:08.624 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:08.626+0000 7f59e11c7d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:19:08.624 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:08.630+0000 7f59e11c7d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:19:08.669 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:08.674+0000 7f59e11c7d80 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2020-11-16T21:19:13.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2020-11-16T21:19:13.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:19:13.067 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:19:13.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:19:13.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:19:13.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:19:13.550 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:19:13.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:19:13.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:19:13.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:19:13.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:19:13.583 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:13.586+0000 7f0683fbdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:13.585 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:13.590+0000 7f0683fbdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:13.587 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:13.590+0000 7f0683fbdd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:19:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:19:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:19:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:19:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:19:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:19:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:19:13.935 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:19:13.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:19:13.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:19:14.121 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:14.126+0000 7f0683fbdd80 -1 Falling back to public interface 2020-11-16T21:19:14.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:19:15.008 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:15.014+0000 7f0683fbdd80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:19:15.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:19:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:19:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:19:15.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:19:15.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:19:15.347 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:19:15.739 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/352271,v1:127.0.0.1:6819/352271] [v2:127.0.0.1:6820/352271,v1:127.0.0.1:6821/352271] exists,up e219be78-8fa7-4fda-b3b4-cce6bfbf4768 2020-11-16T21:19:15.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:19:15.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:19:15.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:19:15.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:19:15.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:304: TEST_backfill_test_sametarget: run_osd td/osd-backfill-space 3 2020-11-16T21:19:15.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:19:15.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:19:15.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:19:15.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:19:15.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/3 2020-11-16T21:19:15.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:19:15.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:19:15.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:19:15.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:19:15.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/3' 2020-11-16T21:19:15.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/3/journal' 2020-11-16T21:19:15.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:19:15.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:19:15.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:19:15.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:19:15.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:19:15.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:19:15.745 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:19:15.745 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:19:15.745 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:19:15.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:19:15.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:19:15.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:19:15.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:19:15.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:19:15.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:19:15.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:19:15.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:19:15.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:19:15.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:19:15.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:19:15.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:19:15.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/3 2020-11-16T21:19:15.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:19:15.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=bc2c3639-063f-4b72-aeee-c134957e25cb 2020-11-16T21:19:15.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 bc2c3639-063f-4b72-aeee-c134957e25cb' 2020-11-16T21:19:15.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:19:15.748 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 bc2c3639-063f-4b72-aeee-c134957e25cb 2020-11-16T21:19:15.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDT7LJfl7PALRAAByqPXK7htNIHmWbKpn54nQ== 2020-11-16T21:19:15.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDT7LJfl7PALRAAByqPXK7htNIHmWbKpn54nQ=="}' 2020-11-16T21:19:15.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new bc2c3639-063f-4b72-aeee-c134957e25cb -i td/osd-backfill-space/3/new.json 2020-11-16T21:19:16.517 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:19:16.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/3/new.json 2020-11-16T21:19:16.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDT7LJfl7PALRAAByqPXK7htNIHmWbKpn54nQ== --osd-uuid bc2c3639-063f-4b72-aeee-c134957e25cb 2020-11-16T21:19:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:16.566+0000 7feab5abcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:16.565 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:16.570+0000 7feab5abcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:16.567 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:16.570+0000 7feab5abcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:16.574+0000 7feab5abcd80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:19:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:16.574+0000 7feab5abcd80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:19:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:16.574+0000 7feab5abcd80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:19:16.613 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:16.618+0000 7feab5abcd80 -1 bluestore(td/osd-backfill-space/3) _read_fsid unparsable uuid 2020-11-16T21:19:21.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/3/keyring 2020-11-16T21:19:25.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:19:25.720 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:19:25.720 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:19:25.720 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:19:25.721 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:19:25.721 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/353265,v1:127.0.0.1:6827/353265] [v2:127.0.0.1:6828/353265,v1:127.0.0.1:6829/353265] exists,up bc2c3639-063f-4b72-aeee-c134957e25cb 2020-11-16T21:19:25.721 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 04c2be1b-847f-42cb-ac9b-d10101e280f6 2020-11-16T21:19:25.721 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:19:25.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:19:25.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:19:25.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:19:25.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:19:25.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:19:25.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:19:25.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:19:25.722 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:22.014+0000 7fb89b213d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:25.723 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:22.018+0000 7fb89b213d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:25.723 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:22.018+0000 7fb89b213d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:25.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:19:25.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:19:25.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:19:25.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:19:25.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:19:25.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:19:25.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:19:25.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:19:25.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:19:25.724 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:22.558+0000 7fb89b213d80 -1 Falling back to public interface 2020-11-16T21:19:25.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:19:25.724 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:23.410+0000 7fb89b213d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:19:25.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:19:25.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:19:25.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:19:25.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:19:25.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:19:25.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:19:25.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:19:25.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:19:25.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:19:25.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:304: TEST_backfill_test_sametarget: run_osd td/osd-backfill-space 4 2020-11-16T21:19:25.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:19:25.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:19:25.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:19:25.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:19:25.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/4 2020-11-16T21:19:25.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:19:25.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:19:25.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:19:25.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:19:25.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/4' 2020-11-16T21:19:25.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/4/journal' 2020-11-16T21:19:25.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:19:25.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:19:25.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:19:25.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:19:25.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:19:25.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:19:25.728 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:19:25.728 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:19:25.728 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:19:25.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:19:25.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:19:25.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:19:25.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:19:25.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:19:25.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:19:25.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:19:25.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:19:25.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:19:25.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:19:25.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:19:25.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:19:25.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/4 2020-11-16T21:19:25.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:19:25.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=04c2be1b-847f-42cb-ac9b-d10101e280f6 2020-11-16T21:19:25.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 04c2be1b-847f-42cb-ac9b-d10101e280f6' 2020-11-16T21:19:25.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:19:25.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDc7LJfow+PChAAvoFAyG9vY9+WvXMgV+HIhQ== 2020-11-16T21:19:25.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDc7LJfow+PChAAvoFAyG9vY9+WvXMgV+HIhQ=="}' 2020-11-16T21:19:25.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 04c2be1b-847f-42cb-ac9b-d10101e280f6 -i td/osd-backfill-space/4/new.json 2020-11-16T21:19:25.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/4/new.json 2020-11-16T21:19:25.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDc7LJfow+PChAAvoFAyG9vY9+WvXMgV+HIhQ== --osd-uuid 04c2be1b-847f-42cb-ac9b-d10101e280f6 2020-11-16T21:19:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:24.810+0000 7f6438e23d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:24.814+0000 7f6438e23d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:24.814+0000 7f6438e23d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:24.814+0000 7f6438e23d80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:19:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:24.814+0000 7f6438e23d80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:19:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:24.814+0000 7f6438e23d80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:19:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:24.894+0000 7f6438e23d80 -1 bluestore(td/osd-backfill-space/4) _read_fsid unparsable uuid 2020-11-16T21:19:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/4/keyring 2020-11-16T21:19:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:19:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:19:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:19:29.216 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:19:29.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:19:29.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:19:29.729 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:19:29.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:19:29.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:19:29.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:19:29.753 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:29.757+0000 7f85b1138d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:29.756 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:29.761+0000 7f85b1138d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:29.758 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:29.761+0000 7f85b1138d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:19:30.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:19:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:19:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:19:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:19:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:19:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:19:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:19:30.098 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:19:30.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:19:30.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:19:30.302 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:30.305+0000 7f85b1138d80 -1 Falling back to public interface 2020-11-16T21:19:30.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:19:31.192 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:19:31.189+0000 7f85b1138d80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:19:31.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:19:31.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:19:31.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:19:31.512 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:19:31.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:19:31.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:19:31.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:19:32.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:19:32.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:19:32.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:19:32.905 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:19:32.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:19:32.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:19:33.316 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/354260,v1:127.0.0.1:6835/354260] [v2:127.0.0.1:6836/354260,v1:127.0.0.1:6837/354260] exists,up 04c2be1b-847f-42cb-ac9b-d10101e280f6 2020-11-16T21:19:33.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:19:33.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:19:33.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:19:33.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:307: TEST_backfill_test_sametarget: ceph osd set-backfillfull-ratio .85 2020-11-16T21:19:34.105 INFO:tasks.workunit.client.0.smithi189.stderr:osd set-backfillfull-ratio 0.85 2020-11-16T21:19:34.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: seq 1 10 2020-11-16T21:19:34.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:19:34.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test1 1 1 2020-11-16T21:19:34.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2020-11-16T21:19:34.687 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' already exists 2020-11-16T21:19:34.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:19:35.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test1 size 2 2020-11-16T21:19:36.493 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T21:19:36.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:19:36.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test2 1 1 2020-11-16T21:19:36.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2020-11-16T21:19:37.178 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' already exists 2020-11-16T21:19:37.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:19:38.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test2 size 2 2020-11-16T21:19:39.307 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 2 2020-11-16T21:19:39.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:19:39.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test3 1 1 2020-11-16T21:19:39.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2020-11-16T21:19:39.957 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' already exists 2020-11-16T21:19:39.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:19:40.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test3 size 2 2020-11-16T21:19:41.969 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 2 2020-11-16T21:19:41.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:19:41.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test4 1 1 2020-11-16T21:19:41.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test4 1 1 2020-11-16T21:19:42.615 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test4' already exists 2020-11-16T21:19:42.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:19:43.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test4 size 2 2020-11-16T21:19:44.440 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 2 2020-11-16T21:19:44.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:19:44.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test5 1 1 2020-11-16T21:19:44.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test5 1 1 2020-11-16T21:19:45.099 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test5' already exists 2020-11-16T21:19:45.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:19:46.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test5 size 2 2020-11-16T21:19:46.856 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 5 size to 2 2020-11-16T21:19:46.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:19:46.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test6 1 1 2020-11-16T21:19:46.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test6 1 1 2020-11-16T21:19:47.470 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test6' already exists 2020-11-16T21:19:47.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:19:48.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test6 size 2 2020-11-16T21:19:49.187 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 6 size to 2 2020-11-16T21:19:49.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:19:49.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test7 1 1 2020-11-16T21:19:49.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test7 1 1 2020-11-16T21:19:49.948 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test7' already exists 2020-11-16T21:19:49.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:19:50.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test7 size 2 2020-11-16T21:19:51.759 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 7 size to 2 2020-11-16T21:19:51.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:19:51.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test8 1 1 2020-11-16T21:19:51.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test8 1 1 2020-11-16T21:19:52.314 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test8' already exists 2020-11-16T21:19:52.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:19:53.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test8 size 2 2020-11-16T21:19:54.159 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 8 size to 2 2020-11-16T21:19:54.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:19:54.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test9 1 1 2020-11-16T21:19:54.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test9 1 1 2020-11-16T21:19:54.850 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test9' already exists 2020-11-16T21:19:54.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:19:55.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test9 size 2 2020-11-16T21:19:56.884 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 9 size to 2 2020-11-16T21:19:56.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:19:56.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test10 1 1 2020-11-16T21:19:56.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test10 1 1 2020-11-16T21:19:57.464 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test10' already exists 2020-11-16T21:19:57.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:19:58.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test10 size 2 2020-11-16T21:19:59.252 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 10 size to 2 2020-11-16T21:19:59.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:314: TEST_backfill_test_sametarget: sleep 5 2020-11-16T21:20:04.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:316: TEST_backfill_test_sametarget: wait_for_clean 2020-11-16T21:20:04.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:20:04.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:20:04.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:20:04.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:20:04.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:20:04.268 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:20:04.268 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:20:04.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:20:04.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:20:04.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:20:04.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:20:04.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:20:04.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:20:04.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:20:04.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:20:04.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:20:04.910 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:20:04.911 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:20:04.911 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:20:04.911 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T21:20:04.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:20:04.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:20:04.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:20:05.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836495 2020-11-16T21:20:05.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836495 2020-11-16T21:20:05.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495' 2020-11-16T21:20:05.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:20:05.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:20:05.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672975 2020-11-16T21:20:05.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672975 2020-11-16T21:20:05.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672975' 2020-11-16T21:20:05.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:20:05.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:20:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2020-11-16T21:20:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2020-11-16T21:20:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509452' 2020-11-16T21:20:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:20:05.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:20:05.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345931 2020-11-16T21:20:05.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345931 2020-11-16T21:20:05.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509452 3-85899345931' 2020-11-16T21:20:05.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:20:05.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:20:05.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182409 2020-11-16T21:20:05.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182409 2020-11-16T21:20:05.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-64424509452 3-85899345931 4-107374182409' 2020-11-16T21:20:05.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:20:05.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2020-11-16T21:20:05.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:20:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:20:05.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2020-11-16T21:20:05.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:20:05.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836495 2020-11-16T21:20:05.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2020-11-16T21:20:05.797 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836495 2020-11-16T21:20:05.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:20:06.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836495 2020-11-16T21:20:06.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:20:06.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672975 2020-11-16T21:20:06.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:20:06.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:20:06.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672975 2020-11-16T21:20:06.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:20:06.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672975 2020-11-16T21:20:06.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2020-11-16T21:20:06.215 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672975 2020-11-16T21:20:06.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:20:06.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672975 -lt 42949672975 2020-11-16T21:20:06.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:20:06.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2020-11-16T21:20:06.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:20:06.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:20:06.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2020-11-16T21:20:06.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:20:06.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2020-11-16T21:20:06.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2020-11-16T21:20:06.643 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509452 2020-11-16T21:20:06.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:20:07.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509452 -lt 64424509452 2020-11-16T21:20:07.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:20:07.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345931 2020-11-16T21:20:07.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:20:07.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:20:07.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345931 2020-11-16T21:20:07.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:20:07.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345931 2020-11-16T21:20:07.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2020-11-16T21:20:07.039 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 85899345931 2020-11-16T21:20:07.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:20:07.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345931 -lt 85899345931 2020-11-16T21:20:07.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:20:07.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182409 2020-11-16T21:20:07.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:20:07.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:20:07.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182409 2020-11-16T21:20:07.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:20:07.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182409 2020-11-16T21:20:07.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2020-11-16T21:20:07.412 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 107374182409 2020-11-16T21:20:07.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:20:07.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182409 -lt 107374182409 2020-11-16T21:20:07.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:20:07.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:20:07.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:20:08.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 10 == 0 2020-11-16T21:20:08.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:20:08.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:20:08.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:20:08.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:20:08.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:20:08.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:20:08.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:20:08.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=10 2020-11-16T21:20:08.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:20:08.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:20:08.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:20:09.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 10 = 10 2020-11-16T21:20:09.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:20:09.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:20:09.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:318: TEST_backfill_test_sametarget: ceph pg dump pgs 2020-11-16T21:20:09.510 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:20:09.511 INFO:tasks.workunit.client.0.smithi189.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:19:59.851695+0000 0'0 81:19 [2,1] 2 [2,1] 2 0'0 2020-11-16T21:19:57.292245+0000 0'0 2020-11-16T21:19:57.292245+0000 0 2020-11-16T21:20:09.511 INFO:tasks.workunit.client.0.smithi189.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:19:57.605054+0000 0'0 80:25 [1,2] 1 [1,2] 1 0'0 2020-11-16T21:19:54.644228+0000 0'0 2020-11-16T21:19:54.644228+0000 0 2020-11-16T21:20:09.511 INFO:tasks.workunit.client.0.smithi189.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:19:55.034093+0000 0'0 78:26 [3,4] 3 [3,4] 3 0'0 2020-11-16T21:19:52.134981+0000 0'0 2020-11-16T21:19:52.134981+0000 0 2020-11-16T21:20:09.511 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:19:52.546987+0000 0'0 80:32 [1,4] 1 [1,4] 1 0'0 2020-11-16T21:19:49.563130+0000 0'0 2020-11-16T21:19:49.563130+0000 0 2020-11-16T21:20:13.626 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:19:40.514990+0000 0'0 78:50 [3,1] 3 [3,1] 3 0'0 2020-11-16T21:19:36.913131+0000 0'0 2020-11-16T21:19:36.913131+0000 0 2020-11-16T21:20:13.626 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:19:37.741804+0000 0'0 80:47 [1,0] 1 [1,0] 1 0'0 2020-11-16T21:19:34.487409+0000 0'0 2020-11-16T21:19:34.487409+0000 0 2020-11-16T21:20:13.627 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:19:42.705088+0000 0'0 80:41 [1,2] 1 [1,2] 1 0'0 2020-11-16T21:19:39.774259+0000 0'0 2020-11-16T21:19:39.774259+0000 0 2020-11-16T21:20:13.627 INFO:tasks.workunit.client.0.smithi189.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:19:44.675362+0000 0'0 79:44 [4,0] 4 [4,0] 4 0'0 2020-11-16T21:19:42.383233+0000 0'0 2020-11-16T21:19:42.383233+0000 0 2020-11-16T21:20:13.627 INFO:tasks.workunit.client.0.smithi189.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:19:47.523296+0000 0'0 78:38 [3,4] 3 [3,4] 3 0'0 2020-11-16T21:19:44.891365+0000 0'0 2020-11-16T21:19:44.891365+0000 0 2020-11-16T21:20:13.627 INFO:tasks.workunit.client.0.smithi189.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:19:50.286044+0000 0'0 79:33 [4,2] 4 [4,2] 4 0'0 2020-11-16T21:19:47.241529+0000 0'0 2020-11-16T21:19:47.241529+0000 0 2020-11-16T21:20:13.627 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:20:13.627 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:20:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:20:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:322: TEST_backfill_test_sametarget: local PG1 2020-11-16T21:20:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:323: TEST_backfill_test_sametarget: local POOLNUM1 2020-11-16T21:20:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:324: TEST_backfill_test_sametarget: local pool1 2020-11-16T21:20:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:325: TEST_backfill_test_sametarget: local chk_osd1 2020-11-16T21:20:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:326: TEST_backfill_test_sametarget: local chk_osd2 2020-11-16T21:20:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:328: TEST_backfill_test_sametarget: local PG2 2020-11-16T21:20:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:329: TEST_backfill_test_sametarget: local POOLNUM2 2020-11-16T21:20:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:330: TEST_backfill_test_sametarget: local pool2 2020-11-16T21:20:13.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:331: TEST_backfill_test_sametarget: seq 1 10 2020-11-16T21:20:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:331: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: ceph pg map 1.0 --format=json 2020-11-16T21:20:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: jq '.acting[]' 2020-11-16T21:20:13.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: head -1 td/osd-backfill-space/acting 2020-11-16T21:20:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: local test_osd1=1 2020-11-16T21:20:13.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: tail -1 td/osd-backfill-space/acting 2020-11-16T21:20:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: local test_osd2=0 2020-11-16T21:20:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:336: TEST_backfill_test_sametarget: '[' 1 = 1 ']' 2020-11-16T21:20:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:338: TEST_backfill_test_sametarget: PG1=1.0 2020-11-16T21:20:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:339: TEST_backfill_test_sametarget: POOLNUM1=1 2020-11-16T21:20:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:340: TEST_backfill_test_sametarget: pool1=test1 2020-11-16T21:20:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:341: TEST_backfill_test_sametarget: chk_osd1=1 2020-11-16T21:20:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:342: TEST_backfill_test_sametarget: chk_osd2=0 2020-11-16T21:20:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:331: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: jq '.acting[]' 2020-11-16T21:20:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: ceph pg map 2.0 --format=json 2020-11-16T21:20:13.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: head -1 td/osd-backfill-space/acting 2020-11-16T21:20:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: local test_osd1=3 2020-11-16T21:20:13.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: tail -1 td/osd-backfill-space/acting 2020-11-16T21:20:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: local test_osd2=1 2020-11-16T21:20:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:336: TEST_backfill_test_sametarget: '[' 2 = 1 ']' 2020-11-16T21:20:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:343: TEST_backfill_test_sametarget: '[' 1 '!=' 3 -a 0 = 1 ']' 2020-11-16T21:20:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:331: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: ceph pg map 3.0 --format=json 2020-11-16T21:20:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: jq '.acting[]' 2020-11-16T21:20:13.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: head -1 td/osd-backfill-space/acting 2020-11-16T21:20:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: local test_osd1=1 2020-11-16T21:20:13.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: tail -1 td/osd-backfill-space/acting 2020-11-16T21:20:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: local test_osd2=2 2020-11-16T21:20:13.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:336: TEST_backfill_test_sametarget: '[' 3 = 1 ']' 2020-11-16T21:20:13.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:343: TEST_backfill_test_sametarget: '[' 1 '!=' 1 -a 0 = 2 ']' 2020-11-16T21:20:13.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:331: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: jq '.acting[]' 2020-11-16T21:20:13.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: ceph pg map 4.0 --format=json 2020-11-16T21:20:13.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: head -1 td/osd-backfill-space/acting 2020-11-16T21:20:13.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: local test_osd1=4 2020-11-16T21:20:13.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: tail -1 td/osd-backfill-space/acting 2020-11-16T21:20:13.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: local test_osd2=0 2020-11-16T21:20:13.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:336: TEST_backfill_test_sametarget: '[' 4 = 1 ']' 2020-11-16T21:20:13.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:343: TEST_backfill_test_sametarget: '[' 1 '!=' 4 -a 0 = 0 ']' 2020-11-16T21:20:13.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:345: TEST_backfill_test_sametarget: PG2=4.0 2020-11-16T21:20:13.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:346: TEST_backfill_test_sametarget: POOLNUM2=4 2020-11-16T21:20:13.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:347: TEST_backfill_test_sametarget: pool2=test4 2020-11-16T21:20:13.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:348: TEST_backfill_test_sametarget: break 2020-11-16T21:20:13.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:351: TEST_backfill_test_sametarget: rm -f td/osd-backfill-space/acting 2020-11-16T21:20:13.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:353: TEST_backfill_test_sametarget: '[' test4 = '' ']' 2020-11-16T21:20:13.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: seq 1 10 2020-11-16T21:20:13.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 1 '!=' 1 -a 1 '!=' 4 ']' 2020-11-16T21:20:13.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 2 '!=' 1 -a 2 '!=' 4 ']' 2020-11-16T21:20:13.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test2 2020-11-16T21:20:13.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2020-11-16T21:20:13.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-11-16T21:20:13.644 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' does not exist 2020-11-16T21:20:13.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 3 '!=' 1 -a 3 '!=' 4 ']' 2020-11-16T21:20:13.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test3 2020-11-16T21:20:13.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2020-11-16T21:20:13.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-11-16T21:20:13.645 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' does not exist 2020-11-16T21:20:13.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 4 '!=' 1 -a 4 '!=' 4 ']' 2020-11-16T21:20:13.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 5 '!=' 1 -a 5 '!=' 4 ']' 2020-11-16T21:20:13.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test5 2020-11-16T21:20:13.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test5 2020-11-16T21:20:13.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-11-16T21:20:13.646 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test5' does not exist 2020-11-16T21:20:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 6 '!=' 1 -a 6 '!=' 4 ']' 2020-11-16T21:20:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test6 2020-11-16T21:20:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test6 2020-11-16T21:20:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-11-16T21:20:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test6' does not exist 2020-11-16T21:20:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 7 '!=' 1 -a 7 '!=' 4 ']' 2020-11-16T21:20:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test7 2020-11-16T21:20:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test7 2020-11-16T21:20:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-11-16T21:20:13.732 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test7' does not exist 2020-11-16T21:20:13.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:13.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 8 '!=' 1 -a 8 '!=' 4 ']' 2020-11-16T21:20:13.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test8 2020-11-16T21:20:13.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test8 2020-11-16T21:20:13.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-11-16T21:20:14.292 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test8' does not exist 2020-11-16T21:20:14.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:14.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 9 '!=' 1 -a 9 '!=' 4 ']' 2020-11-16T21:20:14.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test9 2020-11-16T21:20:14.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test9 2020-11-16T21:20:14.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-11-16T21:20:14.835 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test9' does not exist 2020-11-16T21:20:14.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2020-11-16T21:20:14.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 10 '!=' 1 -a 10 '!=' 4 ']' 2020-11-16T21:20:14.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test10 2020-11-16T21:20:14.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test10 2020-11-16T21:20:14.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-11-16T21:20:15.358 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test10' does not exist 2020-11-16T21:20:15.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:367: TEST_backfill_test_sametarget: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2020-11-16T21:20:16.079 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 1 2020-11-16T21:20:16.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:368: TEST_backfill_test_sametarget: ceph osd pool set test4 size 1 --yes-i-really-mean-it 2020-11-16T21:20:17.143 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 1 2020-11-16T21:20:17.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:370: TEST_backfill_test_sametarget: wait_for_clean 2020-11-16T21:20:17.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:20:17.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:20:17.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:20:17.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:20:17.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:20:17.159 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:20:17.160 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:20:17.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:20:17.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:20:17.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:20:17.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:20:17.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:20:17.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:20:17.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:20:17.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:20:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:20:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:20:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:20:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:20:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T21:20:17.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:20:17.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:20:17.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:20:17.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836500 2020-11-16T21:20:17.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836500 2020-11-16T21:20:17.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500' 2020-11-16T21:20:17.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:20:17.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:20:18.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672979 2020-11-16T21:20:18.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672979 2020-11-16T21:20:18.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979' 2020-11-16T21:20:18.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:20:18.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:20:18.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509457 2020-11-16T21:20:18.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509457 2020-11-16T21:20:18.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457' 2020-11-16T21:20:18.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:20:18.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:20:18.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345935 2020-11-16T21:20:18.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345935 2020-11-16T21:20:18.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457 3-85899345935' 2020-11-16T21:20:18.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:20:18.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:20:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182414 2020-11-16T21:20:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182414 2020-11-16T21:20:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457 3-85899345935 4-107374182414' 2020-11-16T21:20:18.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:20:18.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836500 2020-11-16T21:20:18.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:20:18.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:20:18.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:20:18.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836500 2020-11-16T21:20:18.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836500 2020-11-16T21:20:18.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2020-11-16T21:20:18.642 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836500 2020-11-16T21:20:18.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:20:19.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836500 2020-11-16T21:20:19.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:20:19.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672979 2020-11-16T21:20:19.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:20:19.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:20:19.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672979 2020-11-16T21:20:19.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:20:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672979 2020-11-16T21:20:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2020-11-16T21:20:19.039 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672979 2020-11-16T21:20:19.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:20:19.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672977 -lt 42949672979 2020-11-16T21:20:23.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:20:23.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:20:23.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:20:23.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672980 -lt 42949672979 2020-11-16T21:20:23.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:20:23.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509457 2020-11-16T21:20:23.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:20:23.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:20:23.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509457 2020-11-16T21:20:23.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:20:23.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509457 2020-11-16T21:20:23.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2020-11-16T21:20:23.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:20:23.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509457 -lt 64424509457 2020-11-16T21:20:23.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:20:23.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345935 2020-11-16T21:20:23.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:20:23.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:20:23.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345935 2020-11-16T21:20:23.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:20:23.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345935 2020-11-16T21:20:23.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345935' 2020-11-16T21:20:23.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:20:23.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345936 -lt 85899345935 2020-11-16T21:20:23.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:20:23.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182414 2020-11-16T21:20:23.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:20:23.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:20:23.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182414 2020-11-16T21:20:23.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:20:23.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182414 2020-11-16T21:20:23.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2020-11-16T21:20:23.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:20:23.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182414 -lt 107374182414 2020-11-16T21:20:23.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:20:23.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:20:23.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:20:23.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2020-11-16T21:20:23.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:20:23.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:20:23.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:20:23.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:20:23.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:20:23.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:20:23.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:20:23.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T21:20:23.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:20:23.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:20:23.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:20:23.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2020-11-16T21:20:23.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:20:23.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:20:23.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:372: TEST_backfill_test_sametarget: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=4 2020-11-16T21:20:23.666 INFO:tasks.workunit.client.0.smithi189.stderr:4+0 records in 2020-11-16T21:20:23.666 INFO:tasks.workunit.client.0.smithi189.stderr:4+0 records out 2020-11-16T21:20:23.667 INFO:tasks.workunit.client.0.smithi189.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000233833 s, 17.5 MB/s 2020-11-16T21:20:23.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: seq 1 600 2020-11-16T21:20:23.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:23.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:20:23.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:20:23.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:23.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:20:23.668 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509457 2020-11-16T21:20:23.668 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 85899345935 2020-11-16T21:20:23.668 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 107374182414 2020-11-16T21:20:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:20:23.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:23.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:20:23.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:20:23.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:23.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:20:24.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:20:24.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:24.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:20:24.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:20:24.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:24.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:20:24.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:20:24.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:24.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:20:24.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:20:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:20:24.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:20:24.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:24.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:20:29.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:20:29.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:20:29.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:20:29.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:20:29.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:20:29.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:20:29.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:20:29.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:20:29.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:20:29.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:20:29.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:20:29.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:20:29.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:20:29.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:20:29.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:20:29.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:20:29.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:20:29.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:20:29.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:20:29.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:20:29.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:20:29.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:20:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:20:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:20:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:20:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:20:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:20:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:20:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:20:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:20:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:20:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:20:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:20:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:20:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:20:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:20:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:20:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:20:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:20:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:20:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:20:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:20:29.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:20:29.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:20:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:20:29.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:29.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:20:29.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:20:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:20:33.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:20:33.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:20:33.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:20:33.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:20:33.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:20:33.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:20:33.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:20:33.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:20:33.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:20:33.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:20:33.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:20:33.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:20:33.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:20:33.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:20:33.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:20:33.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:20:33.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:20:33.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:20:33.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:20:33.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:20:33.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:20:33.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:20:33.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:20:33.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:20:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:20:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:20:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:20:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:20:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:20:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:20:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:20:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:20:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:20:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:20:33.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:20:33.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:20:33.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:20:33.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:20:33.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:20:33.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:20:33.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:20:33.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:20:33.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:20:33.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:33.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:20:33.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:20:34.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:34.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:20:34.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:20:34.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:34.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:20:34.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:20:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:20:34.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:20:34.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:34.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:20:34.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:20:34.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:34.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:20:34.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:20:34.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:34.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:20:35.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:20:35.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:35.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:20:35.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:20:35.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:35.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:20:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:20:35.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:35.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:20:35.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:20:35.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:35.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:20:35.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:20:35.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:35.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:20:35.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:20:36.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:36.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:20:36.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:20:36.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:36.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:20:36.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:20:36.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:36.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:20:36.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:20:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:36.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:20:36.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:20:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:36.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:20:36.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:20:37.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:37.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:20:37.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:20:37.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:37.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:20:37.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:20:37.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:37.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:20:37.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:20:37.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:37.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:20:37.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:20:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:37.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:20:38.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:20:38.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:38.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:20:38.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:20:38.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:38.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:20:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:20:38.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:38.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:20:38.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:20:38.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:38.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:20:38.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:20:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:38.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:20:38.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:20:39.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:39.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:20:39.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:20:39.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:39.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:20:39.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:20:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:20:39.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:20:39.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:39.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:20:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:20:39.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:39.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:20:43.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:20:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:20:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:20:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:20:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:20:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:20:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:20:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:20:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:20:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:20:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:20:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:20:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:20:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:20:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:20:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:20:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:20:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:20:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:20:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:20:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:20:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:20:43.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:20:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:20:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:20:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:20:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:20:43.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:20:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:20:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:20:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:20:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:20:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:20:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:20:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:20:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:20:43.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:20:43.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:20:43.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:20:43.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:20:43.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:20:43.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:20:43.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:20:43.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:43.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:20:44.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:20:44.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:44.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:20:44.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:20:44.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:44.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:20:44.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:20:44.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:44.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:20:44.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:20:44.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:44.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:20:44.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:20:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:44.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:20:45.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:20:45.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:45.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:20:45.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:20:45.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:45.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:20:45.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:20:45.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:45.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:20:45.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:20:45.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:45.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:20:45.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:20:45.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:45.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:20:45.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:20:46.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:46.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:20:46.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:20:46.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:46.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:20:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:20:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:20:46.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:20:46.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:46.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:20:46.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:20:46.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:46.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:20:46.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:20:47.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:47.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:20:47.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:20:47.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:47.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:20:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:20:47.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:47.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:20:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:20:47.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:47.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:20:47.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:20:47.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:47.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:20:47.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:20:47.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:47.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:20:48.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:20:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:20:48.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:20:48.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:48.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:20:48.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:20:48.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:48.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:20:48.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:20:48.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:48.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:20:48.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:20:48.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:48.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:20:49.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:20:49.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:49.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:20:49.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:20:49.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:49.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:20:49.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:20:49.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:49.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:20:49.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:20:49.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:49.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:20:49.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:20:49.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:49.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:20:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:20:54.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:20:54.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:20:54.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:20:54.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:20:54.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:20:54.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:20:54.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:20:54.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:20:54.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:20:54.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:20:54.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:20:54.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:20:54.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:20:54.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:20:54.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:20:54.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:20:54.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:20:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:20:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:20:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:20:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:20:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:20:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:20:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:20:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:20:54.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:20:54.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:20:54.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:20:54.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:20:54.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:20:54.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:20:54.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:20:54.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:20:54.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:20:54.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:20:54.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:20:54.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:20:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:20:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:20:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:20:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:20:54.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:20:54.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:20:54.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:20:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:20:54.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:20:54.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:20:54.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:20:54.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:20:54.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:20:54.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:54.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:20:55.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:20:55.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:55.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:20:55.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:20:55.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:55.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:20:55.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:20:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:20:55.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:20:55.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:55.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:20:55.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:20:55.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:55.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:20:56.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:20:56.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:56.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:20:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:20:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:20:56.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:20:56.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:56.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:20:56.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:20:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:20:56.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:20:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:20:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:20:57.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:57.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:20:57.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:20:57.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:57.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:20:57.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:20:57.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:57.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:20:57.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:20:58.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:58.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:20:58.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:20:58.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:58.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:20:58.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:20:58.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:58.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:20:58.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:20:58.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:58.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:20:58.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:20:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:20:59.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:20:59.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:59.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:20:59.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:20:59.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:59.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:20:59.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:20:59.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:20:59.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:20:59.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:21:03.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:21:03.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:21:03.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:21:03.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:21:03.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:21:03.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:21:03.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:21:03.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:21:03.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:21:03.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:21:03.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:21:03.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:21:03.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:21:03.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:21:03.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:21:03.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:21:03.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:21:03.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:21:03.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:21:03.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:21:03.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:21:03.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:21:03.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:21:03.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:21:03.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:21:03.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:21:03.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:21:03.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:21:03.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:21:03.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:21:03.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:21:03.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:21:03.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:21:03.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:21:03.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:21:03.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:21:03.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:21:03.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:21:03.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:21:03.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:21:03.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:21:03.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:21:03.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:21:03.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:21:03.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:03.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:21:04.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:21:04.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:04.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:21:04.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:21:04.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:04.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:21:04.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:21:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:21:04.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:21:04.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:04.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:21:04.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:21:04.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:04.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:21:05.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:21:05.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:05.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:21:05.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:21:05.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:05.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:21:05.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:21:05.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:05.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:21:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:21:05.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:05.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:21:05.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:21:05.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:05.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:21:05.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:21:06.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:06.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:21:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:21:06.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:06.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:21:06.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:21:06.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:06.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:21:06.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:21:06.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:06.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:21:06.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:21:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:21:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:21:06.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:06.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:21:07.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:21:07.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:07.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:21:07.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:21:07.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:07.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:21:07.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:21:07.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:07.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:21:07.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:21:07.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:07.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:21:07.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:21:07.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:07.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:21:08.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:21:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:21:08.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:21:08.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:08.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:21:08.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:21:08.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:08.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:21:08.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:21:08.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:08.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:21:08.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:21:08.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:08.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:21:09.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:21:09.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:09.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:21:09.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:21:09.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:09.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:21:09.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:21:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:09.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:21:09.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:21:09.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:09.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:21:09.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:21:09.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:21:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:21:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:21:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:21:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:21:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:21:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:21:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:21:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:21:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:21:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:21:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:21:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:21:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:21:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:21:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:21:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:21:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:21:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:21:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:21:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:21:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:21:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:21:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:21:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:21:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:21:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:21:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:21:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:21:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:21:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:21:14.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:21:14.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:21:14.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:21:14.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:21:14.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:21:14.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:21:14.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:21:14.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:21:14.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:21:14.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:21:14.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:21:14.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:21:14.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:21:14.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:21:14.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:21:14.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:14.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:21:14.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:21:18.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:21:18.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:21:18.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:21:18.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:21:18.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:21:18.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:21:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:21:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:21:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:21:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:21:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:21:18.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:21:18.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:21:18.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:21:18.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:21:18.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:21:18.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:21:18.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:21:18.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:21:18.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:21:18.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:21:18.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:21:18.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:21:18.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:21:18.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:21:18.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:21:18.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:21:18.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:21:18.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:21:18.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:21:18.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:21:18.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:21:18.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:21:18.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:21:18.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:21:18.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:21:18.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:21:18.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:21:18.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:21:18.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:21:18.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:21:18.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:21:18.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:21:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:21:18.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:21:18.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:21:18.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:18.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:21:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:21:19.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:19.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:21:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:21:19.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:19.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:21:19.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:21:19.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:19.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:21:19.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:21:19.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:19.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:21:19.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:21:19.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:19.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:21:19.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:21:20.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:20.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:21:20.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:21:20.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:20.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:21:20.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:21:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:20.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:21:20.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:21:20.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:20.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:21:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:21:20.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:20.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:21:20.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:21:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:21:21.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:21:21.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:21:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:21:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:21:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:21:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:21:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:21:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:21:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:21:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:21:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:21:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:21:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:21:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:21:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:21:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:21:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:21:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:21:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:21:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:21:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:21:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:21:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:21:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:21:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:21:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:21:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:21:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:21:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:21:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:21:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:21:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:21:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:21:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:21:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:21:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:21:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:21:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:21:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:21:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:21:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:21:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:21:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:21:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:21:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:21:25.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:21:25.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:21:25.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:21:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:21:25.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:25.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:21:25.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:21:26.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:26.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:21:26.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:21:26.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:26.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:21:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:21:26.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:26.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:21:26.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:21:26.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:26.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:21:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:21:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:21:26.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:21:31.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:21:31.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:21:31.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:21:31.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:21:31.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:21:31.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:21:31.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:21:31.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:21:31.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:21:31.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:21:31.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:21:31.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:21:31.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:21:31.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:21:31.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:21:31.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:21:31.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:21:31.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:21:31.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:21:31.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:21:31.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:21:31.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:21:31.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:21:31.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:21:31.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:21:31.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:21:31.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:21:31.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:21:31.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:21:31.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:21:31.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:21:31.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:21:31.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:21:31.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:21:31.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:21:31.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:21:31.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:21:31.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:21:31.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:21:31.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:21:31.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:21:31.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:21:31.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:21:31.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:21:31.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:21:31.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:21:31.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:21:31.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:21:31.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:31.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:21:32.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:21:32.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:32.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:21:32.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:21:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:21:32.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:21:32.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:32.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:21:32.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:21:32.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:32.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:21:32.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:21:32.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:32.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:21:33.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:21:33.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:33.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:21:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:21:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:21:33.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:21:33.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:33.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:21:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:21:33.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:33.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:21:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:21:33.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:33.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:21:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:21:34.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:34.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:21:34.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:21:34.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:34.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:21:34.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:21:34.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:34.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:21:34.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:21:34.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:34.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:21:34.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:21:34.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:34.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:21:35.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:21:35.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:35.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:21:35.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:21:35.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:35.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:21:35.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:21:35.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:35.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:21:35.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:21:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:35.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:21:35.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:21:35.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:35.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:21:35.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:21:36.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:36.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:21:36.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:21:36.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:36.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:21:36.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:21:36.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:21:40.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:21:40.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:21:40.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:21:40.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:21:40.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:21:40.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:21:40.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:21:40.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:21:40.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:21:40.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:21:40.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:21:40.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:21:40.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:21:40.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:21:40.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:21:40.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:21:40.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:21:40.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:21:40.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:21:40.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:21:40.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:21:40.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:21:40.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:21:40.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:21:40.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:21:40.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:21:40.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:21:40.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:21:40.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:21:40.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:21:40.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:21:40.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:21:40.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:21:40.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:21:40.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:21:40.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:21:40.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:21:40.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:21:40.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:21:40.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:21:40.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:21:40.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:21:40.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:21:40.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:21:40.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:21:40.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:40.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:21:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:21:41.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:41.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:21:41.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:21:41.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:41.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:21:41.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:21:41.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:41.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:21:41.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:21:41.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:41.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:21:41.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:21:41.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:41.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:21:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:21:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:41.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:21:42.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:21:42.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:42.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:21:42.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:21:42.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:42.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:21:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:21:42.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:42.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:21:42.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:21:42.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:42.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:21:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:21:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:42.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:21:43.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:21:43.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:43.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:21:43.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:21:43.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:43.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:21:43.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:21:43.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:43.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:21:43.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:21:43.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:43.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:21:43.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:21:43.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:43.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:21:43.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:21:44.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:44.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:21:44.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:21:44.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:44.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:21:44.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:21:44.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:44.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:21:44.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:21:44.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:44.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:21:44.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:21:44.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:44.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:21:44.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:21:45.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:45.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:21:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:21:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:45.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:21:45.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:21:45.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:45.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:21:45.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:21:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:45.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:21:45.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:21:45.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:45.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:21:45.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:21:45.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:45.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:21:46.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:21:46.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:46.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:21:46.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:21:46.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:46.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:21:46.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:21:46.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:46.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:21:46.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:21:46.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:46.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:21:46.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:21:46.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:46.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:21:47.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:21:47.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:47.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:21:47.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:21:47.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:47.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:21:47.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:21:47.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:47.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:21:47.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:21:47.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:47.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:21:47.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:21:47.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:47.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:21:48.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:21:48.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:48.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:21:48.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:21:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:48.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:21:48.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:21:48.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:48.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:21:48.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:21:48.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:48.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:21:48.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:21:48.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:48.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:21:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:21:49.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:49.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:21:49.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:21:49.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:49.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:21:49.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:21:49.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:49.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:21:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:21:49.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:49.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:21:49.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:21:49.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:49.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:21:49.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:21:50.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:50.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:21:50.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:21:50.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:50.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:21:50.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:21:50.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:50.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:21:50.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:21:50.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:50.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:21:50.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:21:50.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:50.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:21:50.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:21:51.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:51.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:21:51.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:21:51.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:51.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:21:51.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:21:51.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:51.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:21:51.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:21:51.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:51.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:21:51.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:21:51.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:51.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:21:51.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:21:51.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:51.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:21:52.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:21:52.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:52.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:21:52.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:21:52.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:52.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:21:52.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:21:52.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:52.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:21:52.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:21:52.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:52.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:21:52.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:21:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:52.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:21:53.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:21:53.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:53.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:21:53.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:21:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:21:53.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:21:53.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:53.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:21:53.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:21:53.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:53.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:21:53.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:21:53.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:53.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:21:53.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:21:54.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:54.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:21:54.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:21:54.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:54.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:21:54.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:21:54.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:54.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:21:54.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:21:54.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:54.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:21:54.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:21:54.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:54.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:21:54.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:21:55.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:55.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:21:55.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:21:55.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:55.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:21:55.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:21:55.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:55.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:21:55.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:21:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:21:55.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:21:55.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:55.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:21:55.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:21:55.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:55.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:21:56.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:21:56.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:56.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:21:56.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:21:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:21:56.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:21:56.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:56.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:21:56.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:21:56.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:56.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:21:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:21:56.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:56.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:21:57.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:21:57.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:57.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:21:57.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:21:57.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:57.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:21:57.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:21:57.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:57.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:21:57.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:21:57.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:57.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:21:57.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:21:57.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:57.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:21:58.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:21:58.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:58.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:21:58.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:21:58.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:58.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:21:58.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:21:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:21:58.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:21:58.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:58.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:21:58.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:21:58.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:58.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:21:59.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:21:59.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:59.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:21:59.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:21:59.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:59.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:21:59.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:21:59.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:21:59.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:21:59.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:22:00.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:00.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:22:00.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:22:00.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:00.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:22:00.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:22:00.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:00.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:22:00.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:22:00.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:00.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:22:00.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:22:00.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:00.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:22:00.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:22:01.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:01.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:22:01.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:22:01.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:01.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:22:01.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:22:01.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:01.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:22:01.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:22:01.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:01.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:22:01.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:22:01.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:01.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:22:01.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:22:01.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:01.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:22:02.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:22:02.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:02.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:22:02.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:22:02.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:02.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:22:02.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:22:02.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:02.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:22:02.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:22:02.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:02.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:22:02.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:22:02.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:22:07.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:22:07.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:22:07.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:22:07.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:22:07.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:22:07.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:22:07.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:22:07.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:22:07.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:22:07.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:22:07.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:22:07.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:22:07.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:22:07.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:22:07.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:22:07.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:22:07.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:22:07.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:22:07.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:22:07.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:22:07.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:22:07.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:22:07.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:22:07.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:22:07.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:22:07.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:22:07.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:22:07.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:22:07.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:22:07.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:22:07.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:22:07.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:22:07.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:22:07.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:22:07.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:22:07.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:22:07.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:22:07.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:22:07.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:22:07.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:22:07.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:22:07.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:22:07.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:22:07.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:22:07.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:22:07.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:22:07.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:22:07.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:22:07.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:22:07.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:07.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:22:07.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:22:08.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:08.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:22:08.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:22:08.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:08.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:22:08.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:22:08.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:08.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:22:08.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:22:08.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:08.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:22:08.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:22:08.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:08.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:22:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:22:08.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:08.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:22:09.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:22:09.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:09.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:22:09.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:22:09.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:09.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:22:09.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:22:09.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:09.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:22:09.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:22:09.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:09.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:22:09.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:22:09.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:09.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:22:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:22:10.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:10.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:22:10.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:22:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:22:10.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:22:10.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:10.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:22:10.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:22:10.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:10.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:22:10.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:22:10.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:10.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:22:11.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:22:11.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:11.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:22:11.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:22:11.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:11.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:22:11.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:22:11.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:11.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:22:11.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:22:11.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:11.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:22:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:22:11.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:11.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:22:11.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:22:12.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:12.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:22:16.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:22:16.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:22:16.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:22:16.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:22:16.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:22:16.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:22:16.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:22:16.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:22:16.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:22:16.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:22:16.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:22:16.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:22:16.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:22:16.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:22:16.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:22:16.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:22:16.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:22:16.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:22:16.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:22:16.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:22:16.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:22:16.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:22:16.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:22:16.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:22:16.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:22:16.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:22:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:22:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:22:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:22:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:22:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:22:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:22:16.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:22:16.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:22:16.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:22:16.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:22:16.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:22:16.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:22:16.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:22:16.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:22:16.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:22:16.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:22:16.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:22:16.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:22:16.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:22:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:22:16.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:22:16.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:22:16.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:22:16.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:16.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:22:17.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:22:17.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:17.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:22:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:22:17.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:17.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:22:17.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:22:17.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:17.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:22:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:22:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:22:17.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:22:21.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:22:21.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:22:21.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:22:21.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:22:21.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:22:21.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:22:21.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:22:21.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:22:21.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:22:21.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:22:21.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:22:21.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:22:21.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:22:21.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:22:21.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:22:21.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:22:21.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:22:21.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:22:21.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:22:21.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:22:21.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:22:21.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:22:21.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2020-11-16T21:22:21.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:22:21.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:22:21.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:379: TEST_backfill_test_sametarget: ceph osd pool set test1 size 2 2020-11-16T21:22:21.932 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T21:22:21.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:380: TEST_backfill_test_sametarget: ceph osd pool set test4 size 2 2020-11-16T21:22:21.932 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 2 2020-11-16T21:22:21.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:381: TEST_backfill_test_sametarget: sleep 30 2020-11-16T21:22:51.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:383: TEST_backfill_test_sametarget: wait_for_not_backfilling 240 2020-11-16T21:22:51.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2020-11-16T21:22:51.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2020-11-16T21:22:51.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2020-11-16T21:22:51.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:22:51.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:22:51.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:22:51.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2020-11-16T21:22:51.738 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:22:51.738 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:22:51.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:22:51.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:22:51.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:22:52.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:22:52.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:22:52.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:22:52.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:22:52.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:22:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:22:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:22:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:22:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:22:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T21:22:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:22:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:22:52.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:22:52.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836533 2020-11-16T21:22:52.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836533 2020-11-16T21:22:52.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533' 2020-11-16T21:22:52.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:22:52.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:22:52.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673012 2020-11-16T21:22:52.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673012 2020-11-16T21:22:52.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-42949673012' 2020-11-16T21:22:52.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:22:52.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:22:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509490 2020-11-16T21:22:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509490 2020-11-16T21:22:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-42949673012 2-64424509490' 2020-11-16T21:22:52.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:22:52.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:22:53.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345968 2020-11-16T21:22:53.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345968 2020-11-16T21:22:53.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-42949673012 2-64424509490 3-85899345968' 2020-11-16T21:22:53.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:22:53.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:22:53.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182447 2020-11-16T21:22:53.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182447 2020-11-16T21:22:53.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-42949673012 2-64424509490 3-85899345968 4-107374182447' 2020-11-16T21:22:53.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:22:53.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836533 2020-11-16T21:22:53.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:22:53.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:22:53.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836533 2020-11-16T21:22:53.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:22:53.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836533 2020-11-16T21:22:53.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2020-11-16T21:22:53.257 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836533 2020-11-16T21:22:53.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:22:53.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836531 -lt 21474836533 2020-11-16T21:22:53.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:22:54.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:22:54.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:22:55.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836533 -lt 21474836533 2020-11-16T21:22:55.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:22:55.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673012 2020-11-16T21:22:55.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:22:55.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:22:55.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673012 2020-11-16T21:22:55.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:22:55.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673012 2020-11-16T21:22:55.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2020-11-16T21:22:55.056 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673012 2020-11-16T21:22:55.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:22:55.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673013 -lt 42949673012 2020-11-16T21:22:55.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:22:55.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509490 2020-11-16T21:22:55.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:22:55.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:22:55.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509490 2020-11-16T21:22:55.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:22:55.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509490 2020-11-16T21:22:55.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509490' 2020-11-16T21:22:55.430 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509490 2020-11-16T21:22:55.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:22:55.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509490 -lt 64424509490 2020-11-16T21:22:55.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:22:55.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345968 2020-11-16T21:22:55.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:22:55.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:22:55.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345968 2020-11-16T21:22:55.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:22:55.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345968 2020-11-16T21:22:55.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345968' 2020-11-16T21:22:55.825 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 85899345968 2020-11-16T21:22:55.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:22:56.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345969 -lt 85899345968 2020-11-16T21:22:56.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:22:56.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182447 2020-11-16T21:22:56.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:22:56.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:22:56.290 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182447 2020-11-16T21:22:56.290 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:22:56.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182447 2020-11-16T21:22:56.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182447' 2020-11-16T21:22:56.292 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 107374182447 2020-11-16T21:22:56.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:22:56.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182447 -lt 107374182447 2020-11-16T21:22:56.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:22:56.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:22:56.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:23:00.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2020-11-16T21:23:00.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:23:00.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:23:00.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:23:00.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:23:00.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:23:00.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:23:00.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:23:00.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:23:00.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:23:00.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' -1 2020-11-16T21:23:00.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:68: wait_for_not_state: loop=0 2020-11-16T21:23:00.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:69: wait_for_not_state: num_in_state=1 2020-11-16T21:23:00.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 5 2020-11-16T21:23:02.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:23:02.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:23:02.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:23:02.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:23:02.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:23:02.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:23:02.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:23:02.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:23:02.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:23:02.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:23:02.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:23:02.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 1 >= 17 )) 2020-11-16T21:23:02.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 10 2020-11-16T21:23:12.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:23:12.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:23:12.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:23:12.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:23:12.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:23:12.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:23:12.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:23:12.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:23:13.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:23:13.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:23:13.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:23:13.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 2 >= 17 )) 2020-11-16T21:23:13.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:23:28.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:23:28.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:23:28.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:23:28.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:23:28.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:23:28.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:23:28.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:23:28.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:23:28.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:23:28.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:23:28.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:23:28.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 3 >= 17 )) 2020-11-16T21:23:28.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:23:43.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:23:43.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:23:43.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:23:43.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:23:43.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:23:43.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:23:43.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:23:43.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:23:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:23:44.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:23:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:23:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 4 >= 17 )) 2020-11-16T21:23:44.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:23:59.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:23:59.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:23:59.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:23:59.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:23:59.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:23:59.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:23:59.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:23:59.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:23:59.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:23:59.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:23:59.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:23:59.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:23:59.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:384: TEST_backfill_test_sametarget: wait_for_not_activating 60 2020-11-16T21:23:59.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2020-11-16T21:23:59.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2020-11-16T21:23:59.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2020-11-16T21:23:59.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:23:59.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:23:59.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:23:59.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2020-11-16T21:23:59.535 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:23:59.535 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:23:59.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:23:59.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:23:59.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:23:59.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:23:59.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:23:59.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:23:59.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:23:59.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:24:00.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:24:00.017 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:24:00.017 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:24:00.017 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:24:00.017 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T21:24:00.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:24:00.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:24:00.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:24:00.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836548 2020-11-16T21:24:00.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836548 2020-11-16T21:24:00.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836548' 2020-11-16T21:24:00.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:24:00.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:24:00.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673028 2020-11-16T21:24:00.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673028 2020-11-16T21:24:00.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836548 1-42949673028' 2020-11-16T21:24:00.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:24:00.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:24:00.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509505 2020-11-16T21:24:00.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509505 2020-11-16T21:24:00.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836548 1-42949673028 2-64424509505' 2020-11-16T21:24:00.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:24:00.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:24:00.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345984 2020-11-16T21:24:00.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345984 2020-11-16T21:24:00.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836548 1-42949673028 2-64424509505 3-85899345984' 2020-11-16T21:24:00.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:24:00.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:24:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182462 2020-11-16T21:24:00.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182462 2020-11-16T21:24:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836548 1-42949673028 2-64424509505 3-85899345984 4-107374182462' 2020-11-16T21:24:00.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:24:00.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836548 2020-11-16T21:24:00.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:24:00.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:24:00.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836548 2020-11-16T21:24:00.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:24:00.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836548 2020-11-16T21:24:00.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2020-11-16T21:24:00.856 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836548 2020-11-16T21:24:00.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:24:01.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836546 -lt 21474836548 2020-11-16T21:24:01.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:24:02.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:24:02.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:24:02.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836549 -lt 21474836548 2020-11-16T21:24:02.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:24:02.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673028 2020-11-16T21:24:02.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:24:02.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:24:02.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673028 2020-11-16T21:24:02.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:24:02.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673028 2020-11-16T21:24:02.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673028' 2020-11-16T21:24:02.644 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673028 2020-11-16T21:24:02.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:24:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673028 -lt 42949673028 2020-11-16T21:24:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:24:03.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509505 2020-11-16T21:24:03.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:24:03.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:24:03.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509505 2020-11-16T21:24:03.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:24:03.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509505 2020-11-16T21:24:03.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2020-11-16T21:24:03.028 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509505 2020-11-16T21:24:03.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:24:03.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509506 -lt 64424509505 2020-11-16T21:24:03.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:24:03.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345984 2020-11-16T21:24:03.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:24:03.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:24:03.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:24:03.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345984 2020-11-16T21:24:03.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345984 2020-11-16T21:24:03.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345984' 2020-11-16T21:24:03.437 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 85899345984 2020-11-16T21:24:03.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:24:03.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345984 -lt 85899345984 2020-11-16T21:24:03.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:24:03.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182462 2020-11-16T21:24:03.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:24:03.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:24:03.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182462 2020-11-16T21:24:03.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:24:03.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182462 2020-11-16T21:24:03.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182462' 2020-11-16T21:24:03.839 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 107374182462 2020-11-16T21:24:03.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:24:04.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182463 -lt 107374182462 2020-11-16T21:24:04.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:24:04.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:24:04.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:24:04.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2020-11-16T21:24:04.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:24:04.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2020-11-16T21:24:04.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2020-11-16T21:24:04.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:24:04.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2020-11-16T21:24:04.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:24:04.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2020-11-16T21:24:05.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:24:05.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:24:05.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:24:05.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:24:05.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:386: TEST_backfill_test_sametarget: ERRORS=0 2020-11-16T21:24:05.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:387: TEST_backfill_test_sametarget: grep +backfill_toofull 2020-11-16T21:24:05.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:387: TEST_backfill_test_sametarget: ceph pg dump pgs 2020-11-16T21:24:05.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:387: TEST_backfill_test_sametarget: wc -l 2020-11-16T21:24:05.472 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:24:05.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:387: TEST_backfill_test_sametarget: '[' 1 '!=' 1 ']' 2020-11-16T21:24:05.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:393: TEST_backfill_test_sametarget: ceph pg dump pgs 2020-11-16T21:24:05.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:393: TEST_backfill_test_sametarget: wc -l 2020-11-16T21:24:05.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:393: TEST_backfill_test_sametarget: grep active+clean 2020-11-16T21:24:05.829 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:24:05.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:393: TEST_backfill_test_sametarget: '[' 1 '!=' 1 ']' 2020-11-16T21:24:05.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:399: TEST_backfill_test_sametarget: ceph pg dump pgs 2020-11-16T21:24:06.211 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:24:06.211 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2020-11-16T21:23:59.201129+0000 95'600 104:1889 [1,0] 1 [1,0] 1 0'0 2020-11-16T21:19:34.487409+0000 0'0 2020-11-16T21:19:34.487409+0000 0 2020-11-16T21:24:06.211 INFO:tasks.workunit.client.0.smithi189.stdout:4.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2020-11-16T21:23:52.693779+0000 95'600 103:685 [4,0] 4 [4] 4 0'0 2020-11-16T21:19:42.383233+0000 0'0 2020-11-16T21:19:42.383233+0000 0 2020-11-16T21:24:06.212 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:24:06.212 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:24:06.213 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:24:06.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:401: TEST_backfill_test_sametarget: '[' 0 '!=' 0 ']' 2020-11-16T21:24:06.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:406: TEST_backfill_test_sametarget: delete_pool test1 2020-11-16T21:24:06.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2020-11-16T21:24:06.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-11-16T21:24:06.811 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' does not exist 2020-11-16T21:24:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:407: TEST_backfill_test_sametarget: delete_pool test4 2020-11-16T21:24:06.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test4 2020-11-16T21:24:06.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-11-16T21:24:07.438 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test4' does not exist 2020-11-16T21:24:07.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:408: TEST_backfill_test_sametarget: kill_daemons td/osd-backfill-space 2020-11-16T21:24:07.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:24:07.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:24:07.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:24:07.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:24:07.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:24:07.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:24:07.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:409: TEST_backfill_test_sametarget: grep -q 'num_bytes mismatch' td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log td/osd-backfill-space/osd.3.log td/osd-backfill-space/osd.4.log 2020-11-16T21:24:07.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2020-11-16T21:24:07.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:24:07.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:24:07.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:24:07.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:24:07.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:24:07.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:24:07.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:24:07.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:24:07.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:24:07.822 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:24:07.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:24:07.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:24:07.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:24:07.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:24:07.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:24:07.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:24:07.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:24:07.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:24:07.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:24:07.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:24:07.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:24:07.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:24:07.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:24:07.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:24:07.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:24:07.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:24:07.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:24:07.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:24:07.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2020-11-16T21:24:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2020-11-16T21:24:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2020-11-16T21:24:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2020-11-16T21:24:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:24:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:24:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:24:07.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:24:07.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:24:07.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:24:07.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:24:07.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:24:07.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:24:07.882 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:24:07.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:24:07.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:24:07.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:24:07.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:24:07.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:24:07.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:24:07.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:24:07.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:24:07.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:24:07.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:24:07.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:24:07.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:24:07.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:24:07.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:24:07.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:24:07.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:24:07.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:24:07.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:24:07.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2020-11-16T21:24:07.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:24:07.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:24:07.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:24:07.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137323 2020-11-16T21:24:07.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:24:07.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:24:07.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:24:07.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2020-11-16T21:24:07.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_backfill_test_simple td/osd-backfill-space 2020-11-16T21:24:07.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:109: TEST_backfill_test_simple: local dir=td/osd-backfill-space 2020-11-16T21:24:07.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:110: TEST_backfill_test_simple: local pools=2 2020-11-16T21:24:07.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:111: TEST_backfill_test_simple: local OSDS=3 2020-11-16T21:24:07.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:113: TEST_backfill_test_simple: run_mon td/osd-backfill-space a 2020-11-16T21:24:07.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2020-11-16T21:24:07.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:24:07.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:24:07.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:24:07.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2020-11-16T21:24:07.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2020-11-16T21:24:08.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:24:08.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:24:08.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:24:08.205 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:24:08.205 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:24:08.205 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:24:08.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:24:08.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:24:08.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:24:08.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:24:08.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:24:08.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:24:08.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:24:08.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:24:08.412 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:24:08.412 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:24:08.412 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:24:08.412 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:24:08.413 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:24:08.413 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:24:08.413 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:24:08.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:24:08.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get fsid 2020-11-16T21:24:08.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:24:08.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:24:08.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:24:08.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:24:08.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:24:08.520 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:24:08.520 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:24:08.521 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:24:08.521 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:24:08.521 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:24:08.521 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:24:08.522 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:24:08.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:24:08.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get mon_host 2020-11-16T21:24:08.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:114: TEST_backfill_test_simple: run_mgr td/osd-backfill-space x 2020-11-16T21:24:08.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2020-11-16T21:24:08.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:24:08.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:24:08.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:24:08.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2020-11-16T21:24:08.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:24:09.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:24:09.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:24:09.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:24:09.156 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:24:09.156 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:24:09.156 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:24:09.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:24:09.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:24:09.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:24:09.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:115: TEST_backfill_test_simple: export CEPH_ARGS 2020-11-16T21:24:09.188 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:117: TEST_backfill_test_simple: expr 3 - 1 2020-11-16T21:24:09.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:117: TEST_backfill_test_simple: seq 0 2 2020-11-16T21:24:09.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:117: TEST_backfill_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:24:09.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:119: TEST_backfill_test_simple: run_osd td/osd-backfill-space 0 2020-11-16T21:24:09.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:24:09.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:24:09.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:24:09.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:24:09.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2020-11-16T21:24:09.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:24:09.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:24:09.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:24:09.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:24:09.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2020-11-16T21:24:09.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2020-11-16T21:24:09.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:24:09.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:24:09.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:24:09.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:24:09.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:24:09.196 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:24:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:24:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:24:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:24:09.196 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:24:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:24:09.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:24:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:24:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:24:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:24:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:24:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:24:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:24:09.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:24:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:24:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:24:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2020-11-16T21:24:09.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:24:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3a454442-c82d-4da8-a096-e05c544cd901 2020-11-16T21:24:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 3a454442-c82d-4da8-a096-e05c544cd901' 2020-11-16T21:24:09.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:24:09.199 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 3a454442-c82d-4da8-a096-e05c544cd901 2020-11-16T21:24:09.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD57bJfSbd8DRAAmuIpBxWwklQTjg7XT53VvA== 2020-11-16T21:24:09.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD57bJfSbd8DRAAmuIpBxWwklQTjg7XT53VvA=="}' 2020-11-16T21:24:09.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3a454442-c82d-4da8-a096-e05c544cd901 -i td/osd-backfill-space/0/new.json 2020-11-16T21:24:09.567 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:24:09.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2020-11-16T21:24:09.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD57bJfSbd8DRAAmuIpBxWwklQTjg7XT53VvA== --osd-uuid 3a454442-c82d-4da8-a096-e05c544cd901 2020-11-16T21:24:09.609 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:09.612+0000 7f95c305bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:09.611 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:09.616+0000 7f95c305bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:09.612 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:09.616+0000 7f95c305bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:09.614 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:09.616+0000 7f95c305bd80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:24:09.614 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:09.616+0000 7f95c305bd80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:24:09.614 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:09.616+0000 7f95c305bd80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:24:09.664 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:09.668+0000 7f95c305bd80 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2020-11-16T21:24:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2020-11-16T21:24:14.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:24:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:24:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:24:14.066 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:24:14.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:24:14.557 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:24:14.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:24:14.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:24:14.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:24:14.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:24:14.587 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:14.592+0000 7f796fc46d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:14.590 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:14.592+0000 7f796fc46d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:14.591 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:14.592+0000 7f796fc46d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:14.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:24:14.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:24:14.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:24:14.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:24:14.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:24:14.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:24:14.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:24:14.937 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:24:14.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:24:14.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:24:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:15.132+0000 7f796fc46d80 -1 Falling back to public interface 2020-11-16T21:24:15.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:24:16.021 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:16.020+0000 7f796fc46d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:24:16.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:24:16.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:24:16.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:24:16.334 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:24:16.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:24:16.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:24:16.745 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/382724,v1:127.0.0.1:6803/382724] [v2:127.0.0.1:6804/382724,v1:127.0.0.1:6805/382724] exists,up 3a454442-c82d-4da8-a096-e05c544cd901 2020-11-16T21:24:16.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:24:16.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:24:16.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:24:16.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:117: TEST_backfill_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:24:16.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:119: TEST_backfill_test_simple: run_osd td/osd-backfill-space 1 2020-11-16T21:24:16.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:24:16.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:24:16.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:24:16.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:24:16.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2020-11-16T21:24:16.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:24:16.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:24:16.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:24:16.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:24:16.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2020-11-16T21:24:16.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2020-11-16T21:24:16.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:24:16.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:24:16.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:24:16.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:24:16.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:24:16.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:24:16.750 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:24:16.750 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:24:16.750 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:24:16.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:24:16.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:24:16.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:24:16.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:24:16.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:24:16.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:24:16.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:24:16.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:24:16.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:24:16.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:24:16.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:24:16.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:24:16.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2020-11-16T21:24:16.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:24:16.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9633280d-5212-438d-a5b3-8ac22b1bc00e 2020-11-16T21:24:16.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 9633280d-5212-438d-a5b3-8ac22b1bc00e' 2020-11-16T21:24:16.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:24:16.753 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 9633280d-5212-438d-a5b3-8ac22b1bc00e 2020-11-16T21:24:16.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAA7rJfktaVLhAAJsXil2eEfxXfMRxngc74iA== 2020-11-16T21:24:16.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAA7rJfktaVLhAAJsXil2eEfxXfMRxngc74iA=="}' 2020-11-16T21:24:16.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9633280d-5212-438d-a5b3-8ac22b1bc00e -i td/osd-backfill-space/1/new.json 2020-11-16T21:24:17.363 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:24:17.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2020-11-16T21:24:17.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAA7rJfktaVLhAAJsXil2eEfxXfMRxngc74iA== --osd-uuid 9633280d-5212-438d-a5b3-8ac22b1bc00e 2020-11-16T21:24:17.410 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:17.412+0000 7f20fa807d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:17.412 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:17.416+0000 7f20fa807d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:17.414 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:17.416+0000 7f20fa807d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:17.414 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:17.416+0000 7f20fa807d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:24:17.414 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:17.416+0000 7f20fa807d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:24:17.415 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:17.416+0000 7f20fa807d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:24:17.463 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:17.468+0000 7f20fa807d80 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2020-11-16T21:24:21.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2020-11-16T21:24:21.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:24:21.858 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:24:21.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:24:21.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:24:22.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:24:22.357 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:24:22.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:24:22.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:24:22.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:24:22.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:24:22.388 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:22.392+0000 7f46f2022d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:22.390 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:22.392+0000 7f46f2022d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:22.392 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:22.396+0000 7f46f2022d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:22.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:24:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:24:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:24:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:24:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:24:22.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:24:22.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:24:22.728 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:24:22.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:24:22.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:24:22.937 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:22.940+0000 7f46f2022d80 -1 Falling back to public interface 2020-11-16T21:24:23.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:24:23.794 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:23.796+0000 7f46f2022d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:24:24.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:24:24.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:24:24.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:24:24.110 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:24:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:24:24.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:24:24.507 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/383718,v1:127.0.0.1:6811/383718] [v2:127.0.0.1:6812/383718,v1:127.0.0.1:6813/383718] exists,up 9633280d-5212-438d-a5b3-8ac22b1bc00e 2020-11-16T21:24:24.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:24:24.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:24:24.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:24:24.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:117: TEST_backfill_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:24:24.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:119: TEST_backfill_test_simple: run_osd td/osd-backfill-space 2 2020-11-16T21:24:24.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:24:24.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:24:24.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:24:24.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:24:24.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2020-11-16T21:24:24.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:24:24.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:24:24.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:24:24.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:24:24.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2020-11-16T21:24:24.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2020-11-16T21:24:24.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:24:24.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:24:24.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:24:24.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:24:24.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:24:24.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:24:24.512 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:24:24.512 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:24:24.512 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:24:24.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:24:24.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:24:24.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:24:24.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:24:24.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:24:24.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:24:24.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:24:24.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:24:24.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:24:24.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:24:24.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:24:24.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:24:24.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2020-11-16T21:24:24.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:24:24.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1f8ef613-3222-46ad-b7a3-35bf1bf0ec8e 2020-11-16T21:24:24.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 1f8ef613-3222-46ad-b7a3-35bf1bf0ec8e' 2020-11-16T21:24:24.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:24:24.515 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 1f8ef613-3222-46ad-b7a3-35bf1bf0ec8e 2020-11-16T21:24:24.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAI7rJfSjJfIBAAWo2zk2VVNpPowwvkrCuizw== 2020-11-16T21:24:24.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAI7rJfSjJfIBAAWo2zk2VVNpPowwvkrCuizw=="}' 2020-11-16T21:24:24.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1f8ef613-3222-46ad-b7a3-35bf1bf0ec8e -i td/osd-backfill-space/2/new.json 2020-11-16T21:24:25.146 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:24:25.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2020-11-16T21:24:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAI7rJfSjJfIBAAWo2zk2VVNpPowwvkrCuizw== --osd-uuid 1f8ef613-3222-46ad-b7a3-35bf1bf0ec8e 2020-11-16T21:24:25.193 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:25.196+0000 7fb2d2630d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:25.202 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:25.204+0000 7fb2d2630d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:25.208+0000 7fb2d2630d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:25.208+0000 7fb2d2630d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:24:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:25.208+0000 7fb2d2630d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:24:25.204 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:25.208+0000 7fb2d2630d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:24:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:25.272+0000 7fb2d2630d80 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2020-11-16T21:24:29.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2020-11-16T21:24:29.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:24:29.654 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:24:29.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:24:29.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:24:30.151 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:24:30.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:24:30.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:24:30.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:24:30.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:24:30.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:24:30.183 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:30.187+0000 7f06bbb58d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:30.185 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:30.191+0000 7f06bbb58d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:30.186 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:30.191+0000 7f06bbb58d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:24:30.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:24:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:24:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:24:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:24:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:24:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:24:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:24:30.542 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:24:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:24:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:24:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:30.743+0000 7f06bbb58d80 -1 Falling back to public interface 2020-11-16T21:24:30.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:24:31.598 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:24:31.599+0000 7f06bbb58d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:24:31.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:24:31.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:24:31.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:24:31.925 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:24:31.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:24:31.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:24:32.334 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/384712,v1:127.0.0.1:6819/384712] [v2:127.0.0.1:6820/384712,v1:127.0.0.1:6821/384712] exists,up 1f8ef613-3222-46ad-b7a3-35bf1bf0ec8e 2020-11-16T21:24:32.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:24:32.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:24:32.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:24:32.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:122: TEST_backfill_test_simple: ceph osd set-backfillfull-ratio .85 2020-11-16T21:24:33.189 INFO:tasks.workunit.client.0.smithi189.stderr:osd set-backfillfull-ratio 0.85 2020-11-16T21:24:33.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:124: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:33.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:124: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:33.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:126: TEST_backfill_test_simple: create_pool test1 1 1 2020-11-16T21:24:33.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2020-11-16T21:24:33.917 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' already exists 2020-11-16T21:24:33.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:24:34.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:127: TEST_backfill_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2020-11-16T21:24:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 1 2020-11-16T21:24:35.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:124: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:35.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:126: TEST_backfill_test_simple: create_pool test2 1 1 2020-11-16T21:24:35.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2020-11-16T21:24:36.359 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' already exists 2020-11-16T21:24:36.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:24:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:127: TEST_backfill_test_simple: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2020-11-16T21:24:38.102 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 1 2020-11-16T21:24:38.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:130: TEST_backfill_test_simple: wait_for_clean 2020-11-16T21:24:38.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:24:38.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:24:38.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:24:38.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:24:38.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:24:38.119 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:24:38.119 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:24:38.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:24:38.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:24:38.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:24:38.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:24:38.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:24:38.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:24:38.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:24:38.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:24:38.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:24:38.716 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:24:38.716 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T21:24:38.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:24:38.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:24:38.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:24:38.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2020-11-16T21:24:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2020-11-16T21:24:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2020-11-16T21:24:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:24:38.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:24:39.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2020-11-16T21:24:39.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2020-11-16T21:24:39.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2020-11-16T21:24:39.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:24:39.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:24:39.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509445 2020-11-16T21:24:39.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509445 2020-11-16T21:24:39.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509445' 2020-11-16T21:24:39.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:24:39.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2020-11-16T21:24:39.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:24:39.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:24:39.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2020-11-16T21:24:39.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:24:39.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2020-11-16T21:24:39.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-11-16T21:24:39.275 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836487 2020-11-16T21:24:39.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:24:39.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2020-11-16T21:24:39.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:24:40.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:24:40.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:24:41.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2020-11-16T21:24:41.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:24:41.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2020-11-16T21:24:41.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:24:41.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:24:41.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2020-11-16T21:24:41.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:24:41.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2020-11-16T21:24:41.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-11-16T21:24:41.082 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672965 2020-11-16T21:24:41.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:24:41.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672965 2020-11-16T21:24:41.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:24:41.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509445 2020-11-16T21:24:41.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:24:41.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:24:41.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509445 2020-11-16T21:24:41.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:24:41.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509445 2020-11-16T21:24:41.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2020-11-16T21:24:41.490 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509445 2020-11-16T21:24:41.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:24:41.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509445 -lt 64424509445 2020-11-16T21:24:41.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:24:41.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:24:41.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:24:42.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2020-11-16T21:24:42.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:24:42.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:24:42.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:24:42.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:24:42.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:24:42.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:24:42.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:24:42.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T21:24:42.757 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:24:42.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:24:42.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:24:43.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2020-11-16T21:24:43.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:24:43.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:24:43.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:135: TEST_backfill_test_simple: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=4 2020-11-16T21:24:43.225 INFO:tasks.workunit.client.0.smithi189.stderr:4+0 records in 2020-11-16T21:24:43.225 INFO:tasks.workunit.client.0.smithi189.stderr:4+0 records out 2020-11-16T21:24:43.226 INFO:tasks.workunit.client.0.smithi189.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000230856 s, 17.7 MB/s 2020-11-16T21:24:43.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: seq 1 600 2020-11-16T21:24:43.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:43.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:43.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:43.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:24:43.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:43.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:24:43.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:43.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:43.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:43.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:24:43.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:43.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:24:43.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:43.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:43.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:43.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:24:43.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:43.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:24:43.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:43.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:43.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:43.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:24:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:43.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:24:43.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:43.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:44.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:44.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:24:44.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:44.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:24:44.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:44.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:44.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:44.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:24:44.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:44.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:24:44.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:44.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:44.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:44.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:24:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:24:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:44.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:44.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:44.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:24:44.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:44.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:24:44.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:44.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:44.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:44.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:24:44.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:44.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:24:44.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:44.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:44.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:44.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:24:45.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:45.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:24:45.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:45.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:45.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:45.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:24:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:45.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:24:45.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:45.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:45.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:45.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:24:45.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:45.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:24:45.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:45.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:45.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:45.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:24:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:45.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:24:45.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:45.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:45.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:45.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:24:45.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:45.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:24:45.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:45.906 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:24:45.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:45.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:24:46.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:46.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:46.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:46.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:24:46.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:46.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:24:46.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:46.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:46.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:46.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:24:46.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:46.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:24:46.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:46.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:46.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:46.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:24:46.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:46.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:24:46.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:46.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:46.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:46.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:24:46.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:46.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:24:46.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:46.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:46.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:46.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:24:46.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:46.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:24:47.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:47.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:47.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:47.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:24:47.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:47.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:24:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:47.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:47.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:47.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:24:47.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:47.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:24:47.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:47.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:47.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:47.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:24:47.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:47.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:24:47.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:47.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:47.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:47.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:24:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:24:47.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:47.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:47.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:47.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:24:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:47.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:24:48.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:48.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:48.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:48.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:24:48.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:48.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:24:48.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:48.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:24:48.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:48.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:24:48.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:48.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:48.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:48.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:24:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:24:48.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:48.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:48.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:48.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:24:48.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:48.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:24:48.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:48.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:24:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:48.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:24:49.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:49.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:49.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:49.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:24:49.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:49.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:24:49.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:49.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:49.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:49.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:24:49.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:49.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:24:49.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:49.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:49.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:49.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:24:49.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:49.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:24:49.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:49.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:49.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:49.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:24:49.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:49.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:24:49.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:49.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:49.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:49.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:24:49.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:49.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:24:49.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:49.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:24:50.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:50.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:24:50.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:50.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:50.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:50.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:24:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:24:50.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:50.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:50.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:50.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:24:50.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:50.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:24:50.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:50.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:50.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:50.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:24:50.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:50.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:24:50.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:50.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:50.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:50.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:24:50.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:50.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:24:50.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:50.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:50.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:50.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:24:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:24:51.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:51.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:51.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:51.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:24:51.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:51.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:24:51.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:51.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:51.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:51.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:24:51.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:51.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:24:51.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:51.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:51.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:51.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:24:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:24:51.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:51.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:51.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:51.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:24:51.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:51.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:24:51.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:51.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:51.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:51.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:24:51.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:51.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:24:52.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:52.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:52.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:52.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:24:52.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:52.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:24:52.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:52.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:52.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:52.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:24:52.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:52.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:24:52.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:52.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:52.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:52.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:24:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:52.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:24:52.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:52.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:52.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:52.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:24:52.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:52.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:24:52.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:52.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:52.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:52.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:24:52.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:52.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:24:53.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:53.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:53.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:53.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:24:53.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:53.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:24:53.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:53.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:53.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:53.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:24:53.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:53.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:24:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:53.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:24:53.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:53.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:24:53.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:53.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:53.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:53.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:24:53.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:53.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:24:53.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:53.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:53.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:53.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:24:53.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:53.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:24:54.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:54.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:54.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:54.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:24:54.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:54.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:24:54.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:54.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:54.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:54.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:24:54.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:54.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:24:54.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:54.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:54.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:54.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:24:54.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:54.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:24:54.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:54.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:54.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:54.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:24:54.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:54.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:24:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:54.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:24:54.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:54.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:24:55.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:55.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:55.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:55.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:24:55.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:24:55.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:55.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:55.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:55.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:24:55.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:55.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:24:55.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:55.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:24:55.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:55.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:24:55.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:55.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:24:55.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:55.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:24:55.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:55.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:55.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:55.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:24:55.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:55.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:24:55.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:55.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:55.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:55.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:24:56.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:56.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:24:56.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:56.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:56.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:56.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:24:56.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:56.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:24:56.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:56.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:56.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:56.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:24:56.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:56.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:24:56.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:56.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:56.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:56.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:24:56.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:56.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:24:56.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:56.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:56.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:56.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:24:56.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:56.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:24:56.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:56.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:56.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:56.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:24:57.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:57.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:24:57.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:57.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:57.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:57.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:24:57.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:24:57.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:57.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:57.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:57.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:24:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:57.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:24:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:57.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:24:57.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:57.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:24:57.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:57.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:57.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:57.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:24:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:57.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:24:57.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:24:57.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:24:57.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:57.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:24:57.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:24:57.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:24:58.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:25:02.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:25:02.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:25:02.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:25:02.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:25:02.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:25:02.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:25:02.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:25:02.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:25:02.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:25:02.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:25:02.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:25:02.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:25:02.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:25:02.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:25:02.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:25:02.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:25:02.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:25:02.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:25:02.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:25:02.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:25:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:25:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:25:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:25:02.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:25:02.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:25:02.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:25:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:25:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:25:02.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:25:02.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:25:02.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:25:02.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:25:02.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:25:02.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:25:02.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:25:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:25:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:25:02.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:25:02.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:25:02.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:25:02.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:25:02.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:25:02.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:25:02.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:25:02.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:25:02.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:25:02.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:25:02.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:25:02.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:25:02.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:02.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:02.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:25:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:25:03.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:03.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:03.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:03.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:25:03.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:03.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:25:03.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:03.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:03.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:03.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:25:03.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:03.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:25:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:03.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:25:03.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:03.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:25:03.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:03.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:03.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:03.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:25:03.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:03.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:25:03.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:03.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:03.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:03.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:25:04.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:04.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:25:04.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:04.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:04.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:04.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:25:04.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:04.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:25:04.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:04.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:04.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:04.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:25:04.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:04.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:25:04.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:04.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:04.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:04.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:25:04.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:04.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:25:04.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:04.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:04.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:04.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:25:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:25:04.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:04.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:04.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:04.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:25:04.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:04.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:25:05.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:05.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:05.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:05.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:25:05.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:05.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:25:05.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:05.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:05.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:05.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:25:05.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:05.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:25:05.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:05.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:05.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:05.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:25:05.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:05.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:25:05.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:05.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:05.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:05.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:25:05.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:05.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:25:05.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:05.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:05.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:05.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:25:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:25:06.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:06.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:06.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:06.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:25:06.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:06.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:25:06.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:06.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:06.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:06.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:25:06.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:06.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:25:06.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:06.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:06.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:06.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:25:06.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:06.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:25:06.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:06.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:06.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:06.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:25:06.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:06.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:25:06.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:06.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:25:06.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:06.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:25:07.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:07.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:07.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:07.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:25:07.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:07.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:25:07.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:07.194 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:07.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:07.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:25:07.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:07.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:25:07.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:07.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:07.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:07.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:25:07.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:07.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:25:07.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:07.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:07.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:07.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:25:07.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:07.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:25:07.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:07.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:07.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:07.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:25:07.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:07.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:25:07.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:07.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:25:08.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:08.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:25:08.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:08.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:08.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:08.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:25:08.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:08.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:25:08.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:08.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:08.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:08.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:25:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:25:08.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:08.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:25:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:25:08.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:08.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:08.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:08.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:25:08.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:08.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:25:08.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:08.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:08.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:08.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:25:09.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:09.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:25:09.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:09.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:09.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:09.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:25:09.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:09.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:25:09.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:09.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:09.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:09.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:25:09.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:09.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:25:09.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:09.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:09.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:09.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:25:09.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:09.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:25:09.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:09.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:09.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:09.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:25:09.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:09.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:25:09.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:09.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:09.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:09.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:25:10.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:10.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:25:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:10.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:10.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:10.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:25:10.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:10.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:25:10.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:10.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:25:10.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:10.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:25:10.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:10.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:10.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:10.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:25:10.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:10.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:25:10.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:10.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:10.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:10.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:25:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:25:10.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:10.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:10.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:10.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:25:10.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:10.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:25:11.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:11.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:11.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:11.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:25:11.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:11.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:25:11.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:11.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:11.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:11.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:25:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:11.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:25:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:11.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:11.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:11.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:25:11.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:11.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:25:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:11.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:11.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:11.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:25:11.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:11.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:25:11.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:11.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:25:11.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:11.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:25:11.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:11.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:11.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:11.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:25:12.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:12.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:25:12.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:12.185 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:12.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:12.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:25:12.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:12.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:25:12.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:12.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:12.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:12.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:25:12.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:12.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:25:12.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:12.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:12.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:12.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:25:12.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:12.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:25:12.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:12.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:12.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:12.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:25:12.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:12.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:25:12.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:12.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:12.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:12.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:25:13.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:13.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:25:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:13.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:13.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:13.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:25:13.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:13.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:25:13.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:13.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:13.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:13.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:25:13.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:13.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:25:13.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:13.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:13.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:13.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:25:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:25:14.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:14.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:14.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:25:14.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:14.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:25:14.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:14.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:14.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:14.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:25:14.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:14.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:25:14.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:14.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:14.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:14.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:25:14.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:14.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:25:14.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:14.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:14.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:14.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:25:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:25:14.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:14.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:14.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:14.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:25:14.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:14.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:25:14.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:14.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:14.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:14.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:25:15.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:15.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:25:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:15.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:25:15.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:15.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:25:15.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:15.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:15.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:15.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:25:15.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:15.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:25:15.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:15.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:15.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:15.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:25:15.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:15.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:25:15.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:15.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:15.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:15.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:25:15.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:15.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:25:15.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:15.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:15.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:15.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:25:16.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:16.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:25:16.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:16.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:16.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:16.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:25:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:25:16.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:16.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:16.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:16.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:25:16.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:16.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:25:16.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:16.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:16.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:16.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:25:16.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:16.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:25:16.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:16.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:16.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:16.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:25:16.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:16.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:25:16.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:16.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:16.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:16.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:25:16.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:16.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:25:17.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:17.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:17.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:17.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:25:17.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:17.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:25:17.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:17.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:17.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:17.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:25:17.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:17.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:25:17.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:17.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:17.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:17.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:25:17.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:17.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:25:17.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:17.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:17.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:17.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:25:17.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:17.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:25:17.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:17.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:17.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:17.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:25:17.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:17.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:25:18.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:18.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:18.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:18.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:25:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:25:18.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:18.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:18.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:18.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:25:18.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:18.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:25:18.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:18.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:18.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:18.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:25:18.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:18.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:25:18.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:18.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:18.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:18.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:25:18.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:18.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:25:18.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:18.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:18.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:25:18.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:18.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:25:18.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:18.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:19.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:19.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:25:19.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:19.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:25:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:19.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:25:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:25:19.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:19.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:19.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:19.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:25:19.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:19.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:25:19.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:19.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:19.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:19.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:25:19.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:19.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:25:19.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:19.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:19.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:19.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:25:19.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:19.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:25:19.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:19.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:19.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:19.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:25:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:25:20.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:20.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:20.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:20.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:25:20.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:20.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:25:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:20.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:25:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:25:20.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:20.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:20.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:20.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:25:20.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:20.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:25:20.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:20.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:20.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:25:20.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:20.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:25:20.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:20.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:20.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:25:21.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:21.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:25:21.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:21.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:21.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:21.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:25:21.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:21.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:25:21.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:21.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:21.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:21.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:25:21.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:21.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:25:21.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:21.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:21.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:21.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:25:21.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:21.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:25:21.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:21.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:21.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:25:21.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:21.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:25:21.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:21.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:21.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:21.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:25:21.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:21.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:25:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:22.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:25:22.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:22.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj201 td/osd-backfill-space/datafile 2020-11-16T21:25:22.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:22.235 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:22.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:22.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:25:22.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:22.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj202 td/osd-backfill-space/datafile 2020-11-16T21:25:22.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:22.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:22.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:22.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:25:22.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:22.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj203 td/osd-backfill-space/datafile 2020-11-16T21:25:22.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:22.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:22.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:22.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:25:22.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:22.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj204 td/osd-backfill-space/datafile 2020-11-16T21:25:22.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:22.779 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:22.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:22.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:25:22.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:22.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj205 td/osd-backfill-space/datafile 2020-11-16T21:25:22.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:22.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:22.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:22.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:25:23.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:23.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj206 td/osd-backfill-space/datafile 2020-11-16T21:25:23.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:23.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:23.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:23.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:25:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj207 td/osd-backfill-space/datafile 2020-11-16T21:25:23.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:23.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:23.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:23.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:25:23.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:23.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj208 td/osd-backfill-space/datafile 2020-11-16T21:25:23.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:23.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:23.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:23.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:25:23.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:23.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj209 td/osd-backfill-space/datafile 2020-11-16T21:25:23.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:23.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:23.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:23.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:25:23.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:23.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj210 td/osd-backfill-space/datafile 2020-11-16T21:25:23.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:23.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:23.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:23.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:25:24.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:24.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj211 td/osd-backfill-space/datafile 2020-11-16T21:25:24.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:24.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:24.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:24.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:25:24.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:24.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj212 td/osd-backfill-space/datafile 2020-11-16T21:25:24.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:24.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:24.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:24.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:25:24.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:24.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj213 td/osd-backfill-space/datafile 2020-11-16T21:25:24.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:24.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:24.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:24.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:25:24.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:24.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj214 td/osd-backfill-space/datafile 2020-11-16T21:25:24.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:24.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:24.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:24.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:25:24.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:24.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj215 td/osd-backfill-space/datafile 2020-11-16T21:25:24.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:24.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:24.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:24.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:25:25.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:25.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj216 td/osd-backfill-space/datafile 2020-11-16T21:25:25.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:25.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:25.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:25:25.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:25.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj217 td/osd-backfill-space/datafile 2020-11-16T21:25:25.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:25.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:25.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:25:25.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:25.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj218 td/osd-backfill-space/datafile 2020-11-16T21:25:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:25.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:25.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:25.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:25:25.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:25.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj219 td/osd-backfill-space/datafile 2020-11-16T21:25:25.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:25.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:25.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:25.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:25:25.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:25.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj220 td/osd-backfill-space/datafile 2020-11-16T21:25:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:25.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:25.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:25:25.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:25.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj221 td/osd-backfill-space/datafile 2020-11-16T21:25:26.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:26.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:26.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:26.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:25:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj222 td/osd-backfill-space/datafile 2020-11-16T21:25:26.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:26.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:26.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:26.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:25:26.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:26.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj223 td/osd-backfill-space/datafile 2020-11-16T21:25:26.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:26.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:26.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:26.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:25:26.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:26.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj224 td/osd-backfill-space/datafile 2020-11-16T21:25:26.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:26.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:26.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:26.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:25:26.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:26.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj225 td/osd-backfill-space/datafile 2020-11-16T21:25:26.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:26.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:26.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:26.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:25:26.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:26.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj226 td/osd-backfill-space/datafile 2020-11-16T21:25:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:27.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:25:27.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:27.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj227 td/osd-backfill-space/datafile 2020-11-16T21:25:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:27.185 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:25:27.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:27.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj228 td/osd-backfill-space/datafile 2020-11-16T21:25:27.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:27.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:27.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:27.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:25:27.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:27.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj229 td/osd-backfill-space/datafile 2020-11-16T21:25:27.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:27.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:27.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:27.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:25:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj230 td/osd-backfill-space/datafile 2020-11-16T21:25:27.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:27.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:27.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:27.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:25:27.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:27.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj231 td/osd-backfill-space/datafile 2020-11-16T21:25:27.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:27.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:27.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:27.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:25:28.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:28.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj232 td/osd-backfill-space/datafile 2020-11-16T21:25:28.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:28.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:28.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:28.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:25:28.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:28.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj233 td/osd-backfill-space/datafile 2020-11-16T21:25:28.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:28.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:25:28.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:28.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj234 td/osd-backfill-space/datafile 2020-11-16T21:25:28.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:28.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:28.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:28.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:25:28.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:28.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj235 td/osd-backfill-space/datafile 2020-11-16T21:25:28.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:28.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:28.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:28.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:25:28.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:28.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj236 td/osd-backfill-space/datafile 2020-11-16T21:25:28.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:28.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:28.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:28.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:25:28.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:28.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj237 td/osd-backfill-space/datafile 2020-11-16T21:25:29.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:29.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:29.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:29.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:25:29.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:29.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj238 td/osd-backfill-space/datafile 2020-11-16T21:25:29.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:29.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:29.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:29.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:25:29.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:29.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj239 td/osd-backfill-space/datafile 2020-11-16T21:25:29.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:29.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:29.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:29.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:25:29.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:29.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj240 td/osd-backfill-space/datafile 2020-11-16T21:25:29.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:29.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:29.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:29.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:25:29.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:29.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj241 td/osd-backfill-space/datafile 2020-11-16T21:25:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:29.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:25:29.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:29.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj242 td/osd-backfill-space/datafile 2020-11-16T21:25:30.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:30.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:30.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:30.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:25:30.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:30.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj243 td/osd-backfill-space/datafile 2020-11-16T21:25:30.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:30.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:30.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:30.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:25:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj244 td/osd-backfill-space/datafile 2020-11-16T21:25:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:30.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:30.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:30.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:25:30.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:30.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj245 td/osd-backfill-space/datafile 2020-11-16T21:25:30.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:30.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:30.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:30.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:25:30.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:30.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj246 td/osd-backfill-space/datafile 2020-11-16T21:25:30.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:30.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:30.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:30.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:25:30.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:30.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj247 td/osd-backfill-space/datafile 2020-11-16T21:25:31.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:31.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:31.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:31.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:25:31.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:31.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj248 td/osd-backfill-space/datafile 2020-11-16T21:25:31.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:31.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:31.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:31.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:25:31.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:31.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj249 td/osd-backfill-space/datafile 2020-11-16T21:25:31.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:31.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:31.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:31.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:25:31.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:31.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj250 td/osd-backfill-space/datafile 2020-11-16T21:25:31.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:31.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:31.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:31.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:25:31.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:31.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj251 td/osd-backfill-space/datafile 2020-11-16T21:25:31.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:31.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:31.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:31.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:25:31.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:31.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj252 td/osd-backfill-space/datafile 2020-11-16T21:25:31.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:31.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:31.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:31.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:25:32.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:32.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj253 td/osd-backfill-space/datafile 2020-11-16T21:25:32.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:32.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:32.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:32.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:25:32.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:32.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj254 td/osd-backfill-space/datafile 2020-11-16T21:25:32.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:32.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:32.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:32.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:25:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:32.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj255 td/osd-backfill-space/datafile 2020-11-16T21:25:32.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:32.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:32.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:32.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:25:32.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:32.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj256 td/osd-backfill-space/datafile 2020-11-16T21:25:32.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:32.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:32.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:32.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:25:32.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:32.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj257 td/osd-backfill-space/datafile 2020-11-16T21:25:32.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:32.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:32.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:32.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:25:33.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:33.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj258 td/osd-backfill-space/datafile 2020-11-16T21:25:33.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:33.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:33.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:33.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:25:33.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:33.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj259 td/osd-backfill-space/datafile 2020-11-16T21:25:33.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:33.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:33.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:33.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:25:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj260 td/osd-backfill-space/datafile 2020-11-16T21:25:33.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:33.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:33.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:33.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:25:33.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:33.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj261 td/osd-backfill-space/datafile 2020-11-16T21:25:33.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:33.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:33.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:25:33.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:33.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj262 td/osd-backfill-space/datafile 2020-11-16T21:25:33.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:33.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:33.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:25:33.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:33.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj263 td/osd-backfill-space/datafile 2020-11-16T21:25:34.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:34.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:25:34.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:34.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj264 td/osd-backfill-space/datafile 2020-11-16T21:25:34.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:34.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:34.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:25:34.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:34.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj265 td/osd-backfill-space/datafile 2020-11-16T21:25:34.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:34.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:34.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:34.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:25:34.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:34.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj266 td/osd-backfill-space/datafile 2020-11-16T21:25:34.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:34.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:34.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:34.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:25:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:34.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj267 td/osd-backfill-space/datafile 2020-11-16T21:25:34.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:34.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:34.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:34.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:25:35.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:35.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj268 td/osd-backfill-space/datafile 2020-11-16T21:25:35.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:35.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:35.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:35.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:25:35.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:35.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj269 td/osd-backfill-space/datafile 2020-11-16T21:25:35.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:35.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:25:35.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:35.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj270 td/osd-backfill-space/datafile 2020-11-16T21:25:35.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:35.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:35.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:35.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:25:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj271 td/osd-backfill-space/datafile 2020-11-16T21:25:35.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:35.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:35.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:35.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:25:35.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:35.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj272 td/osd-backfill-space/datafile 2020-11-16T21:25:35.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:35.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:35.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:35.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:25:36.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:36.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj273 td/osd-backfill-space/datafile 2020-11-16T21:25:36.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:36.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:36.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:36.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:25:36.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:36.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj274 td/osd-backfill-space/datafile 2020-11-16T21:25:36.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:36.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:36.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:25:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj275 td/osd-backfill-space/datafile 2020-11-16T21:25:36.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:36.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:36.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:25:36.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:36.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj276 td/osd-backfill-space/datafile 2020-11-16T21:25:36.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:36.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:36.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:36.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:25:36.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:36.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj277 td/osd-backfill-space/datafile 2020-11-16T21:25:36.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:36.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:36.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:36.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:25:36.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:36.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj278 td/osd-backfill-space/datafile 2020-11-16T21:25:37.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:37.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:37.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:37.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:25:37.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:37.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj279 td/osd-backfill-space/datafile 2020-11-16T21:25:37.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:37.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:37.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:37.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:25:37.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:37.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj280 td/osd-backfill-space/datafile 2020-11-16T21:25:37.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:37.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:37.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:37.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:25:37.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:37.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj281 td/osd-backfill-space/datafile 2020-11-16T21:25:37.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:37.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:37.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:37.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:25:37.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:37.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj282 td/osd-backfill-space/datafile 2020-11-16T21:25:37.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:37.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:37.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:37.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:25:37.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:37.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj283 td/osd-backfill-space/datafile 2020-11-16T21:25:38.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:38.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:38.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:38.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:25:38.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:38.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj284 td/osd-backfill-space/datafile 2020-11-16T21:25:38.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:38.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:38.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:38.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:25:38.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:38.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj285 td/osd-backfill-space/datafile 2020-11-16T21:25:38.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:38.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:38.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:38.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:25:38.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:38.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj286 td/osd-backfill-space/datafile 2020-11-16T21:25:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:38.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:38.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:25:38.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:38.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj287 td/osd-backfill-space/datafile 2020-11-16T21:25:38.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:38.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:38.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:38.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:25:38.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:38.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj288 td/osd-backfill-space/datafile 2020-11-16T21:25:38.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:38.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:38.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:38.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:25:39.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:39.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj289 td/osd-backfill-space/datafile 2020-11-16T21:25:39.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:39.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:39.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:39.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:25:39.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:39.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj290 td/osd-backfill-space/datafile 2020-11-16T21:25:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:39.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:25:39.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:39.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj291 td/osd-backfill-space/datafile 2020-11-16T21:25:39.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:39.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:39.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:39.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:25:39.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:39.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj292 td/osd-backfill-space/datafile 2020-11-16T21:25:39.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:39.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:39.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:39.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:25:39.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:39.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj293 td/osd-backfill-space/datafile 2020-11-16T21:25:39.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:39.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:39.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:39.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:25:40.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:40.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj294 td/osd-backfill-space/datafile 2020-11-16T21:25:40.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:40.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:40.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:40.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:25:40.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:40.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj295 td/osd-backfill-space/datafile 2020-11-16T21:25:40.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:40.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:40.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:40.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:25:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj296 td/osd-backfill-space/datafile 2020-11-16T21:25:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:40.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:40.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:40.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:25:40.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:40.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj297 td/osd-backfill-space/datafile 2020-11-16T21:25:40.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:40.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:40.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:40.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:25:40.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:40.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj298 td/osd-backfill-space/datafile 2020-11-16T21:25:40.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:40.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:40.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:40.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:25:40.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:40.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj299 td/osd-backfill-space/datafile 2020-11-16T21:25:41.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:41.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:41.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:41.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:25:41.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:41.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj300 td/osd-backfill-space/datafile 2020-11-16T21:25:41.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:41.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:41.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:41.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:25:41.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:41.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj301 td/osd-backfill-space/datafile 2020-11-16T21:25:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:41.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:25:41.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:41.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj302 td/osd-backfill-space/datafile 2020-11-16T21:25:41.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:41.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:41.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:41.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:25:41.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:41.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj303 td/osd-backfill-space/datafile 2020-11-16T21:25:41.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:41.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:41.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:41.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:25:41.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:41.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj304 td/osd-backfill-space/datafile 2020-11-16T21:25:42.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:42.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:42.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:42.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:25:42.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:42.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj305 td/osd-backfill-space/datafile 2020-11-16T21:25:42.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:42.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:42.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:42.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:25:42.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:42.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj306 td/osd-backfill-space/datafile 2020-11-16T21:25:42.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:42.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:42.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:42.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:25:42.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:42.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj307 td/osd-backfill-space/datafile 2020-11-16T21:25:42.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:42.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:25:42.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:42.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj308 td/osd-backfill-space/datafile 2020-11-16T21:25:42.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:42.779 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:42.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:42.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:25:42.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:42.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj309 td/osd-backfill-space/datafile 2020-11-16T21:25:42.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:42.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:42.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:42.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:25:43.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:43.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj310 td/osd-backfill-space/datafile 2020-11-16T21:25:43.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:43.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:25:43.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:43.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj311 td/osd-backfill-space/datafile 2020-11-16T21:25:43.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:43.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:43.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:43.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:25:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj312 td/osd-backfill-space/datafile 2020-11-16T21:25:43.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:43.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:43.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:43.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:25:43.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:43.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj313 td/osd-backfill-space/datafile 2020-11-16T21:25:43.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:43.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:43.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:43.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:25:43.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:43.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj314 td/osd-backfill-space/datafile 2020-11-16T21:25:43.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:43.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:43.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:43.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:25:44.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:44.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj315 td/osd-backfill-space/datafile 2020-11-16T21:25:44.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:44.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:44.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:44.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:25:44.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:44.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj316 td/osd-backfill-space/datafile 2020-11-16T21:25:44.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:44.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:44.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:44.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:25:44.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:44.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj317 td/osd-backfill-space/datafile 2020-11-16T21:25:44.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:44.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:44.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:44.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:25:44.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:44.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj318 td/osd-backfill-space/datafile 2020-11-16T21:25:45.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:45.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:45.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:45.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:25:45.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:45.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj319 td/osd-backfill-space/datafile 2020-11-16T21:25:45.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:45.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:45.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:45.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:25:45.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:45.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj320 td/osd-backfill-space/datafile 2020-11-16T21:25:45.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:45.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:45.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:45.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:25:45.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:45.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj321 td/osd-backfill-space/datafile 2020-11-16T21:25:45.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:45.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:45.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:45.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:25:45.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:45.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj322 td/osd-backfill-space/datafile 2020-11-16T21:25:45.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:45.788 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:45.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:45.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:25:45.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:45.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj323 td/osd-backfill-space/datafile 2020-11-16T21:25:45.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:45.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:45.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:45.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:25:46.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:46.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj324 td/osd-backfill-space/datafile 2020-11-16T21:25:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:46.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:46.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:25:46.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:46.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj325 td/osd-backfill-space/datafile 2020-11-16T21:25:46.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:46.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:25:46.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:46.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj326 td/osd-backfill-space/datafile 2020-11-16T21:25:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:46.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:46.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:46.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:25:46.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:46.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj327 td/osd-backfill-space/datafile 2020-11-16T21:25:46.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:46.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:46.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:46.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:25:46.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:46.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj328 td/osd-backfill-space/datafile 2020-11-16T21:25:46.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:46.908 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:46.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:46.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:25:47.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:47.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj329 td/osd-backfill-space/datafile 2020-11-16T21:25:47.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:47.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:47.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:47.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:25:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj330 td/osd-backfill-space/datafile 2020-11-16T21:25:47.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:47.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:47.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:47.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:25:47.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:47.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj331 td/osd-backfill-space/datafile 2020-11-16T21:25:47.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:47.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:47.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:47.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:25:47.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:47.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj332 td/osd-backfill-space/datafile 2020-11-16T21:25:47.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:47.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:47.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:47.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:25:47.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:47.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj333 td/osd-backfill-space/datafile 2020-11-16T21:25:47.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:47.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:47.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:47.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:25:47.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:47.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj334 td/osd-backfill-space/datafile 2020-11-16T21:25:48.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:48.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:48.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:48.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:25:48.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:48.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj335 td/osd-backfill-space/datafile 2020-11-16T21:25:48.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:48.230 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:25:48.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:48.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj336 td/osd-backfill-space/datafile 2020-11-16T21:25:48.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:48.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:48.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:48.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:25:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj337 td/osd-backfill-space/datafile 2020-11-16T21:25:48.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:48.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:48.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:48.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:25:48.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:48.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj338 td/osd-backfill-space/datafile 2020-11-16T21:25:48.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:48.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:48.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:48.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:25:48.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:48.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj339 td/osd-backfill-space/datafile 2020-11-16T21:25:49.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:49.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:49.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:49.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:25:49.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:49.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj340 td/osd-backfill-space/datafile 2020-11-16T21:25:49.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:49.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:49.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:49.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:25:49.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:49.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj341 td/osd-backfill-space/datafile 2020-11-16T21:25:49.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:49.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:49.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:25:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj342 td/osd-backfill-space/datafile 2020-11-16T21:25:49.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:49.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:49.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:49.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:25:49.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:49.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj343 td/osd-backfill-space/datafile 2020-11-16T21:25:49.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:49.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:49.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:49.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:25:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj344 td/osd-backfill-space/datafile 2020-11-16T21:25:50.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:50.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:50.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:50.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:25:50.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:50.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj345 td/osd-backfill-space/datafile 2020-11-16T21:25:50.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:50.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:50.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:50.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:25:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj346 td/osd-backfill-space/datafile 2020-11-16T21:25:50.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:50.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:50.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:50.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:25:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj347 td/osd-backfill-space/datafile 2020-11-16T21:25:50.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:50.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:50.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:50.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:25:50.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:50.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj348 td/osd-backfill-space/datafile 2020-11-16T21:25:50.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:50.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:50.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:50.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:25:50.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:50.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj349 td/osd-backfill-space/datafile 2020-11-16T21:25:51.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:25:54.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj350 td/osd-backfill-space/datafile 2020-11-16T21:25:54.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:25:54.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj351 td/osd-backfill-space/datafile 2020-11-16T21:25:54.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:25:54.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj352 td/osd-backfill-space/datafile 2020-11-16T21:25:54.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:25:54.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj353 td/osd-backfill-space/datafile 2020-11-16T21:25:54.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:25:54.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj354 td/osd-backfill-space/datafile 2020-11-16T21:25:54.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:25:54.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj355 td/osd-backfill-space/datafile 2020-11-16T21:25:54.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:25:54.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj356 td/osd-backfill-space/datafile 2020-11-16T21:25:54.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:25:54.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj357 td/osd-backfill-space/datafile 2020-11-16T21:25:54.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:25:54.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj358 td/osd-backfill-space/datafile 2020-11-16T21:25:54.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:25:54.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj359 td/osd-backfill-space/datafile 2020-11-16T21:25:54.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.972 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:25:54.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj360 td/osd-backfill-space/datafile 2020-11-16T21:25:54.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:25:54.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj361 td/osd-backfill-space/datafile 2020-11-16T21:25:54.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:25:54.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj362 td/osd-backfill-space/datafile 2020-11-16T21:25:54.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:25:54.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj363 td/osd-backfill-space/datafile 2020-11-16T21:25:54.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:25:54.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj364 td/osd-backfill-space/datafile 2020-11-16T21:25:54.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:25:54.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj365 td/osd-backfill-space/datafile 2020-11-16T21:25:54.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:25:54.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj366 td/osd-backfill-space/datafile 2020-11-16T21:25:54.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:25:54.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj367 td/osd-backfill-space/datafile 2020-11-16T21:25:54.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:25:54.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj368 td/osd-backfill-space/datafile 2020-11-16T21:25:54.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:25:54.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj369 td/osd-backfill-space/datafile 2020-11-16T21:25:54.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:54.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:54.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:54.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:25:55.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:55.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj370 td/osd-backfill-space/datafile 2020-11-16T21:25:55.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:55.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:25:55.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:55.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj371 td/osd-backfill-space/datafile 2020-11-16T21:25:55.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:55.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:55.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:55.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:25:55.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:55.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj372 td/osd-backfill-space/datafile 2020-11-16T21:25:55.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:55.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:55.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:55.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:25:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj373 td/osd-backfill-space/datafile 2020-11-16T21:25:55.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:55.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:55.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:55.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:25:55.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:55.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj374 td/osd-backfill-space/datafile 2020-11-16T21:25:55.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:55.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:55.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:55.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:25:55.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:55.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj375 td/osd-backfill-space/datafile 2020-11-16T21:25:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:56.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:25:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj376 td/osd-backfill-space/datafile 2020-11-16T21:25:56.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:56.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:56.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:56.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:25:56.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:56.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj377 td/osd-backfill-space/datafile 2020-11-16T21:25:56.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:56.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:56.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:25:56.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:56.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj378 td/osd-backfill-space/datafile 2020-11-16T21:25:56.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:56.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:25:56.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:56.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj379 td/osd-backfill-space/datafile 2020-11-16T21:25:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:56.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:25:56.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:56.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj380 td/osd-backfill-space/datafile 2020-11-16T21:25:57.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:57.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:57.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:57.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:25:57.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:57.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj381 td/osd-backfill-space/datafile 2020-11-16T21:25:57.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:57.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:57.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:57.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:25:57.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:57.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj382 td/osd-backfill-space/datafile 2020-11-16T21:25:57.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:57.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:57.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:57.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:25:57.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:57.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj383 td/osd-backfill-space/datafile 2020-11-16T21:25:57.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:57.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:57.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:57.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:25:57.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:57.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj384 td/osd-backfill-space/datafile 2020-11-16T21:25:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:57.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:25:57.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:57.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj385 td/osd-backfill-space/datafile 2020-11-16T21:25:57.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:57.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:57.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:57.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:25:58.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:58.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj386 td/osd-backfill-space/datafile 2020-11-16T21:25:58.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:58.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:58.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:58.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:25:58.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:58.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj387 td/osd-backfill-space/datafile 2020-11-16T21:25:58.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:58.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:58.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:58.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:25:58.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:58.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj388 td/osd-backfill-space/datafile 2020-11-16T21:25:58.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:58.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:58.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:58.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:25:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:58.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj389 td/osd-backfill-space/datafile 2020-11-16T21:25:58.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:58.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:58.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:58.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:25:58.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:58.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj390 td/osd-backfill-space/datafile 2020-11-16T21:25:58.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:58.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:58.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:58.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:25:59.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:59.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj391 td/osd-backfill-space/datafile 2020-11-16T21:25:59.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:59.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:59.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:59.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:25:59.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:59.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj392 td/osd-backfill-space/datafile 2020-11-16T21:25:59.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:59.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:59.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:59.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:25:59.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:59.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj393 td/osd-backfill-space/datafile 2020-11-16T21:25:59.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:59.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:59.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:59.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:25:59.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:59.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj394 td/osd-backfill-space/datafile 2020-11-16T21:25:59.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:59.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:59.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:59.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:25:59.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:59.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj395 td/osd-backfill-space/datafile 2020-11-16T21:25:59.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:25:59.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:25:59.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:25:59.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:26:00.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:00.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj396 td/osd-backfill-space/datafile 2020-11-16T21:26:00.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:00.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:00.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:00.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:26:00.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:00.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj397 td/osd-backfill-space/datafile 2020-11-16T21:26:00.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:00.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:00.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:00.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:26:00.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:00.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj398 td/osd-backfill-space/datafile 2020-11-16T21:26:00.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:00.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:00.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:00.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:26:00.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:00.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj399 td/osd-backfill-space/datafile 2020-11-16T21:26:00.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:00.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:00.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:00.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:26:00.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:00.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj400 td/osd-backfill-space/datafile 2020-11-16T21:26:00.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:00.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:00.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:00.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:26:00.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:00.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj401 td/osd-backfill-space/datafile 2020-11-16T21:26:01.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:01.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:01.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:01.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:26:01.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:01.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj402 td/osd-backfill-space/datafile 2020-11-16T21:26:01.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:01.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:01.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:01.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:26:01.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:01.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj403 td/osd-backfill-space/datafile 2020-11-16T21:26:01.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:01.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:01.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:01.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:26:01.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:01.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj404 td/osd-backfill-space/datafile 2020-11-16T21:26:01.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:01.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:01.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:01.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:26:01.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:01.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj405 td/osd-backfill-space/datafile 2020-11-16T21:26:01.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:01.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:01.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:01.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:26:01.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:01.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj406 td/osd-backfill-space/datafile 2020-11-16T21:26:02.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:02.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:02.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:02.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:26:02.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:02.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj407 td/osd-backfill-space/datafile 2020-11-16T21:26:02.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:02.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:02.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:02.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:26:02.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:02.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj408 td/osd-backfill-space/datafile 2020-11-16T21:26:02.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:02.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:02.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:02.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:26:02.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:02.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj409 td/osd-backfill-space/datafile 2020-11-16T21:26:02.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:02.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:02.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:02.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:26:02.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:02.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj410 td/osd-backfill-space/datafile 2020-11-16T21:26:02.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:02.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:02.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:02.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:26:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:02.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj411 td/osd-backfill-space/datafile 2020-11-16T21:26:03.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:03.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:03.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:03.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:26:03.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:03.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj412 td/osd-backfill-space/datafile 2020-11-16T21:26:03.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:03.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:03.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:03.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:26:03.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:03.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj413 td/osd-backfill-space/datafile 2020-11-16T21:26:03.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:03.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:03.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:03.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:26:03.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:03.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj414 td/osd-backfill-space/datafile 2020-11-16T21:26:03.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:03.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:03.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:03.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:26:03.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:03.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj415 td/osd-backfill-space/datafile 2020-11-16T21:26:03.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:03.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:03.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:03.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:26:03.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:03.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj416 td/osd-backfill-space/datafile 2020-11-16T21:26:03.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:03.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:03.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:03.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:26:04.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:04.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj417 td/osd-backfill-space/datafile 2020-11-16T21:26:04.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:04.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:04.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:04.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:26:04.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:04.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj418 td/osd-backfill-space/datafile 2020-11-16T21:26:04.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:04.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:04.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:04.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:26:04.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:04.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj419 td/osd-backfill-space/datafile 2020-11-16T21:26:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:04.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:04.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:26:04.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:04.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj420 td/osd-backfill-space/datafile 2020-11-16T21:26:04.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:04.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:04.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:04.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:26:04.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:04.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj421 td/osd-backfill-space/datafile 2020-11-16T21:26:05.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:05.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:05.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:05.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:26:05.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:05.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj422 td/osd-backfill-space/datafile 2020-11-16T21:26:05.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:05.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:05.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:05.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:26:05.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:05.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj423 td/osd-backfill-space/datafile 2020-11-16T21:26:05.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:05.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:05.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:05.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:26:05.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:05.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj424 td/osd-backfill-space/datafile 2020-11-16T21:26:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:05.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:26:05.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:05.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj425 td/osd-backfill-space/datafile 2020-11-16T21:26:05.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:05.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:05.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:05.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:26:05.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:05.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj426 td/osd-backfill-space/datafile 2020-11-16T21:26:05.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:05.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:05.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:05.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:26:06.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:06.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj427 td/osd-backfill-space/datafile 2020-11-16T21:26:06.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:06.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:06.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:06.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:26:06.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:06.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj428 td/osd-backfill-space/datafile 2020-11-16T21:26:06.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:06.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:06.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:06.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:26:06.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:06.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj429 td/osd-backfill-space/datafile 2020-11-16T21:26:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:06.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:06.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:06.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:26:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj430 td/osd-backfill-space/datafile 2020-11-16T21:26:06.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:06.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:06.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:06.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:26:06.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:06.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj431 td/osd-backfill-space/datafile 2020-11-16T21:26:06.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:06.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:06.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:06.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:26:06.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:06.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj432 td/osd-backfill-space/datafile 2020-11-16T21:26:07.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:07.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:07.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:07.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:26:07.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:07.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj433 td/osd-backfill-space/datafile 2020-11-16T21:26:07.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:07.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:07.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:07.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:26:07.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:07.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj434 td/osd-backfill-space/datafile 2020-11-16T21:26:07.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:07.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:07.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:07.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:26:07.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:07.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj435 td/osd-backfill-space/datafile 2020-11-16T21:26:07.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:07.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:07.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:07.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:26:07.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:07.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj436 td/osd-backfill-space/datafile 2020-11-16T21:26:07.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:07.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:07.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:07.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:26:07.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:07.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj437 td/osd-backfill-space/datafile 2020-11-16T21:26:08.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:08.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:08.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:08.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:26:08.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:08.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj438 td/osd-backfill-space/datafile 2020-11-16T21:26:08.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:08.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:08.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:08.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:26:08.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:08.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj439 td/osd-backfill-space/datafile 2020-11-16T21:26:08.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:08.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:08.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:08.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:26:08.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:08.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj440 td/osd-backfill-space/datafile 2020-11-16T21:26:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:08.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:08.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:08.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:26:08.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:08.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj441 td/osd-backfill-space/datafile 2020-11-16T21:26:08.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:08.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:08.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:08.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:26:08.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:08.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj442 td/osd-backfill-space/datafile 2020-11-16T21:26:08.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:08.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:08.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:08.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:26:09.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:09.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj443 td/osd-backfill-space/datafile 2020-11-16T21:26:09.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:09.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:09.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:09.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:26:09.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:09.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj444 td/osd-backfill-space/datafile 2020-11-16T21:26:09.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:09.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:09.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:09.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:26:09.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:09.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj445 td/osd-backfill-space/datafile 2020-11-16T21:26:09.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:09.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:09.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:09.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:26:09.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:09.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj446 td/osd-backfill-space/datafile 2020-11-16T21:26:09.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:09.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:09.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:09.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:26:09.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:09.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj447 td/osd-backfill-space/datafile 2020-11-16T21:26:09.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:09.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:09.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:09.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:26:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj448 td/osd-backfill-space/datafile 2020-11-16T21:26:10.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:10.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:10.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:10.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:26:10.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:10.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj449 td/osd-backfill-space/datafile 2020-11-16T21:26:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:10.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:10.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:10.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:26:10.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:10.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj450 td/osd-backfill-space/datafile 2020-11-16T21:26:10.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:10.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:10.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:10.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:26:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj451 td/osd-backfill-space/datafile 2020-11-16T21:26:10.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:10.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:10.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:10.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:26:10.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:10.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj452 td/osd-backfill-space/datafile 2020-11-16T21:26:10.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:10.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:10.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:10.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:26:11.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:11.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj453 td/osd-backfill-space/datafile 2020-11-16T21:26:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:11.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:11.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:11.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:26:11.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:11.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj454 td/osd-backfill-space/datafile 2020-11-16T21:26:11.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:11.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:11.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:11.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:26:11.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:11.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj455 td/osd-backfill-space/datafile 2020-11-16T21:26:11.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:11.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:26:11.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:11.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj456 td/osd-backfill-space/datafile 2020-11-16T21:26:11.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:11.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:26:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:11.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj457 td/osd-backfill-space/datafile 2020-11-16T21:26:11.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:11.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:11.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:11.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:26:11.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:11.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj458 td/osd-backfill-space/datafile 2020-11-16T21:26:12.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:12.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:12.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:12.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:26:12.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:12.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj459 td/osd-backfill-space/datafile 2020-11-16T21:26:12.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:12.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:12.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:12.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:26:12.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:12.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj460 td/osd-backfill-space/datafile 2020-11-16T21:26:12.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:12.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:12.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:12.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:26:12.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:12.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj461 td/osd-backfill-space/datafile 2020-11-16T21:26:12.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:12.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:12.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:12.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:26:12.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:12.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj462 td/osd-backfill-space/datafile 2020-11-16T21:26:12.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:12.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:12.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:12.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:26:12.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:12.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj463 td/osd-backfill-space/datafile 2020-11-16T21:26:13.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:13.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:13.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:13.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:26:13.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:13.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj464 td/osd-backfill-space/datafile 2020-11-16T21:26:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:13.196 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:13.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:13.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:26:13.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:13.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj465 td/osd-backfill-space/datafile 2020-11-16T21:26:13.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:13.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:13.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:13.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:26:13.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:13.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj466 td/osd-backfill-space/datafile 2020-11-16T21:26:13.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:13.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:13.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:26:13.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:13.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj467 td/osd-backfill-space/datafile 2020-11-16T21:26:13.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:13.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:13.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:13.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:26:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:13.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj468 td/osd-backfill-space/datafile 2020-11-16T21:26:13.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:13.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:13.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:13.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:26:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:14.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj469 td/osd-backfill-space/datafile 2020-11-16T21:26:14.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:14.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:14.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:14.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:26:14.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:14.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj470 td/osd-backfill-space/datafile 2020-11-16T21:26:14.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:14.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:14.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:14.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:26:14.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:14.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj471 td/osd-backfill-space/datafile 2020-11-16T21:26:14.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:14.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:14.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:14.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:26:14.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:14.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj472 td/osd-backfill-space/datafile 2020-11-16T21:26:14.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:14.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:14.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:14.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:26:14.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:14.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj473 td/osd-backfill-space/datafile 2020-11-16T21:26:14.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:14.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:14.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:14.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:26:15.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:15.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj474 td/osd-backfill-space/datafile 2020-11-16T21:26:15.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:15.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:15.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:15.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:26:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:15.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj475 td/osd-backfill-space/datafile 2020-11-16T21:26:15.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:15.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:26:15.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:15.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj476 td/osd-backfill-space/datafile 2020-11-16T21:26:15.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:15.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:15.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:15.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:26:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj477 td/osd-backfill-space/datafile 2020-11-16T21:26:16.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:16.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:16.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:16.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:26:16.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:16.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj478 td/osd-backfill-space/datafile 2020-11-16T21:26:16.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:16.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:16.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:16.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:26:16.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:16.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj479 td/osd-backfill-space/datafile 2020-11-16T21:26:16.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:16.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:16.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:16.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:26:16.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:16.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj480 td/osd-backfill-space/datafile 2020-11-16T21:26:16.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:16.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:16.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:16.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:26:16.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:16.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj481 td/osd-backfill-space/datafile 2020-11-16T21:26:16.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:16.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:16.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:16.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:26:16.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:16.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj482 td/osd-backfill-space/datafile 2020-11-16T21:26:17.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:17.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:17.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:17.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:26:17.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:17.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj483 td/osd-backfill-space/datafile 2020-11-16T21:26:17.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:17.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:17.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:17.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:26:17.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:17.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj484 td/osd-backfill-space/datafile 2020-11-16T21:26:17.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:17.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:17.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:17.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:26:17.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:17.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj485 td/osd-backfill-space/datafile 2020-11-16T21:26:17.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:17.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:17.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:17.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:26:17.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:17.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj486 td/osd-backfill-space/datafile 2020-11-16T21:26:17.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:17.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:17.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:17.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:26:17.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:17.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj487 td/osd-backfill-space/datafile 2020-11-16T21:26:18.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:18.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:18.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:18.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:26:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj488 td/osd-backfill-space/datafile 2020-11-16T21:26:18.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:18.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:18.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:18.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:26:18.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:18.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj489 td/osd-backfill-space/datafile 2020-11-16T21:26:18.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:18.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:18.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:18.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:26:18.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:18.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj490 td/osd-backfill-space/datafile 2020-11-16T21:26:18.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:18.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:18.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:18.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:26:18.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:18.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj491 td/osd-backfill-space/datafile 2020-11-16T21:26:18.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:18.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:18.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:26:18.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:18.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj492 td/osd-backfill-space/datafile 2020-11-16T21:26:18.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:18.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:18.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:18.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:26:19.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:19.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj493 td/osd-backfill-space/datafile 2020-11-16T21:26:19.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:19.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:26:19.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:19.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj494 td/osd-backfill-space/datafile 2020-11-16T21:26:19.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:19.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:19.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:19.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:26:19.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:19.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj495 td/osd-backfill-space/datafile 2020-11-16T21:26:19.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:19.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:19.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:19.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:26:19.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:19.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj496 td/osd-backfill-space/datafile 2020-11-16T21:26:19.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:19.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:19.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:19.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:26:19.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:19.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj497 td/osd-backfill-space/datafile 2020-11-16T21:26:19.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:19.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:19.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:19.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:26:20.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:20.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj498 td/osd-backfill-space/datafile 2020-11-16T21:26:20.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:20.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:20.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:20.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:26:20.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:20.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj499 td/osd-backfill-space/datafile 2020-11-16T21:26:20.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:20.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:20.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:20.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:26:20.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:20.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj500 td/osd-backfill-space/datafile 2020-11-16T21:26:20.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:20.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:26:20.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:20.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj501 td/osd-backfill-space/datafile 2020-11-16T21:26:20.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:20.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:20.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:20.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:26:20.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:20.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj502 td/osd-backfill-space/datafile 2020-11-16T21:26:20.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:20.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:20.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:20.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:26:21.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:21.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj503 td/osd-backfill-space/datafile 2020-11-16T21:26:21.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:21.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:21.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:21.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:26:21.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:21.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj504 td/osd-backfill-space/datafile 2020-11-16T21:26:21.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:21.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:21.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:21.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:26:21.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:21.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj505 td/osd-backfill-space/datafile 2020-11-16T21:26:21.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:21.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:21.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:21.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:26:21.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:21.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj506 td/osd-backfill-space/datafile 2020-11-16T21:26:21.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:21.707 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:26:21.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:21.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj507 td/osd-backfill-space/datafile 2020-11-16T21:26:21.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:21.908 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:21.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:21.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:26:22.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:22.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj508 td/osd-backfill-space/datafile 2020-11-16T21:26:22.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:22.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:22.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:22.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:26:22.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:22.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj509 td/osd-backfill-space/datafile 2020-11-16T21:26:22.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:22.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:22.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:22.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:26:22.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:22.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj510 td/osd-backfill-space/datafile 2020-11-16T21:26:22.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:22.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:22.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:22.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:26:22.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:22.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj511 td/osd-backfill-space/datafile 2020-11-16T21:26:22.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:22.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:22.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:22.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:26:22.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:22.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj512 td/osd-backfill-space/datafile 2020-11-16T21:26:22.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:22.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:22.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:22.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:26:23.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:23.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj513 td/osd-backfill-space/datafile 2020-11-16T21:26:23.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:23.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:23.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:23.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:26:23.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:23.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj514 td/osd-backfill-space/datafile 2020-11-16T21:26:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:23.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:26:23.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:23.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj515 td/osd-backfill-space/datafile 2020-11-16T21:26:23.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:23.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:23.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:23.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:26:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj516 td/osd-backfill-space/datafile 2020-11-16T21:26:23.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:23.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:23.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:23.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:26:23.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:23.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj517 td/osd-backfill-space/datafile 2020-11-16T21:26:23.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:23.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:23.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:23.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:26:23.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:23.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj518 td/osd-backfill-space/datafile 2020-11-16T21:26:24.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:24.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:24.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:24.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:26:24.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:24.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj519 td/osd-backfill-space/datafile 2020-11-16T21:26:24.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:24.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:24.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:24.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:26:24.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:24.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj520 td/osd-backfill-space/datafile 2020-11-16T21:26:24.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:24.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:24.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:24.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:26:24.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:24.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj521 td/osd-backfill-space/datafile 2020-11-16T21:26:24.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:24.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:24.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:24.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:26:24.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:24.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj522 td/osd-backfill-space/datafile 2020-11-16T21:26:24.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:24.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:24.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:24.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:26:24.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:24.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj523 td/osd-backfill-space/datafile 2020-11-16T21:26:25.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:25.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:25.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:25.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:26:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:25.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj524 td/osd-backfill-space/datafile 2020-11-16T21:26:25.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:25.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:25.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:25.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:26:25.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:25.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj525 td/osd-backfill-space/datafile 2020-11-16T21:26:25.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:25.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:25.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:25.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:26:25.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:25.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj526 td/osd-backfill-space/datafile 2020-11-16T21:26:25.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:25.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:25.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:26:25.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:25.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj527 td/osd-backfill-space/datafile 2020-11-16T21:26:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:25.822 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:25.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:25.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:26:25.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:25.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj528 td/osd-backfill-space/datafile 2020-11-16T21:26:26.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:26.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:26.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:26.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:26:26.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:26.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj529 td/osd-backfill-space/datafile 2020-11-16T21:26:26.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:26.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:26.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:26.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:26:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj530 td/osd-backfill-space/datafile 2020-11-16T21:26:26.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:26.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:26.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:26.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:26:26.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:26.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj531 td/osd-backfill-space/datafile 2020-11-16T21:26:26.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:26.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:26.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:26.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:26:26.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:26.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj532 td/osd-backfill-space/datafile 2020-11-16T21:26:26.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:26.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:26.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:26.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:26:26.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:26.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj533 td/osd-backfill-space/datafile 2020-11-16T21:26:26.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:26.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:26.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:26.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:26:27.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:27.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj534 td/osd-backfill-space/datafile 2020-11-16T21:26:27.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:27.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:27.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:27.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:26:27.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:27.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj535 td/osd-backfill-space/datafile 2020-11-16T21:26:27.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:27.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:26:27.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:27.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj536 td/osd-backfill-space/datafile 2020-11-16T21:26:27.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:27.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:27.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:27.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:26:27.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:27.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj537 td/osd-backfill-space/datafile 2020-11-16T21:26:27.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:27.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:27.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:27.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:26:27.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:27.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj538 td/osd-backfill-space/datafile 2020-11-16T21:26:27.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:27.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:27.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:27.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:26:28.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:28.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj539 td/osd-backfill-space/datafile 2020-11-16T21:26:28.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:28.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:28.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:28.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:26:28.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:28.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj540 td/osd-backfill-space/datafile 2020-11-16T21:26:28.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:28.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:28.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:28.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:26:28.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:28.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj541 td/osd-backfill-space/datafile 2020-11-16T21:26:28.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:28.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:28.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:28.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:26:28.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:28.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj542 td/osd-backfill-space/datafile 2020-11-16T21:26:28.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:28.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:28.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:28.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:26:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj543 td/osd-backfill-space/datafile 2020-11-16T21:26:28.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:28.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:28.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:26:28.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:28.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj544 td/osd-backfill-space/datafile 2020-11-16T21:26:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:29.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:26:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj545 td/osd-backfill-space/datafile 2020-11-16T21:26:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:29.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:26:29.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj546 td/osd-backfill-space/datafile 2020-11-16T21:26:29.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:29.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:29.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:29.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:26:29.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:29.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj547 td/osd-backfill-space/datafile 2020-11-16T21:26:29.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:29.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:29.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:29.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:26:29.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:29.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj548 td/osd-backfill-space/datafile 2020-11-16T21:26:29.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:29.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:29.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:29.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:26:29.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:29.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj549 td/osd-backfill-space/datafile 2020-11-16T21:26:29.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:29.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:26:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:30.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj550 td/osd-backfill-space/datafile 2020-11-16T21:26:30.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:30.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:30.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:30.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:26:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj551 td/osd-backfill-space/datafile 2020-11-16T21:26:30.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:30.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:30.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:30.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:26:30.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:30.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj552 td/osd-backfill-space/datafile 2020-11-16T21:26:30.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:30.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:30.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:30.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:26:30.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:30.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj553 td/osd-backfill-space/datafile 2020-11-16T21:26:30.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:30.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:30.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:30.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:26:30.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:30.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj554 td/osd-backfill-space/datafile 2020-11-16T21:26:31.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:31.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:31.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:31.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:26:31.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:31.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj555 td/osd-backfill-space/datafile 2020-11-16T21:26:31.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:31.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:31.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:31.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:26:31.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:31.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj556 td/osd-backfill-space/datafile 2020-11-16T21:26:31.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:31.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:31.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:31.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:26:31.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:31.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj557 td/osd-backfill-space/datafile 2020-11-16T21:26:31.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:31.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:31.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:31.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:26:31.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:31.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj558 td/osd-backfill-space/datafile 2020-11-16T21:26:31.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:31.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:31.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:31.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:26:31.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:31.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj559 td/osd-backfill-space/datafile 2020-11-16T21:26:31.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:31.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:31.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:31.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:26:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj560 td/osd-backfill-space/datafile 2020-11-16T21:26:32.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:32.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:32.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:32.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:26:32.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:32.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj561 td/osd-backfill-space/datafile 2020-11-16T21:26:32.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:32.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:32.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:32.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:26:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj562 td/osd-backfill-space/datafile 2020-11-16T21:26:32.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:32.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:32.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:32.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:26:32.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:32.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj563 td/osd-backfill-space/datafile 2020-11-16T21:26:32.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:32.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:32.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:32.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:26:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj564 td/osd-backfill-space/datafile 2020-11-16T21:26:32.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:32.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:26:33.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:33.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj565 td/osd-backfill-space/datafile 2020-11-16T21:26:33.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:33.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:33.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:33.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:26:33.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:33.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj566 td/osd-backfill-space/datafile 2020-11-16T21:26:33.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:33.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:33.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:33.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:26:33.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:33.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj567 td/osd-backfill-space/datafile 2020-11-16T21:26:33.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:33.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:33.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:33.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:26:33.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:33.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj568 td/osd-backfill-space/datafile 2020-11-16T21:26:33.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:33.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:33.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:33.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:26:33.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:33.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj569 td/osd-backfill-space/datafile 2020-11-16T21:26:33.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:33.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:33.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:33.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:26:33.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:33.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj570 td/osd-backfill-space/datafile 2020-11-16T21:26:34.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:34.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:34.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:34.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:26:34.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:34.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj571 td/osd-backfill-space/datafile 2020-11-16T21:26:34.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:34.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:34.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:34.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:26:34.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:34.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj572 td/osd-backfill-space/datafile 2020-11-16T21:26:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:34.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:34.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:34.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:26:34.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:34.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj573 td/osd-backfill-space/datafile 2020-11-16T21:26:34.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:34.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:34.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:34.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:26:34.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:34.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj574 td/osd-backfill-space/datafile 2020-11-16T21:26:34.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:34.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:34.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:34.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:26:34.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:34.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj575 td/osd-backfill-space/datafile 2020-11-16T21:26:35.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:35.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:35.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:35.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:26:35.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:35.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj576 td/osd-backfill-space/datafile 2020-11-16T21:26:35.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:35.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:35.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:35.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:26:35.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:35.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj577 td/osd-backfill-space/datafile 2020-11-16T21:26:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:35.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:26:35.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:35.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj578 td/osd-backfill-space/datafile 2020-11-16T21:26:35.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:35.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:35.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:35.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:26:35.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:35.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj579 td/osd-backfill-space/datafile 2020-11-16T21:26:35.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:35.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:35.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:26:35.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:35.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj580 td/osd-backfill-space/datafile 2020-11-16T21:26:36.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:36.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:36.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:26:36.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:36.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj581 td/osd-backfill-space/datafile 2020-11-16T21:26:36.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:36.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:36.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:36.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:26:36.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:36.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj582 td/osd-backfill-space/datafile 2020-11-16T21:26:36.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:36.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:36.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:36.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:26:36.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:36.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj583 td/osd-backfill-space/datafile 2020-11-16T21:26:36.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:36.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:36.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:36.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:26:36.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:36.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj584 td/osd-backfill-space/datafile 2020-11-16T21:26:36.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:36.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:36.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:36.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:26:36.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:36.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj585 td/osd-backfill-space/datafile 2020-11-16T21:26:37.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:37.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:37.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:37.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:26:37.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:37.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj586 td/osd-backfill-space/datafile 2020-11-16T21:26:37.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:37.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:37.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:37.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:26:37.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:37.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj587 td/osd-backfill-space/datafile 2020-11-16T21:26:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:37.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:26:37.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:37.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj588 td/osd-backfill-space/datafile 2020-11-16T21:26:37.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:37.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:37.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:37.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:26:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj589 td/osd-backfill-space/datafile 2020-11-16T21:26:37.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:37.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:37.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:37.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:26:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:37.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj590 td/osd-backfill-space/datafile 2020-11-16T21:26:37.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:37.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:37.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:37.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:26:38.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:38.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj591 td/osd-backfill-space/datafile 2020-11-16T21:26:38.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:38.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:38.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:38.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:26:38.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:38.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj592 td/osd-backfill-space/datafile 2020-11-16T21:26:38.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:38.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:38.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:38.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:26:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj593 td/osd-backfill-space/datafile 2020-11-16T21:26:38.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:38.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:38.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:38.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:26:38.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:38.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj594 td/osd-backfill-space/datafile 2020-11-16T21:26:38.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:38.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:38.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:38.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:26:38.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:38.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj595 td/osd-backfill-space/datafile 2020-11-16T21:26:38.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:38.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:38.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:38.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:26:39.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:39.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj596 td/osd-backfill-space/datafile 2020-11-16T21:26:39.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:39.098 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:39.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:39.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:26:39.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:39.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj597 td/osd-backfill-space/datafile 2020-11-16T21:26:39.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:39.290 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:26:39.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:39.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj598 td/osd-backfill-space/datafile 2020-11-16T21:26:39.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:39.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:39.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:39.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:26:39.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:39.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj599 td/osd-backfill-space/datafile 2020-11-16T21:26:39.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2020-11-16T21:26:39.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:39.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:26:39.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:39.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj600 td/osd-backfill-space/datafile 2020-11-16T21:26:39.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:144: TEST_backfill_test_simple: ceph pg dump pgs 2020-11-16T21:26:40.267 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:26:40.267 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 589 0 0 0 0 2412544 0 0 89 89 active+clean 2020-11-16T21:24:38.305724+0000 27'589 27:608 [2] 2 [2] 2 0'0 2020-11-16T21:24:36.167294+0000 0'0 2020-11-16T21:24:36.167294+0000 0 2020-11-16T21:26:40.268 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 576 0 0 0 0 2359296 0 0 76 76 active+clean 2020-11-16T21:24:35.993032+0000 27'576 27:599 [1] 1 [1] 1 0'0 2020-11-16T21:24:33.565798+0000 0'0 2020-11-16T21:24:33.565798+0000 0 2020-11-16T21:26:40.268 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:26:40.268 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:26:40.270 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:26:40.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:146: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:26:40.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:146: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:40.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:148: TEST_backfill_test_simple: ceph osd pool set test1 size 2 2020-11-16T21:26:41.008 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T21:26:41.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:146: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2020-11-16T21:26:41.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:148: TEST_backfill_test_simple: ceph osd pool set test2 size 2 2020-11-16T21:26:41.751 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 2 2020-11-16T21:26:41.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:150: TEST_backfill_test_simple: sleep 30 2020-11-16T21:27:11.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:152: TEST_backfill_test_simple: wait_for_not_backfilling 240 2020-11-16T21:27:11.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2020-11-16T21:27:11.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2020-11-16T21:27:11.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2020-11-16T21:27:11.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:27:11.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:27:11.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:27:11.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2020-11-16T21:27:11.771 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:27:11.771 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:27:11.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:27:11.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:27:11.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:27:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:27:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:27:12.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:27:12.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:27:12.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:27:12.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:27:12.481 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:27:12.481 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T21:27:12.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:27:12.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:27:12.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:27:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836520 2020-11-16T21:27:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836520 2020-11-16T21:27:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836520' 2020-11-16T21:27:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:27:12.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:27:12.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672998 2020-11-16T21:27:12.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672998 2020-11-16T21:27:12.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836520 1-42949672998' 2020-11-16T21:27:12.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:27:12.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:27:13.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509478 2020-11-16T21:27:13.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509478 2020-11-16T21:27:13.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836520 1-42949672998 2-64424509478' 2020-11-16T21:27:13.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:27:13.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836520 2020-11-16T21:27:13.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:27:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:27:13.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836520 2020-11-16T21:27:13.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:27:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836520 2020-11-16T21:27:13.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2020-11-16T21:27:13.050 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836520 2020-11-16T21:27:13.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:27:13.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836520 -lt 21474836520 2020-11-16T21:27:13.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:27:13.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672998 2020-11-16T21:27:13.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:27:13.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:27:13.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672998 2020-11-16T21:27:13.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:27:13.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672998 2020-11-16T21:27:13.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2020-11-16T21:27:13.452 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672998 2020-11-16T21:27:13.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:27:13.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672998 -lt 42949672998 2020-11-16T21:27:13.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:27:13.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509478 2020-11-16T21:27:13.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:27:13.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:27:13.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509478 2020-11-16T21:27:13.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:27:13.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509478 2020-11-16T21:27:13.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2020-11-16T21:27:13.845 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509478 2020-11-16T21:27:13.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:27:14.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509475 -lt 64424509478 2020-11-16T21:27:14.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:27:15.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:27:15.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:27:15.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509478 -lt 64424509478 2020-11-16T21:27:15.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:27:15.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:27:15.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:27:16.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2020-11-16T21:27:16.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:27:16.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:27:16.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:27:16.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:27:16.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:27:16.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:27:16.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:27:16.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:27:16.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:27:16.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' -1 2020-11-16T21:27:16.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:68: wait_for_not_state: loop=0 2020-11-16T21:27:16.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:69: wait_for_not_state: num_in_state=1 2020-11-16T21:27:16.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 5 2020-11-16T21:27:21.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:27:21.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:27:21.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:27:21.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:27:21.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:27:21.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:27:21.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:27:21.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:27:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:27:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:27:21.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:27:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 1 >= 17 )) 2020-11-16T21:27:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 10 2020-11-16T21:27:31.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:27:31.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:27:31.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:27:31.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:27:31.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:27:31.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:27:31.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:27:31.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:27:32.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:27:32.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:27:32.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:27:32.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 2 >= 17 )) 2020-11-16T21:27:32.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:27:47.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:27:47.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:27:47.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:27:47.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:27:47.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:27:47.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:27:47.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:27:47.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:27:47.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:27:47.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:27:47.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:27:47.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 3 >= 17 )) 2020-11-16T21:27:47.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:28:02.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:28:02.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:28:02.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:28:02.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:28:02.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:28:02.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:28:02.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:28:02.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:28:03.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:28:03.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:28:03.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:28:03.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 4 >= 17 )) 2020-11-16T21:28:03.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:28:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:28:18.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:28:18.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:28:18.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:28:18.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:28:18.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:28:18.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:28:18.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:28:18.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:28:18.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:28:18.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:28:18.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 5 >= 17 )) 2020-11-16T21:28:18.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2020-11-16T21:28:33.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:28:33.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:28:33.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:28:33.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:28:33.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:28:33.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:28:33.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:28:33.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:28:33.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:28:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:28:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:28:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:28:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:153: TEST_backfill_test_simple: wait_for_not_activating 60 2020-11-16T21:28:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2020-11-16T21:28:33.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2020-11-16T21:28:33.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2020-11-16T21:28:33.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:28:33.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:28:33.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:28:33.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2020-11-16T21:28:33.854 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:28:33.854 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:28:33.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:28:33.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:28:33.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:28:33.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:28:33.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:28:33.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:28:33.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:28:33.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:28:34.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:28:34.333 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:28:34.333 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T21:28:34.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:28:34.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:28:34.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:28:34.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836538 2020-11-16T21:28:34.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836538 2020-11-16T21:28:34.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836538' 2020-11-16T21:28:34.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:28:34.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:28:34.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673016 2020-11-16T21:28:34.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673016 2020-11-16T21:28:34.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836538 1-42949673016' 2020-11-16T21:28:34.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:28:34.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:28:34.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509496 2020-11-16T21:28:34.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509496 2020-11-16T21:28:34.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836538 1-42949673016 2-64424509496' 2020-11-16T21:28:34.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:28:34.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836538 2020-11-16T21:28:34.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:28:34.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:28:34.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:28:34.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836538 2020-11-16T21:28:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836538 2020-11-16T21:28:34.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2020-11-16T21:28:34.823 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836538 2020-11-16T21:28:34.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:28:35.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836538 -lt 21474836538 2020-11-16T21:28:35.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:28:35.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673016 2020-11-16T21:28:35.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:28:35.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:28:35.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673016 2020-11-16T21:28:35.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:28:35.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673016 2020-11-16T21:28:35.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673016' 2020-11-16T21:28:35.216 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673016 2020-11-16T21:28:35.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:28:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673016 -lt 42949673016 2020-11-16T21:28:35.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:28:35.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509496 2020-11-16T21:28:35.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:28:35.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:28:35.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509496 2020-11-16T21:28:35.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:28:35.643 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509496 2020-11-16T21:28:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509496 2020-11-16T21:28:35.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2020-11-16T21:28:35.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:28:36.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509496 -lt 64424509496 2020-11-16T21:28:36.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:28:36.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:28:36.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:28:36.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2020-11-16T21:28:36.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:28:36.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2020-11-16T21:28:36.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2020-11-16T21:28:36.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:28:36.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2020-11-16T21:28:36.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:28:36.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2020-11-16T21:28:36.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:28:36.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:28:36.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:28:36.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:28:36.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:155: TEST_backfill_test_simple: ERRORS=0 2020-11-16T21:28:36.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:156: TEST_backfill_test_simple: ceph pg dump pgs 2020-11-16T21:28:36.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:156: TEST_backfill_test_simple: grep +backfill_toofull 2020-11-16T21:28:36.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:156: TEST_backfill_test_simple: wc -l 2020-11-16T21:28:37.223 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:28:37.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:156: TEST_backfill_test_simple: '[' 1 '!=' 1 ']' 2020-11-16T21:28:37.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:162: TEST_backfill_test_simple: expr 2 - 1 2020-11-16T21:28:37.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:162: TEST_backfill_test_simple: expected=1 2020-11-16T21:28:37.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:163: TEST_backfill_test_simple: ceph pg dump pgs 2020-11-16T21:28:37.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:163: TEST_backfill_test_simple: grep active+clean 2020-11-16T21:28:37.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:163: TEST_backfill_test_simple: wc -l 2020-11-16T21:28:37.579 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:28:37.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:163: TEST_backfill_test_simple: '[' 1 '!=' 1 ']' 2020-11-16T21:28:37.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:169: TEST_backfill_test_simple: ceph pg dump pgs 2020-11-16T21:28:37.950 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:28:37.950 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2020-11-16T21:28:12.346531+0000 27'600 34:649 [2,1] 2 [2] 2 0'0 2020-11-16T21:24:36.167294+0000 0'0 2020-11-16T21:24:36.167294+0000 0 2020-11-16T21:28:37.950 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2020-11-16T21:28:20.071859+0000 27'600 35:1849 [1,0] 1 [1,0] 1 0'0 2020-11-16T21:24:33.565798+0000 0'0 2020-11-16T21:24:33.565798+0000 0 2020-11-16T21:28:37.950 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:28:37.951 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:28:37.953 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:28:37.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:171: TEST_backfill_test_simple: '[' 0 '!=' 0 ']' 2020-11-16T21:28:37.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:176: TEST_backfill_test_simple: seq 1 2 2020-11-16T21:28:37.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:176: TEST_backfill_test_simple: for i in $(seq 1 $pools) 2020-11-16T21:28:37.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:178: TEST_backfill_test_simple: delete_pool test1 2020-11-16T21:28:37.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2020-11-16T21:28:37.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-11-16T21:28:38.442 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' does not exist 2020-11-16T21:28:38.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:176: TEST_backfill_test_simple: for i in $(seq 1 $pools) 2020-11-16T21:28:38.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:178: TEST_backfill_test_simple: delete_pool test2 2020-11-16T21:28:38.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2020-11-16T21:28:38.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-11-16T21:28:38.974 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' does not exist 2020-11-16T21:28:38.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:180: TEST_backfill_test_simple: kill_daemons td/osd-backfill-space 2020-11-16T21:28:38.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:28:38.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:28:38.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:28:38.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:28:38.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:28:39.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:28:39.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:181: TEST_backfill_test_simple: grep -q 'num_bytes mismatch' td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log 2020-11-16T21:28:39.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2020-11-16T21:28:39.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:28:39.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:28:39.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:28:39.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:28:39.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:28:39.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:28:39.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:28:39.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:28:39.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:28:39.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:28:39.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:28:39.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:28:39.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:28:39.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:28:39.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:28:39.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:28:39.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:28:39.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:28:39.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:28:39.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:28:39.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:28:39.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:28:39.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:28:39.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:28:39.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:28:39.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:28:39.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:28:39.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:28:39.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2020-11-16T21:28:39.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2020-11-16T21:28:39.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2020-11-16T21:28:39.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2020-11-16T21:28:39.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:28:39.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:28:39.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:28:39.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:28:39.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:28:39.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:28:39.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:28:39.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:28:39.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:28:39.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:28:39.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:28:39.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:28:39.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:28:39.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:28:39.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:28:39.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:28:39.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:28:39.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:28:39.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:28:39.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:28:39.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:28:39.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:28:39.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:28:39.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:28:39.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:28:39.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:28:39.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:28:39.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:28:39.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2020-11-16T21:28:39.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:28:39.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:28:39.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:28:39.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137323 2020-11-16T21:28:39.221 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:28:39.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:28:39.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:28:39.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2020-11-16T21:28:39.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_ec_backfill_grow td/osd-backfill-space 2020-11-16T21:28:39.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1110: TEST_ec_backfill_grow: local dir=td/osd-backfill-space 2020-11-16T21:28:39.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1111: TEST_ec_backfill_grow: local poolname=test 2020-11-16T21:28:39.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1112: TEST_ec_backfill_grow: local OSDS=6 2020-11-16T21:28:39.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1113: TEST_ec_backfill_grow: local k=3 2020-11-16T21:28:39.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1114: TEST_ec_backfill_grow: local m=2 2020-11-16T21:28:39.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1115: TEST_ec_backfill_grow: expr 600 / 3 2020-11-16T21:28:39.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1115: TEST_ec_backfill_grow: local ecobjects=200 2020-11-16T21:28:39.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1117: TEST_ec_backfill_grow: run_mon td/osd-backfill-space a 2020-11-16T21:28:39.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2020-11-16T21:28:39.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:28:39.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:28:39.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:28:39.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2020-11-16T21:28:39.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2020-11-16T21:28:39.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:28:39.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:28:39.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:28:39.536 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:28:39.536 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:28:39.536 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:28:39.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:28:39.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:28:39.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:28:39.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:28:39.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:28:39.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:28:39.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:28:39.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:28:39.742 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:28:39.742 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:28:39.742 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:28:39.743 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:28:39.743 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:28:39.743 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:28:39.743 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:28:39.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:28:39.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get fsid 2020-11-16T21:28:39.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:28:39.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:28:39.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:28:39.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:28:39.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:28:39.853 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:28:39.853 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:28:39.853 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:28:39.854 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:28:39.854 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:28:39.854 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:28:39.854 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:28:39.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:28:39.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get mon_host 2020-11-16T21:28:39.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1118: TEST_ec_backfill_grow: run_mgr td/osd-backfill-space x 2020-11-16T21:28:39.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2020-11-16T21:28:39.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:28:39.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:28:39.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:28:39.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2020-11-16T21:28:39.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:28:40.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:28:40.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:28:40.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:28:40.482 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:28:40.482 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:28:40.482 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:28:40.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:28:40.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:28:40.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:28:40.517 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: expr 6 - 1 2020-11-16T21:28:40.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: seq 0 5 2020-11-16T21:28:40.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:28:40.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 0 2020-11-16T21:28:40.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:28:40.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:28:40.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:28:40.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:28:40.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2020-11-16T21:28:40.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:28:40.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:28:40.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:28:40.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:28:40.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2020-11-16T21:28:40.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2020-11-16T21:28:40.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:28:40.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:28:40.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:28:40.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:28:40.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:28:40.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:28:40.525 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:28:40.526 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:28:40.526 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:28:40.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:28:40.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:28:40.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:28:40.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:28:40.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:28:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:28:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:28:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:28:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:28:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:28:40.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:28:40.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:28:40.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2020-11-16T21:28:40.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:28:40.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6711bafb-5165-41b9-bed9-ceaa296f93f2 2020-11-16T21:28:40.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 6711bafb-5165-41b9-bed9-ceaa296f93f2' 2020-11-16T21:28:40.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:28:40.531 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 6711bafb-5165-41b9-bed9-ceaa296f93f2 2020-11-16T21:28:40.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAI77Jf+agFIRAA4Rq0MRAadJx5ZYRPo8Qx5Q== 2020-11-16T21:28:40.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAI77Jf+agFIRAA4Rq0MRAadJx5ZYRPo8Qx5Q=="}' 2020-11-16T21:28:40.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6711bafb-5165-41b9-bed9-ceaa296f93f2 -i td/osd-backfill-space/0/new.json 2020-11-16T21:28:40.900 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:28:40.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2020-11-16T21:28:40.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAI77Jf+agFIRAA4Rq0MRAadJx5ZYRPo8Qx5Q== --osd-uuid 6711bafb-5165-41b9-bed9-ceaa296f93f2 2020-11-16T21:28:40.941 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:40.946+0000 7fa613150d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:40.943 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:40.950+0000 7fa613150d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:40.950+0000 7fa613150d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:40.945 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:40.950+0000 7fa613150d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:28:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:40.950+0000 7fa613150d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:28:40.946 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:40.950+0000 7fa613150d80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:28:40.995 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:40.998+0000 7fa613150d80 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2020-11-16T21:28:45.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2020-11-16T21:28:45.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:28:45.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:28:45.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:28:45.444 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:28:45.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:28:45.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:28:45.950 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:28:45.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:28:45.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:28:45.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:28:45.974 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:45.978+0000 7f43d004dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:45.976 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:45.982+0000 7f43d004dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:45.976 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:45.982+0000 7f43d004dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:28:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:28:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:28:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:28:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:28:46.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:28:46.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:28:46.334 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:28:46.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:28:46.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:28:46.517 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:46.522+0000 7f43d004dd80 -1 Falling back to public interface 2020-11-16T21:28:46.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:28:47.377 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:47.374+0000 7f43d004dd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:28:47.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:28:47.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:28:47.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:28:47.716 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:28:47.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:28:47.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:28:48.119 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/412180,v1:127.0.0.1:6803/412180] [v2:127.0.0.1:6804/412180,v1:127.0.0.1:6805/412180] exists,up 6711bafb-5165-41b9-bed9-ceaa296f93f2 2020-11-16T21:28:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:28:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:28:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:28:48.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:28:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 1 2020-11-16T21:28:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:28:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:28:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:28:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:28:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2020-11-16T21:28:48.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:28:48.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:28:48.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:28:48.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:28:48.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2020-11-16T21:28:48.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2020-11-16T21:28:48.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:28:48.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:28:48.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:28:48.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:28:48.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:28:48.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:28:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:28:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:28:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:28:48.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:28:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:28:48.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:28:48.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:28:48.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:28:48.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:28:48.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:28:48.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:28:48.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:28:48.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:28:48.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:28:48.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:28:48.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2020-11-16T21:28:48.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:28:48.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=015901b7-10a0-49e6-9eaf-c24cf5f0cb32 2020-11-16T21:28:48.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 015901b7-10a0-49e6-9eaf-c24cf5f0cb32' 2020-11-16T21:28:48.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:28:48.127 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 015901b7-10a0-49e6-9eaf-c24cf5f0cb32 2020-11-16T21:28:48.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAQ77JfXZ4CCRAAt+5ybmHVo42EwkaDFtWP9Q== 2020-11-16T21:28:48.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAQ77JfXZ4CCRAAt+5ybmHVo42EwkaDFtWP9Q=="}' 2020-11-16T21:28:48.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 015901b7-10a0-49e6-9eaf-c24cf5f0cb32 -i td/osd-backfill-space/1/new.json 2020-11-16T21:28:48.718 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:28:48.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2020-11-16T21:28:48.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAQ77JfXZ4CCRAAt+5ybmHVo42EwkaDFtWP9Q== --osd-uuid 015901b7-10a0-49e6-9eaf-c24cf5f0cb32 2020-11-16T21:28:48.767 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:48.770+0000 7f1e5f6a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:48.769 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:48.774+0000 7f1e5f6a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:48.771 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:48.778+0000 7f1e5f6a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:48.772 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:48.778+0000 7f1e5f6a2d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:28:48.772 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:48.778+0000 7f1e5f6a2d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:28:48.772 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:48.778+0000 7f1e5f6a2d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:28:48.818 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:48.822+0000 7f1e5f6a2d80 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2020-11-16T21:28:53.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2020-11-16T21:28:53.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:28:53.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:28:53.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:28:53.213 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:28:53.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:28:53.711 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:28:53.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:28:53.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:28:53.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:28:53.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:28:53.745 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:53.750+0000 7fb68f2b7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:53.748 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:53.754+0000 7fb68f2b7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:53.749 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:53.754+0000 7fb68f2b7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:54.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:28:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:28:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:28:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:28:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:28:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:28:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:28:54.099 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:28:54.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:28:54.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:28:54.288 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:54.294+0000 7fb68f2b7d80 -1 Falling back to public interface 2020-11-16T21:28:54.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:28:55.124 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:55.130+0000 7fb68f2b7d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:28:55.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:28:55.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:28:55.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:28:55.482 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:28:55.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:28:55.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:28:55.856 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/413174,v1:127.0.0.1:6811/413174] [v2:127.0.0.1:6812/413174,v1:127.0.0.1:6813/413174] exists,up 015901b7-10a0-49e6-9eaf-c24cf5f0cb32 2020-11-16T21:28:55.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:28:55.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:28:55.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:28:55.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:28:55.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 2 2020-11-16T21:28:55.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:28:55.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:28:55.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:28:55.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:28:55.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2020-11-16T21:28:55.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:28:55.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:28:55.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:28:55.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:28:55.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2020-11-16T21:28:55.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2020-11-16T21:28:55.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:28:55.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:28:55.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:28:55.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:28:55.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:28:55.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:28:55.863 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:28:55.863 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:28:55.863 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:28:55.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:28:55.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:28:55.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:28:55.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:28:55.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:28:55.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:28:55.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:28:55.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:28:55.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:28:55.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:28:55.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:28:55.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:28:55.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2020-11-16T21:28:55.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:28:55.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8c07aca5-f934-4889-b90e-d09a3ee61b06 2020-11-16T21:28:55.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 8c07aca5-f934-4889-b90e-d09a3ee61b06' 2020-11-16T21:28:55.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:28:55.867 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 8c07aca5-f934-4889-b90e-d09a3ee61b06 2020-11-16T21:28:55.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAX77Jf0kFHNRAAntzpW2V8vLi3dpu2LL/yAA== 2020-11-16T21:28:55.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAX77Jf0kFHNRAAntzpW2V8vLi3dpu2LL/yAA=="}' 2020-11-16T21:28:55.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8c07aca5-f934-4889-b90e-d09a3ee61b06 -i td/osd-backfill-space/2/new.json 2020-11-16T21:28:56.412 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:28:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2020-11-16T21:28:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAX77Jf0kFHNRAAntzpW2V8vLi3dpu2LL/yAA== --osd-uuid 8c07aca5-f934-4889-b90e-d09a3ee61b06 2020-11-16T21:28:56.453 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:56.458+0000 7fedcb896d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:56.456 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:56.462+0000 7fedcb896d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:56.458 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:56.462+0000 7fedcb896d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:28:56.459 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:56.466+0000 7fedcb896d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:28:56.459 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:56.466+0000 7fedcb896d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:28:56.459 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:56.466+0000 7fedcb896d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:28:56.498 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:28:56.502+0000 7fedcb896d80 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2020-11-16T21:29:00.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2020-11-16T21:29:00.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:29:00.909 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:29:00.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:29:00.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:29:01.403 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:29:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:29:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:29:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:29:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:29:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:29:01.436 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:01.442+0000 7f44effebd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:01.471 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:01.478+0000 7f44effebd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:01.472 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:01.478+0000 7f44effebd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:01.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:29:01.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:29:01.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:29:01.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:29:01.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:29:01.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:29:01.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:29:01.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:29:01.755 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:29:01.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:29:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:02.018+0000 7f44effebd80 -1 Falling back to public interface 2020-11-16T21:29:02.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:29:02.879 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:02.886+0000 7f44effebd80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:29:03.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:29:03.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:29:03.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:29:03.180 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:29:03.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:29:03.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:29:03.633 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/414168,v1:127.0.0.1:6819/414168] [v2:127.0.0.1:6820/414168,v1:127.0.0.1:6821/414168] exists,up 8c07aca5-f934-4889-b90e-d09a3ee61b06 2020-11-16T21:29:03.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:29:03.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:29:03.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:29:03.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:29:03.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 3 2020-11-16T21:29:03.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:29:03.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:29:03.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:29:03.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:29:03.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/3 2020-11-16T21:29:03.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:29:03.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:29:03.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:29:03.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:29:03.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/3' 2020-11-16T21:29:03.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/3/journal' 2020-11-16T21:29:03.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:29:03.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:29:03.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:29:03.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:29:03.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:29:03.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:29:03.638 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:29:03.638 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:29:03.638 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:29:03.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:29:03.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:29:03.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:29:03.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:29:03.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:29:03.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:29:03.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:29:03.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:29:03.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:29:03.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:29:03.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:29:03.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:29:03.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/3 2020-11-16T21:29:03.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:29:03.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a9971aa9-314d-49f4-a43e-a5349a20ccf9 2020-11-16T21:29:03.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 a9971aa9-314d-49f4-a43e-a5349a20ccf9' 2020-11-16T21:29:03.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:29:03.641 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 a9971aa9-314d-49f4-a43e-a5349a20ccf9 2020-11-16T21:29:03.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAf77JfJy7FJxAAeKehRH3T0hZuXrV+YT8IKQ== 2020-11-16T21:29:03.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAf77JfJy7FJxAAeKehRH3T0hZuXrV+YT8IKQ=="}' 2020-11-16T21:29:03.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a9971aa9-314d-49f4-a43e-a5349a20ccf9 -i td/osd-backfill-space/3/new.json 2020-11-16T21:29:04.261 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:29:04.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/3/new.json 2020-11-16T21:29:04.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAf77JfJy7FJxAAeKehRH3T0hZuXrV+YT8IKQ== --osd-uuid a9971aa9-314d-49f4-a43e-a5349a20ccf9 2020-11-16T21:29:04.311 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:04.318+0000 7f4884edbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:04.315 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:04.322+0000 7f4884edbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:04.316 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:04.322+0000 7f4884edbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:04.317 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:04.322+0000 7f4884edbd80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:29:04.317 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:04.322+0000 7f4884edbd80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:29:04.318 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:04.322+0000 7f4884edbd80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:29:04.403 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:04.410+0000 7f4884edbd80 -1 bluestore(td/osd-backfill-space/3) _read_fsid unparsable uuid 2020-11-16T21:29:08.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/3/keyring 2020-11-16T21:29:08.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:29:08.751 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:29:08.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:29:08.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:29:09.250 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:29:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:29:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:29:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:29:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:29:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:29:09.285 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:09.290+0000 7f43caeedd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:09.288 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:09.294+0000 7f43caeedd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:09.289 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:09.294+0000 7f43caeedd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:09.577 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:09.582+0000 7f43caeedd80 -1 Falling back to public interface 2020-11-16T21:29:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:29:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:29:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:29:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:29:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:29:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:29:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:29:09.606 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:29:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:29:09.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:29:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:29:10.514 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:10.514+0000 7f43caeedd80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:29:11.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:29:11.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:29:11.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:29:11.031 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:29:11.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:29:11.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:29:11.432 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/415162,v1:127.0.0.1:6827/415162] [v2:127.0.0.1:6828/415162,v1:127.0.0.1:6829/415162] exists,up a9971aa9-314d-49f4-a43e-a5349a20ccf9 2020-11-16T21:29:11.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:29:11.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:29:11.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:29:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:29:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 4 2020-11-16T21:29:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:29:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:29:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:29:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:29:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/4 2020-11-16T21:29:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:29:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:29:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:29:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:29:11.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/4' 2020-11-16T21:29:11.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/4/journal' 2020-11-16T21:29:11.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:29:11.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:29:11.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:29:11.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:29:11.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:29:11.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:29:11.437 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:29:11.438 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:29:11.438 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:29:11.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:29:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:29:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:29:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:29:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:29:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:29:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:29:11.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:29:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:29:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:29:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:29:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:29:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/4 2020-11-16T21:29:11.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:29:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9bca3424-09cf-48de-be91-45c6918cc223 2020-11-16T21:29:11.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 9bca3424-09cf-48de-be91-45c6918cc223' 2020-11-16T21:29:11.441 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 9bca3424-09cf-48de-be91-45c6918cc223 2020-11-16T21:29:11.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:29:11.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAn77JfHVO1GxAA++D3KjGixKoctKbrefxaYg== 2020-11-16T21:29:11.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAn77JfHVO1GxAA++D3KjGixKoctKbrefxaYg=="}' 2020-11-16T21:29:11.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9bca3424-09cf-48de-be91-45c6918cc223 -i td/osd-backfill-space/4/new.json 2020-11-16T21:29:11.992 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:29:12.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/4/new.json 2020-11-16T21:29:12.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAn77JfHVO1GxAA++D3KjGixKoctKbrefxaYg== --osd-uuid 9bca3424-09cf-48de-be91-45c6918cc223 2020-11-16T21:29:12.037 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:12.042+0000 7fa0fb71bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:12.040 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:12.046+0000 7fa0fb71bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:12.041 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:12.046+0000 7fa0fb71bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:12.042 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:12.046+0000 7fa0fb71bd80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:29:12.042 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:12.046+0000 7fa0fb71bd80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:29:12.043 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:12.046+0000 7fa0fb71bd80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:29:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:12.110+0000 7fa0fb71bd80 -1 bluestore(td/osd-backfill-space/4) _read_fsid unparsable uuid 2020-11-16T21:29:16.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/4/keyring 2020-11-16T21:29:16.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:29:16.682 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:29:16.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:29:16.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:29:17.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:29:17.190 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:29:17.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:29:17.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:29:17.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:29:17.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:29:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:17.226+0000 7f2d4029dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:17.224 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:17.230+0000 7f2d4029dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:17.226 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:17.230+0000 7f2d4029dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:17.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:29:17.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:29:17.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:29:17.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:29:17.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:29:17.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:29:17.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:29:17.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:29:17.578 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:29:17.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:29:17.773 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:17.778+0000 7f2d4029dd80 -1 Falling back to public interface 2020-11-16T21:29:17.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:29:18.628 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:18.634+0000 7f2d4029dd80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:29:18.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:29:18.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:29:18.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:29:18.973 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:29:18.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:29:18.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:29:19.402 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/416156,v1:127.0.0.1:6835/416156] [v2:127.0.0.1:6836/416156,v1:127.0.0.1:6837/416156] exists,up 9bca3424-09cf-48de-be91-45c6918cc223 2020-11-16T21:29:19.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:29:19.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:29:19.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:29:19.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:29:19.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 5 2020-11-16T21:29:19.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:29:19.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:29:19.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T21:29:19.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:29:19.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/5 2020-11-16T21:29:19.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:29:19.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:29:19.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:29:19.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:29:19.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/5' 2020-11-16T21:29:19.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/5/journal' 2020-11-16T21:29:19.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:29:19.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:29:19.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:29:19.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:29:19.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:29:19.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:29:19.409 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:29:19.409 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:29:19.409 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:29:19.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:29:19.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:29:19.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:29:19.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:29:19.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:29:19.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:29:19.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:29:19.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:29:19.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:29:19.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:29:19.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:29:19.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:29:19.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/5 2020-11-16T21:29:19.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:29:19.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e87c752d-16c3-4f4a-996a-136c6f029952 2020-11-16T21:29:19.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 e87c752d-16c3-4f4a-996a-136c6f029952' 2020-11-16T21:29:19.412 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 e87c752d-16c3-4f4a-996a-136c6f029952 2020-11-16T21:29:19.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:29:19.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAv77JfpA52GhAAHCfTLkVIOQV1ZHn1UcLy6Q== 2020-11-16T21:29:19.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAv77JfpA52GhAAHCfTLkVIOQV1ZHn1UcLy6Q=="}' 2020-11-16T21:29:19.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e87c752d-16c3-4f4a-996a-136c6f029952 -i td/osd-backfill-space/5/new.json 2020-11-16T21:29:20.023 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T21:29:20.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/5/new.json 2020-11-16T21:29:20.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAv77JfpA52GhAAHCfTLkVIOQV1ZHn1UcLy6Q== --osd-uuid e87c752d-16c3-4f4a-996a-136c6f029952 2020-11-16T21:29:20.066 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:20.070+0000 7f20aa5f5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:20.073 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:20.078+0000 7f20aa5f5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:20.082+0000 7f20aa5f5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:20.082+0000 7f20aa5f5d80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:29:20.076 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:20.082+0000 7f20aa5f5d80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:29:20.077 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:20.082+0000 7f20aa5f5d80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:29:20.139 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:20.146+0000 7f20aa5f5d80 -1 bluestore(td/osd-backfill-space/5) _read_fsid unparsable uuid 2020-11-16T21:29:24.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/5/keyring 2020-11-16T21:29:24.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:29:24.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T21:29:24.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:29:24.447 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T21:29:24.967 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T21:29:24.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T21:29:24.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:29:24.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:29:24.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:29:24.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:29:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:25.006+0000 7f59ca782d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:25.010+0000 7f59ca782d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:25.005 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:25.010+0000 7f59ca782d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:29:25.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T21:29:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:29:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T21:29:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:29:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:29:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:29:25.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:29:25.359 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:29:25.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:29:25.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:29:25.541 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:25.546+0000 7f59ca782d80 -1 Falling back to public interface 2020-11-16T21:29:25.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:29:26.456 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:29:26.462+0000 7f59ca782d80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T21:29:26.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:29:26.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:29:26.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:29:26.750 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:29:26.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:29:26.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:29:27.183 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/417151,v1:127.0.0.1:6843/417151] [v2:127.0.0.1:6844/417151,v1:127.0.0.1:6845/417151] exists,up e87c752d-16c3-4f4a-996a-136c6f029952 2020-11-16T21:29:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:29:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:29:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:29:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1125: TEST_ec_backfill_grow: ceph osd set-backfillfull-ratio .85 2020-11-16T21:29:28.231 INFO:tasks.workunit.client.0.smithi189.stderr:osd set-backfillfull-ratio 0.85 2020-11-16T21:29:28.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1127: TEST_ec_backfill_grow: ceph osd set-require-min-compat-client luminous 2020-11-16T21:29:28.947 INFO:tasks.workunit.client.0.smithi189.stderr:set require_min_compat_client to luminous 2020-11-16T21:29:28.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1128: TEST_ec_backfill_grow: ceph osd erasure-code-profile set ec-profile k=3 m=2 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure 2020-11-16T21:29:29.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1129: TEST_ec_backfill_grow: ceph osd pool create test 1 1 erasure ec-profile 2020-11-16T21:29:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T21:29:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1131: TEST_ec_backfill_grow: wait_for_clean 2020-11-16T21:29:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:29:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:29:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:29:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:29:30.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:29:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:29:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:29:30.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:29:30.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:29:30.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:29:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:29:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:29:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:29:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:29:30.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:29:30.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:29:30.941 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:29:30.941 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:29:30.942 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:29:30.942 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:29:30.942 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:29:30.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:29:30.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:29:30.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:29:31.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836492 2020-11-16T21:29:31.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836492 2020-11-16T21:29:31.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492' 2020-11-16T21:29:31.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:29:31.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:29:31.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672970 2020-11-16T21:29:31.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672970 2020-11-16T21:29:31.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672970' 2020-11-16T21:29:31.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:29:31.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:29:31.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542152 2020-11-16T21:29:31.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542152 2020-11-16T21:29:31.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-60129542152' 2020-11-16T21:29:31.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:29:31.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:29:31.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378630 2020-11-16T21:29:31.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378630 2020-11-16T21:29:31.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-60129542152 3-81604378630' 2020-11-16T21:29:31.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:29:31.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:29:31.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215109 2020-11-16T21:29:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215109 2020-11-16T21:29:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-60129542152 3-81604378630 4-103079215109' 2020-11-16T21:29:31.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:29:31.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:29:31.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051587 2020-11-16T21:29:31.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051587 2020-11-16T21:29:31.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-60129542152 3-81604378630 4-103079215109 5-124554051587' 2020-11-16T21:29:31.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:29:31.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836492 2020-11-16T21:29:31.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:29:31.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:29:31.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836492 2020-11-16T21:29:31.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:29:31.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836492 2020-11-16T21:29:31.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-11-16T21:29:31.999 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836492 2020-11-16T21:29:31.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:29:32.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836492 2020-11-16T21:29:32.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:29:32.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2020-11-16T21:29:32.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:29:32.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:29:32.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672970 2020-11-16T21:29:32.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:29:32.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672970 2020-11-16T21:29:32.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2020-11-16T21:29:32.525 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672970 2020-11-16T21:29:32.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:29:32.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672970 2020-11-16T21:29:32.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:29:32.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542152 2020-11-16T21:29:32.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:29:32.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:29:32.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:29:32.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542152 2020-11-16T21:29:32.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542152 2020-11-16T21:29:32.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2020-11-16T21:29:32.948 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542152 2020-11-16T21:29:32.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:29:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542152 -lt 60129542152 2020-11-16T21:29:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:29:33.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378630 2020-11-16T21:29:33.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:29:33.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:29:33.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378630 2020-11-16T21:29:33.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:29:33.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378630 2020-11-16T21:29:33.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378630' 2020-11-16T21:29:33.359 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378630 2020-11-16T21:29:33.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:29:33.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378631 -lt 81604378630 2020-11-16T21:29:33.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:29:33.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215109 2020-11-16T21:29:33.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:29:33.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:29:33.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215109 2020-11-16T21:29:33.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:29:33.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215109 2020-11-16T21:29:33.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215109' 2020-11-16T21:29:33.748 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215109 2020-11-16T21:29:33.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:29:34.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215109 -lt 103079215109 2020-11-16T21:29:34.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:29:34.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051587 2020-11-16T21:29:34.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:29:34.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:29:34.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051587 2020-11-16T21:29:34.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:29:34.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051587 2020-11-16T21:29:34.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051587' 2020-11-16T21:29:34.146 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051587 2020-11-16T21:29:34.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:29:34.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051588 -lt 124554051587 2020-11-16T21:29:34.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:29:34.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:29:34.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:29:35.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:29:35.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:29:35.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:29:35.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:29:35.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:29:35.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:29:35.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:29:35.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:29:35.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:29:35.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:29:35.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:29:35.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:29:35.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:29:35.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T21:29:35.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T21:29:35.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T21:29:35.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:29:35.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:29:35.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:29:35.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:29:35.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:29:35.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:29:35.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:29:35.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:29:35.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:29:35.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:29:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:29:36.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:29:36.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:29:36.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:29:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:29:36.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:29:36.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:29:36.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:29:36.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:29:36.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:29:36.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:29:36.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:29:36.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:29:37.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T21:29:37.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T21:29:37.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2020-11-16T21:29:37.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:29:37.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-11-16T21:29:37.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:29:37.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:29:37.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:29:37.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:29:37.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:29:37.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:29:37.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:29:37.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:29:37.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:29:37.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:29:37.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:29:37.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:29:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:29:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:29:38.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:29:38.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:29:38.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:29:38.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:29:38.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:29:38.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:29:38.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:29:38.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T21:29:38.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T21:29:38.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 13 )) 2020-11-16T21:29:38.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:29:38.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2020-11-16T21:29:39.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:29:39.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:29:39.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:29:39.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:29:39.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:29:39.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:29:39.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:29:39.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:29:39.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:29:39.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:29:39.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:29:39.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:29:40.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:29:40.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:29:40.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:29:40.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1133: TEST_ec_backfill_grow: dd if=/dev/urandom of=td/osd-backfill-space/12kdata bs=1k count=12 2020-11-16T21:29:40.160 INFO:tasks.workunit.client.0.smithi189.stderr:12+0 records in 2020-11-16T21:29:40.160 INFO:tasks.workunit.client.0.smithi189.stderr:12+0 records out 2020-11-16T21:29:40.161 INFO:tasks.workunit.client.0.smithi189.stderr:12288 bytes (12 kB, 12 KiB) copied, 0.000228807 s, 53.7 MB/s 2020-11-16T21:29:40.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: seq 1 200 2020-11-16T21:29:40.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:40.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj1 td/osd-backfill-space/12kdata 2020-11-16T21:29:40.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:40.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj2 td/osd-backfill-space/12kdata 2020-11-16T21:29:40.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:40.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj3 td/osd-backfill-space/12kdata 2020-11-16T21:29:40.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:40.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj4 td/osd-backfill-space/12kdata 2020-11-16T21:29:41.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:41.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj5 td/osd-backfill-space/12kdata 2020-11-16T21:29:41.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:41.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj6 td/osd-backfill-space/12kdata 2020-11-16T21:29:41.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:41.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj7 td/osd-backfill-space/12kdata 2020-11-16T21:29:41.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:41.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj8 td/osd-backfill-space/12kdata 2020-11-16T21:29:42.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:42.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj9 td/osd-backfill-space/12kdata 2020-11-16T21:29:42.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj10 td/osd-backfill-space/12kdata 2020-11-16T21:29:42.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:42.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj11 td/osd-backfill-space/12kdata 2020-11-16T21:29:42.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:42.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj12 td/osd-backfill-space/12kdata 2020-11-16T21:29:43.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:43.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj13 td/osd-backfill-space/12kdata 2020-11-16T21:29:43.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:43.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj14 td/osd-backfill-space/12kdata 2020-11-16T21:29:43.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:43.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj15 td/osd-backfill-space/12kdata 2020-11-16T21:29:43.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:43.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj16 td/osd-backfill-space/12kdata 2020-11-16T21:29:43.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:43.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj17 td/osd-backfill-space/12kdata 2020-11-16T21:29:44.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:44.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj18 td/osd-backfill-space/12kdata 2020-11-16T21:29:44.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:44.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj19 td/osd-backfill-space/12kdata 2020-11-16T21:29:44.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:44.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj20 td/osd-backfill-space/12kdata 2020-11-16T21:29:45.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:45.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj21 td/osd-backfill-space/12kdata 2020-11-16T21:29:45.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:45.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj22 td/osd-backfill-space/12kdata 2020-11-16T21:29:45.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:45.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj23 td/osd-backfill-space/12kdata 2020-11-16T21:29:45.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:45.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj24 td/osd-backfill-space/12kdata 2020-11-16T21:29:46.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:46.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj25 td/osd-backfill-space/12kdata 2020-11-16T21:29:46.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:46.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj26 td/osd-backfill-space/12kdata 2020-11-16T21:29:46.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:46.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj27 td/osd-backfill-space/12kdata 2020-11-16T21:29:46.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:46.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj28 td/osd-backfill-space/12kdata 2020-11-16T21:29:47.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:47.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj29 td/osd-backfill-space/12kdata 2020-11-16T21:29:47.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:47.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj30 td/osd-backfill-space/12kdata 2020-11-16T21:29:47.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:47.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj31 td/osd-backfill-space/12kdata 2020-11-16T21:29:47.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:47.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj32 td/osd-backfill-space/12kdata 2020-11-16T21:29:48.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:48.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj33 td/osd-backfill-space/12kdata 2020-11-16T21:29:48.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:48.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj34 td/osd-backfill-space/12kdata 2020-11-16T21:29:48.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:48.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj35 td/osd-backfill-space/12kdata 2020-11-16T21:29:48.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:48.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj36 td/osd-backfill-space/12kdata 2020-11-16T21:29:49.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:49.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj37 td/osd-backfill-space/12kdata 2020-11-16T21:29:49.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:49.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj38 td/osd-backfill-space/12kdata 2020-11-16T21:29:49.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:49.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj39 td/osd-backfill-space/12kdata 2020-11-16T21:29:49.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:49.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj40 td/osd-backfill-space/12kdata 2020-11-16T21:29:50.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj41 td/osd-backfill-space/12kdata 2020-11-16T21:29:50.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:50.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj42 td/osd-backfill-space/12kdata 2020-11-16T21:29:50.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:50.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj43 td/osd-backfill-space/12kdata 2020-11-16T21:29:50.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:50.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj44 td/osd-backfill-space/12kdata 2020-11-16T21:29:51.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:51.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj45 td/osd-backfill-space/12kdata 2020-11-16T21:29:51.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:51.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj46 td/osd-backfill-space/12kdata 2020-11-16T21:29:51.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:51.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj47 td/osd-backfill-space/12kdata 2020-11-16T21:29:51.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:51.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj48 td/osd-backfill-space/12kdata 2020-11-16T21:29:52.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:52.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj49 td/osd-backfill-space/12kdata 2020-11-16T21:29:52.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:52.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj50 td/osd-backfill-space/12kdata 2020-11-16T21:29:52.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj51 td/osd-backfill-space/12kdata 2020-11-16T21:29:52.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:52.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj52 td/osd-backfill-space/12kdata 2020-11-16T21:29:53.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:53.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj53 td/osd-backfill-space/12kdata 2020-11-16T21:29:53.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:53.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj54 td/osd-backfill-space/12kdata 2020-11-16T21:29:53.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:53.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj55 td/osd-backfill-space/12kdata 2020-11-16T21:29:53.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:53.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj56 td/osd-backfill-space/12kdata 2020-11-16T21:29:54.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:54.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj57 td/osd-backfill-space/12kdata 2020-11-16T21:29:54.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:54.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj58 td/osd-backfill-space/12kdata 2020-11-16T21:29:54.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:54.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj59 td/osd-backfill-space/12kdata 2020-11-16T21:29:54.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:54.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj60 td/osd-backfill-space/12kdata 2020-11-16T21:29:55.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:55.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj61 td/osd-backfill-space/12kdata 2020-11-16T21:29:55.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:55.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj62 td/osd-backfill-space/12kdata 2020-11-16T21:29:55.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:55.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj63 td/osd-backfill-space/12kdata 2020-11-16T21:29:55.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:55.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj64 td/osd-backfill-space/12kdata 2020-11-16T21:29:56.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:56.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj65 td/osd-backfill-space/12kdata 2020-11-16T21:29:56.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:56.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj66 td/osd-backfill-space/12kdata 2020-11-16T21:29:56.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:56.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj67 td/osd-backfill-space/12kdata 2020-11-16T21:29:56.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:56.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj68 td/osd-backfill-space/12kdata 2020-11-16T21:29:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:57.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj69 td/osd-backfill-space/12kdata 2020-11-16T21:29:57.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:57.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj70 td/osd-backfill-space/12kdata 2020-11-16T21:29:57.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:57.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj71 td/osd-backfill-space/12kdata 2020-11-16T21:29:57.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:57.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj72 td/osd-backfill-space/12kdata 2020-11-16T21:29:57.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:57.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj73 td/osd-backfill-space/12kdata 2020-11-16T21:29:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:58.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj74 td/osd-backfill-space/12kdata 2020-11-16T21:29:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj75 td/osd-backfill-space/12kdata 2020-11-16T21:29:58.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:58.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj76 td/osd-backfill-space/12kdata 2020-11-16T21:29:58.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:58.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj77 td/osd-backfill-space/12kdata 2020-11-16T21:29:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj78 td/osd-backfill-space/12kdata 2020-11-16T21:29:59.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:59.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj79 td/osd-backfill-space/12kdata 2020-11-16T21:29:59.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:59.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj80 td/osd-backfill-space/12kdata 2020-11-16T21:29:59.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:29:59.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj81 td/osd-backfill-space/12kdata 2020-11-16T21:30:00.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:00.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj82 td/osd-backfill-space/12kdata 2020-11-16T21:30:00.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:00.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj83 td/osd-backfill-space/12kdata 2020-11-16T21:30:00.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:00.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj84 td/osd-backfill-space/12kdata 2020-11-16T21:30:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:00.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj85 td/osd-backfill-space/12kdata 2020-11-16T21:30:01.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:01.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj86 td/osd-backfill-space/12kdata 2020-11-16T21:30:01.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:01.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj87 td/osd-backfill-space/12kdata 2020-11-16T21:30:01.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:01.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj88 td/osd-backfill-space/12kdata 2020-11-16T21:30:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj89 td/osd-backfill-space/12kdata 2020-11-16T21:30:02.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:02.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj90 td/osd-backfill-space/12kdata 2020-11-16T21:30:02.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:02.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj91 td/osd-backfill-space/12kdata 2020-11-16T21:30:02.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:02.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj92 td/osd-backfill-space/12kdata 2020-11-16T21:30:02.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:02.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj93 td/osd-backfill-space/12kdata 2020-11-16T21:30:03.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:03.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj94 td/osd-backfill-space/12kdata 2020-11-16T21:30:03.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:03.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj95 td/osd-backfill-space/12kdata 2020-11-16T21:30:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:03.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj96 td/osd-backfill-space/12kdata 2020-11-16T21:30:03.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:03.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj97 td/osd-backfill-space/12kdata 2020-11-16T21:30:04.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:04.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj98 td/osd-backfill-space/12kdata 2020-11-16T21:30:04.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:04.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj99 td/osd-backfill-space/12kdata 2020-11-16T21:30:04.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:04.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj100 td/osd-backfill-space/12kdata 2020-11-16T21:30:04.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:04.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj101 td/osd-backfill-space/12kdata 2020-11-16T21:30:05.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:05.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj102 td/osd-backfill-space/12kdata 2020-11-16T21:30:05.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:05.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj103 td/osd-backfill-space/12kdata 2020-11-16T21:30:05.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:05.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj104 td/osd-backfill-space/12kdata 2020-11-16T21:30:05.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:05.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj105 td/osd-backfill-space/12kdata 2020-11-16T21:30:06.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:06.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj106 td/osd-backfill-space/12kdata 2020-11-16T21:30:06.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:06.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj107 td/osd-backfill-space/12kdata 2020-11-16T21:30:06.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:06.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj108 td/osd-backfill-space/12kdata 2020-11-16T21:30:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj109 td/osd-backfill-space/12kdata 2020-11-16T21:30:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj110 td/osd-backfill-space/12kdata 2020-11-16T21:30:07.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:07.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj111 td/osd-backfill-space/12kdata 2020-11-16T21:30:07.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:07.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj112 td/osd-backfill-space/12kdata 2020-11-16T21:30:07.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:07.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj113 td/osd-backfill-space/12kdata 2020-11-16T21:30:08.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:08.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj114 td/osd-backfill-space/12kdata 2020-11-16T21:30:08.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:08.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj115 td/osd-backfill-space/12kdata 2020-11-16T21:30:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj116 td/osd-backfill-space/12kdata 2020-11-16T21:30:08.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:08.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj117 td/osd-backfill-space/12kdata 2020-11-16T21:30:08.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:08.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj118 td/osd-backfill-space/12kdata 2020-11-16T21:30:09.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:09.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj119 td/osd-backfill-space/12kdata 2020-11-16T21:30:09.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:09.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj120 td/osd-backfill-space/12kdata 2020-11-16T21:30:09.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:09.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj121 td/osd-backfill-space/12kdata 2020-11-16T21:30:10.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:10.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj122 td/osd-backfill-space/12kdata 2020-11-16T21:30:10.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:10.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj123 td/osd-backfill-space/12kdata 2020-11-16T21:30:10.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:10.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj124 td/osd-backfill-space/12kdata 2020-11-16T21:30:10.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:10.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj125 td/osd-backfill-space/12kdata 2020-11-16T21:30:10.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:10.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj126 td/osd-backfill-space/12kdata 2020-11-16T21:30:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj127 td/osd-backfill-space/12kdata 2020-11-16T21:30:11.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:11.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj128 td/osd-backfill-space/12kdata 2020-11-16T21:30:11.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:11.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj129 td/osd-backfill-space/12kdata 2020-11-16T21:30:11.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:11.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj130 td/osd-backfill-space/12kdata 2020-11-16T21:30:12.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:12.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj131 td/osd-backfill-space/12kdata 2020-11-16T21:30:12.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:12.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj132 td/osd-backfill-space/12kdata 2020-11-16T21:30:12.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:12.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj133 td/osd-backfill-space/12kdata 2020-11-16T21:30:12.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:12.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj134 td/osd-backfill-space/12kdata 2020-11-16T21:30:13.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:13.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj135 td/osd-backfill-space/12kdata 2020-11-16T21:30:13.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:13.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj136 td/osd-backfill-space/12kdata 2020-11-16T21:30:13.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:13.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj137 td/osd-backfill-space/12kdata 2020-11-16T21:30:13.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:13.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj138 td/osd-backfill-space/12kdata 2020-11-16T21:30:14.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:14.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj139 td/osd-backfill-space/12kdata 2020-11-16T21:30:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:14.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj140 td/osd-backfill-space/12kdata 2020-11-16T21:30:14.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:14.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj141 td/osd-backfill-space/12kdata 2020-11-16T21:30:14.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:14.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj142 td/osd-backfill-space/12kdata 2020-11-16T21:30:15.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:15.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj143 td/osd-backfill-space/12kdata 2020-11-16T21:30:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:15.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj144 td/osd-backfill-space/12kdata 2020-11-16T21:30:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:15.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj145 td/osd-backfill-space/12kdata 2020-11-16T21:30:16.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj146 td/osd-backfill-space/12kdata 2020-11-16T21:30:16.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:16.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj147 td/osd-backfill-space/12kdata 2020-11-16T21:30:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj148 td/osd-backfill-space/12kdata 2020-11-16T21:30:16.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:16.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj149 td/osd-backfill-space/12kdata 2020-11-16T21:30:17.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:17.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj150 td/osd-backfill-space/12kdata 2020-11-16T21:30:17.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:17.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj151 td/osd-backfill-space/12kdata 2020-11-16T21:30:17.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:17.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj152 td/osd-backfill-space/12kdata 2020-11-16T21:30:17.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:17.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj153 td/osd-backfill-space/12kdata 2020-11-16T21:30:17.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:17.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj154 td/osd-backfill-space/12kdata 2020-11-16T21:30:18.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:18.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj155 td/osd-backfill-space/12kdata 2020-11-16T21:30:18.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:18.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj156 td/osd-backfill-space/12kdata 2020-11-16T21:30:18.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:18.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj157 td/osd-backfill-space/12kdata 2020-11-16T21:30:19.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:19.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj158 td/osd-backfill-space/12kdata 2020-11-16T21:30:19.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:19.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj159 td/osd-backfill-space/12kdata 2020-11-16T21:30:19.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:19.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj160 td/osd-backfill-space/12kdata 2020-11-16T21:30:19.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj161 td/osd-backfill-space/12kdata 2020-11-16T21:30:19.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:19.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj162 td/osd-backfill-space/12kdata 2020-11-16T21:30:20.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:20.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj163 td/osd-backfill-space/12kdata 2020-11-16T21:30:20.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:20.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj164 td/osd-backfill-space/12kdata 2020-11-16T21:30:20.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:20.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj165 td/osd-backfill-space/12kdata 2020-11-16T21:30:21.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:21.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj166 td/osd-backfill-space/12kdata 2020-11-16T21:30:21.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:21.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj167 td/osd-backfill-space/12kdata 2020-11-16T21:30:21.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:21.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj168 td/osd-backfill-space/12kdata 2020-11-16T21:30:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj169 td/osd-backfill-space/12kdata 2020-11-16T21:30:21.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:21.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj170 td/osd-backfill-space/12kdata 2020-11-16T21:30:22.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:22.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj171 td/osd-backfill-space/12kdata 2020-11-16T21:30:22.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:22.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj172 td/osd-backfill-space/12kdata 2020-11-16T21:30:22.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:22.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj173 td/osd-backfill-space/12kdata 2020-11-16T21:30:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:22.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj174 td/osd-backfill-space/12kdata 2020-11-16T21:30:23.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:23.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj175 td/osd-backfill-space/12kdata 2020-11-16T21:30:23.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:23.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj176 td/osd-backfill-space/12kdata 2020-11-16T21:30:23.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:23.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj177 td/osd-backfill-space/12kdata 2020-11-16T21:30:23.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:23.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj178 td/osd-backfill-space/12kdata 2020-11-16T21:30:24.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:24.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj179 td/osd-backfill-space/12kdata 2020-11-16T21:30:24.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:24.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj180 td/osd-backfill-space/12kdata 2020-11-16T21:30:24.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:24.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj181 td/osd-backfill-space/12kdata 2020-11-16T21:30:24.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:24.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj182 td/osd-backfill-space/12kdata 2020-11-16T21:30:25.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:25.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj183 td/osd-backfill-space/12kdata 2020-11-16T21:30:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj184 td/osd-backfill-space/12kdata 2020-11-16T21:30:25.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:25.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj185 td/osd-backfill-space/12kdata 2020-11-16T21:30:25.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:25.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj186 td/osd-backfill-space/12kdata 2020-11-16T21:30:26.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:26.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj187 td/osd-backfill-space/12kdata 2020-11-16T21:30:26.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:26.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj188 td/osd-backfill-space/12kdata 2020-11-16T21:30:26.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:26.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj189 td/osd-backfill-space/12kdata 2020-11-16T21:30:26.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:26.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj190 td/osd-backfill-space/12kdata 2020-11-16T21:30:27.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:27.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj191 td/osd-backfill-space/12kdata 2020-11-16T21:30:27.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:27.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj192 td/osd-backfill-space/12kdata 2020-11-16T21:30:27.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:27.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj193 td/osd-backfill-space/12kdata 2020-11-16T21:30:27.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:27.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj194 td/osd-backfill-space/12kdata 2020-11-16T21:30:28.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:28.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj195 td/osd-backfill-space/12kdata 2020-11-16T21:30:28.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:28.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj196 td/osd-backfill-space/12kdata 2020-11-16T21:30:28.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:28.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj197 td/osd-backfill-space/12kdata 2020-11-16T21:30:28.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:28.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj198 td/osd-backfill-space/12kdata 2020-11-16T21:30:29.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:29.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj199 td/osd-backfill-space/12kdata 2020-11-16T21:30:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2020-11-16T21:30:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj200 td/osd-backfill-space/12kdata 2020-11-16T21:30:29.602 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1139: TEST_ec_backfill_grow: get_pg test obj1 2020-11-16T21:30:29.602 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T21:30:29.602 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T21:30:29.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T21:30:29.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T21:30:29.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1139: TEST_ec_backfill_grow: local PG=1.0 2020-11-16T21:30:29.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1141: TEST_ec_backfill_grow: get_primary test obj1 2020-11-16T21:30:29.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:30:29.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:30:29.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:30:29.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:30:30.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1141: TEST_ec_backfill_grow: local primary=1 2020-11-16T21:30:30.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1142: TEST_ec_backfill_grow: get_not_primary test obj1 2020-11-16T21:30:30.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2020-11-16T21:30:30.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2020-11-16T21:30:30.398 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2020-11-16T21:30:30.398 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:30:30.398 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:30:30.398 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:30:30.399 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:30:30.822 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T21:30:30.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2020-11-16T21:30:30.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T21:30:31.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1142: TEST_ec_backfill_grow: local otherosd=0 2020-11-16T21:30:31.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1144: TEST_ec_backfill_grow: ceph osd set noout 2020-11-16T21:30:31.924 INFO:tasks.workunit.client.0.smithi189.stderr:noout is set 2020-11-16T21:30:31.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1145: TEST_ec_backfill_grow: kill_daemons td/osd-backfill-space TERM 0 2020-11-16T21:30:31.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:30:31.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:30:31.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:30:31.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:30:31.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:30:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:30:32.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1147: TEST_ec_backfill_grow: expr 200 / 4 2020-11-16T21:30:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1147: TEST_ec_backfill_grow: rmobjects=50 2020-11-16T21:30:32.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: seq 1 50 2020-11-16T21:30:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj1 2020-11-16T21:30:33.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:33.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj2 2020-11-16T21:30:33.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:33.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj3 2020-11-16T21:30:33.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:33.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj4 2020-11-16T21:30:33.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:33.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj5 2020-11-16T21:30:33.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:33.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj6 2020-11-16T21:30:34.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:34.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj7 2020-11-16T21:30:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj8 2020-11-16T21:30:34.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:34.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj9 2020-11-16T21:30:34.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:34.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj10 2020-11-16T21:30:35.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:35.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj11 2020-11-16T21:30:35.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:35.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj12 2020-11-16T21:30:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj13 2020-11-16T21:30:35.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:35.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj14 2020-11-16T21:30:35.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:35.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj15 2020-11-16T21:30:36.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:36.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj16 2020-11-16T21:30:36.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:36.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj17 2020-11-16T21:30:36.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:36.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj18 2020-11-16T21:30:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:36.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj19 2020-11-16T21:30:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:36.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj20 2020-11-16T21:30:37.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:37.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj21 2020-11-16T21:30:37.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:37.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj22 2020-11-16T21:30:37.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:37.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj23 2020-11-16T21:30:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj24 2020-11-16T21:30:37.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:37.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj25 2020-11-16T21:30:38.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:38.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj26 2020-11-16T21:30:38.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:38.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj27 2020-11-16T21:30:38.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:38.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj28 2020-11-16T21:30:38.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:38.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj29 2020-11-16T21:30:38.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:38.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj30 2020-11-16T21:30:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:39.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj31 2020-11-16T21:30:39.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:39.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj32 2020-11-16T21:30:39.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:39.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj33 2020-11-16T21:30:39.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:39.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj34 2020-11-16T21:30:39.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:39.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj35 2020-11-16T21:30:40.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:40.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj36 2020-11-16T21:30:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj37 2020-11-16T21:30:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj38 2020-11-16T21:30:40.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:40.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj39 2020-11-16T21:30:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:40.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj40 2020-11-16T21:30:41.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:41.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj41 2020-11-16T21:30:41.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:41.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj42 2020-11-16T21:30:41.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:41.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj43 2020-11-16T21:30:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:41.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj44 2020-11-16T21:30:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj45 2020-11-16T21:30:42.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:42.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj46 2020-11-16T21:30:42.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:42.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj47 2020-11-16T21:30:42.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:42.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj48 2020-11-16T21:30:42.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:42.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj49 2020-11-16T21:30:43.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2020-11-16T21:30:43.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj50 2020-11-16T21:30:43.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1153: TEST_ec_backfill_grow: dd if=/dev/urandom of=td/osd-backfill-space/36kdata bs=1k count=36 2020-11-16T21:30:43.325 INFO:tasks.workunit.client.0.smithi189.stderr:36+0 records in 2020-11-16T21:30:43.325 INFO:tasks.workunit.client.0.smithi189.stderr:36+0 records out 2020-11-16T21:30:43.325 INFO:tasks.workunit.client.0.smithi189.stderr:36864 bytes (37 kB, 36 KiB) copied, 0.000816002 s, 45.2 MB/s 2020-11-16T21:30:43.327 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: expr 50 + 1 2020-11-16T21:30:43.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: seq 51 200 2020-11-16T21:30:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj51 td/osd-backfill-space/36kdata 2020-11-16T21:30:43.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:43.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj52 td/osd-backfill-space/36kdata 2020-11-16T21:30:43.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:43.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj53 td/osd-backfill-space/36kdata 2020-11-16T21:30:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:43.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj54 td/osd-backfill-space/36kdata 2020-11-16T21:30:44.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:44.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj55 td/osd-backfill-space/36kdata 2020-11-16T21:30:44.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:44.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj56 td/osd-backfill-space/36kdata 2020-11-16T21:30:44.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:44.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj57 td/osd-backfill-space/36kdata 2020-11-16T21:30:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:44.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj58 td/osd-backfill-space/36kdata 2020-11-16T21:30:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:45.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj59 td/osd-backfill-space/36kdata 2020-11-16T21:30:45.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:45.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj60 td/osd-backfill-space/36kdata 2020-11-16T21:30:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:45.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj61 td/osd-backfill-space/36kdata 2020-11-16T21:30:45.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:45.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj62 td/osd-backfill-space/36kdata 2020-11-16T21:30:46.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:46.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj63 td/osd-backfill-space/36kdata 2020-11-16T21:30:46.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:46.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj64 td/osd-backfill-space/36kdata 2020-11-16T21:30:46.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:46.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj65 td/osd-backfill-space/36kdata 2020-11-16T21:30:46.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:46.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj66 td/osd-backfill-space/36kdata 2020-11-16T21:30:47.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:47.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj67 td/osd-backfill-space/36kdata 2020-11-16T21:30:47.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:47.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj68 td/osd-backfill-space/36kdata 2020-11-16T21:30:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj69 td/osd-backfill-space/36kdata 2020-11-16T21:30:47.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:47.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj70 td/osd-backfill-space/36kdata 2020-11-16T21:30:48.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:48.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj71 td/osd-backfill-space/36kdata 2020-11-16T21:30:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj72 td/osd-backfill-space/36kdata 2020-11-16T21:30:48.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:48.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj73 td/osd-backfill-space/36kdata 2020-11-16T21:30:49.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:49.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj74 td/osd-backfill-space/36kdata 2020-11-16T21:30:49.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:49.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj75 td/osd-backfill-space/36kdata 2020-11-16T21:30:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj76 td/osd-backfill-space/36kdata 2020-11-16T21:30:49.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:49.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj77 td/osd-backfill-space/36kdata 2020-11-16T21:30:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj78 td/osd-backfill-space/36kdata 2020-11-16T21:30:50.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:50.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj79 td/osd-backfill-space/36kdata 2020-11-16T21:30:50.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:50.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj80 td/osd-backfill-space/36kdata 2020-11-16T21:30:50.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:50.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj81 td/osd-backfill-space/36kdata 2020-11-16T21:30:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj82 td/osd-backfill-space/36kdata 2020-11-16T21:30:51.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:51.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj83 td/osd-backfill-space/36kdata 2020-11-16T21:30:51.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:51.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj84 td/osd-backfill-space/36kdata 2020-11-16T21:30:51.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:51.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj85 td/osd-backfill-space/36kdata 2020-11-16T21:30:52.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:52.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj86 td/osd-backfill-space/36kdata 2020-11-16T21:30:52.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:52.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj87 td/osd-backfill-space/36kdata 2020-11-16T21:30:52.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:52.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj88 td/osd-backfill-space/36kdata 2020-11-16T21:30:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:52.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj89 td/osd-backfill-space/36kdata 2020-11-16T21:30:53.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:53.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj90 td/osd-backfill-space/36kdata 2020-11-16T21:30:53.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:53.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj91 td/osd-backfill-space/36kdata 2020-11-16T21:30:53.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:53.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj92 td/osd-backfill-space/36kdata 2020-11-16T21:30:53.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:53.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj93 td/osd-backfill-space/36kdata 2020-11-16T21:30:54.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj94 td/osd-backfill-space/36kdata 2020-11-16T21:30:54.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:54.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj95 td/osd-backfill-space/36kdata 2020-11-16T21:30:54.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:54.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj96 td/osd-backfill-space/36kdata 2020-11-16T21:30:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj97 td/osd-backfill-space/36kdata 2020-11-16T21:30:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj98 td/osd-backfill-space/36kdata 2020-11-16T21:30:55.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:55.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj99 td/osd-backfill-space/36kdata 2020-11-16T21:30:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj100 td/osd-backfill-space/36kdata 2020-11-16T21:30:55.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:55.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj101 td/osd-backfill-space/36kdata 2020-11-16T21:30:56.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:56.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj102 td/osd-backfill-space/36kdata 2020-11-16T21:30:56.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:56.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj103 td/osd-backfill-space/36kdata 2020-11-16T21:30:56.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:56.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj104 td/osd-backfill-space/36kdata 2020-11-16T21:30:56.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:56.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj105 td/osd-backfill-space/36kdata 2020-11-16T21:30:57.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:57.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj106 td/osd-backfill-space/36kdata 2020-11-16T21:30:57.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:57.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj107 td/osd-backfill-space/36kdata 2020-11-16T21:30:57.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:57.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj108 td/osd-backfill-space/36kdata 2020-11-16T21:30:57.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:57.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj109 td/osd-backfill-space/36kdata 2020-11-16T21:30:57.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:57.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj110 td/osd-backfill-space/36kdata 2020-11-16T21:30:58.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:58.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj111 td/osd-backfill-space/36kdata 2020-11-16T21:30:58.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:58.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj112 td/osd-backfill-space/36kdata 2020-11-16T21:30:58.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:58.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj113 td/osd-backfill-space/36kdata 2020-11-16T21:30:58.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:58.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj114 td/osd-backfill-space/36kdata 2020-11-16T21:30:59.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:59.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj115 td/osd-backfill-space/36kdata 2020-11-16T21:30:59.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:59.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj116 td/osd-backfill-space/36kdata 2020-11-16T21:30:59.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:59.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj117 td/osd-backfill-space/36kdata 2020-11-16T21:30:59.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:30:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj118 td/osd-backfill-space/36kdata 2020-11-16T21:31:00.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:00.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj119 td/osd-backfill-space/36kdata 2020-11-16T21:31:00.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:00.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj120 td/osd-backfill-space/36kdata 2020-11-16T21:31:00.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:00.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj121 td/osd-backfill-space/36kdata 2020-11-16T21:31:00.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:00.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj122 td/osd-backfill-space/36kdata 2020-11-16T21:31:01.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:01.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj123 td/osd-backfill-space/36kdata 2020-11-16T21:31:01.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:01.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj124 td/osd-backfill-space/36kdata 2020-11-16T21:31:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj125 td/osd-backfill-space/36kdata 2020-11-16T21:31:01.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:01.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj126 td/osd-backfill-space/36kdata 2020-11-16T21:31:02.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:02.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj127 td/osd-backfill-space/36kdata 2020-11-16T21:31:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:02.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj128 td/osd-backfill-space/36kdata 2020-11-16T21:31:02.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:02.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj129 td/osd-backfill-space/36kdata 2020-11-16T21:31:03.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:03.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj130 td/osd-backfill-space/36kdata 2020-11-16T21:31:03.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:03.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj131 td/osd-backfill-space/36kdata 2020-11-16T21:31:03.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:03.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj132 td/osd-backfill-space/36kdata 2020-11-16T21:31:03.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:03.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj133 td/osd-backfill-space/36kdata 2020-11-16T21:31:04.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:04.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj134 td/osd-backfill-space/36kdata 2020-11-16T21:31:04.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:04.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj135 td/osd-backfill-space/36kdata 2020-11-16T21:31:04.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:04.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj136 td/osd-backfill-space/36kdata 2020-11-16T21:31:04.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:04.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj137 td/osd-backfill-space/36kdata 2020-11-16T21:31:05.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:05.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj138 td/osd-backfill-space/36kdata 2020-11-16T21:31:05.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:05.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj139 td/osd-backfill-space/36kdata 2020-11-16T21:31:05.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:05.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj140 td/osd-backfill-space/36kdata 2020-11-16T21:31:05.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:05.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj141 td/osd-backfill-space/36kdata 2020-11-16T21:31:05.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:05.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj142 td/osd-backfill-space/36kdata 2020-11-16T21:31:06.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:06.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj143 td/osd-backfill-space/36kdata 2020-11-16T21:31:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj144 td/osd-backfill-space/36kdata 2020-11-16T21:31:06.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:06.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj145 td/osd-backfill-space/36kdata 2020-11-16T21:31:07.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:07.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj146 td/osd-backfill-space/36kdata 2020-11-16T21:31:07.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:07.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj147 td/osd-backfill-space/36kdata 2020-11-16T21:31:07.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:07.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj148 td/osd-backfill-space/36kdata 2020-11-16T21:31:07.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:07.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj149 td/osd-backfill-space/36kdata 2020-11-16T21:31:08.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:08.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj150 td/osd-backfill-space/36kdata 2020-11-16T21:31:08.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:08.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj151 td/osd-backfill-space/36kdata 2020-11-16T21:31:08.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:08.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj152 td/osd-backfill-space/36kdata 2020-11-16T21:31:08.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:08.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj153 td/osd-backfill-space/36kdata 2020-11-16T21:31:09.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:09.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj154 td/osd-backfill-space/36kdata 2020-11-16T21:31:09.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:09.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj155 td/osd-backfill-space/36kdata 2020-11-16T21:31:09.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:09.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj156 td/osd-backfill-space/36kdata 2020-11-16T21:31:09.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:09.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj157 td/osd-backfill-space/36kdata 2020-11-16T21:31:09.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:09.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj158 td/osd-backfill-space/36kdata 2020-11-16T21:31:10.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:10.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj159 td/osd-backfill-space/36kdata 2020-11-16T21:31:10.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:10.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj160 td/osd-backfill-space/36kdata 2020-11-16T21:31:10.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:10.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj161 td/osd-backfill-space/36kdata 2020-11-16T21:31:10.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:10.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj162 td/osd-backfill-space/36kdata 2020-11-16T21:31:11.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:11.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj163 td/osd-backfill-space/36kdata 2020-11-16T21:31:11.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:11.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj164 td/osd-backfill-space/36kdata 2020-11-16T21:31:11.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:11.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj165 td/osd-backfill-space/36kdata 2020-11-16T21:31:11.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:11.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj166 td/osd-backfill-space/36kdata 2020-11-16T21:31:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj167 td/osd-backfill-space/36kdata 2020-11-16T21:31:12.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:12.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj168 td/osd-backfill-space/36kdata 2020-11-16T21:31:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj169 td/osd-backfill-space/36kdata 2020-11-16T21:31:12.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:12.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj170 td/osd-backfill-space/36kdata 2020-11-16T21:31:13.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:13.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj171 td/osd-backfill-space/36kdata 2020-11-16T21:31:13.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:13.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj172 td/osd-backfill-space/36kdata 2020-11-16T21:31:13.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:13.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj173 td/osd-backfill-space/36kdata 2020-11-16T21:31:13.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:13.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj174 td/osd-backfill-space/36kdata 2020-11-16T21:31:14.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:14.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj175 td/osd-backfill-space/36kdata 2020-11-16T21:31:14.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:14.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj176 td/osd-backfill-space/36kdata 2020-11-16T21:31:14.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:14.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj177 td/osd-backfill-space/36kdata 2020-11-16T21:31:14.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:14.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj178 td/osd-backfill-space/36kdata 2020-11-16T21:31:15.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:15.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj179 td/osd-backfill-space/36kdata 2020-11-16T21:31:15.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:15.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj180 td/osd-backfill-space/36kdata 2020-11-16T21:31:15.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:15.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj181 td/osd-backfill-space/36kdata 2020-11-16T21:31:15.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:15.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj182 td/osd-backfill-space/36kdata 2020-11-16T21:31:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj183 td/osd-backfill-space/36kdata 2020-11-16T21:31:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj184 td/osd-backfill-space/36kdata 2020-11-16T21:31:16.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:16.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj185 td/osd-backfill-space/36kdata 2020-11-16T21:31:16.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:16.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj186 td/osd-backfill-space/36kdata 2020-11-16T21:31:17.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:17.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj187 td/osd-backfill-space/36kdata 2020-11-16T21:31:17.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:17.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj188 td/osd-backfill-space/36kdata 2020-11-16T21:31:17.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:17.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj189 td/osd-backfill-space/36kdata 2020-11-16T21:31:17.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:17.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj190 td/osd-backfill-space/36kdata 2020-11-16T21:31:18.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:18.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj191 td/osd-backfill-space/36kdata 2020-11-16T21:31:18.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:18.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj192 td/osd-backfill-space/36kdata 2020-11-16T21:31:18.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:18.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj193 td/osd-backfill-space/36kdata 2020-11-16T21:31:18.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:18.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj194 td/osd-backfill-space/36kdata 2020-11-16T21:31:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj195 td/osd-backfill-space/36kdata 2020-11-16T21:31:19.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:19.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj196 td/osd-backfill-space/36kdata 2020-11-16T21:31:19.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:19.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj197 td/osd-backfill-space/36kdata 2020-11-16T21:31:19.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:19.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj198 td/osd-backfill-space/36kdata 2020-11-16T21:31:20.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:20.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj199 td/osd-backfill-space/36kdata 2020-11-16T21:31:20.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2020-11-16T21:31:20.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj200 td/osd-backfill-space/36kdata 2020-11-16T21:31:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1159: TEST_ec_backfill_grow: activate_osd td/osd-backfill-space 0 2020-11-16T21:31:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2020-11-16T21:31:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T21:31:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T21:31:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T21:31:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/0 2020-11-16T21:31:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:31:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:31:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:31:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:31:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2020-11-16T21:31:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2020-11-16T21:31:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T21:31:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T21:31:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:31:20.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T21:31:20.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:31:20.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:31:20.556 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:31:20.556 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:31:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:31:20.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:31:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:31:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:31:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:31:20.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:31:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:31:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:31:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:31:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T21:31:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T21:31:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/0 2020-11-16T21:31:20.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T21:31:20.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:31:20.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/0/whoami 2020-11-16T21:31:20.559 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:31:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T21:31:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T21:31:20.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T21:31:20.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T21:31:20.588 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:31:20.595+0000 7f8b72ed3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:31:20.591 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:31:20.599+0000 7f8b72ed3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:31:20.592 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:31:20.599+0000 7f8b72ed3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:31:20.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T21:31:20.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:31:20.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:31:20.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:31:20.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:31:20.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:31:20.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:31:20.967 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:31:20.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:31:20.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:31:21.140 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:31:21.147+0000 7f8b72ed3d80 -1 Falling back to public interface 2020-11-16T21:31:21.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:31:22.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:31:22.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:31:22.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:31:22.367 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:31:22.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:31:22.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:31:22.582 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:31:22.587+0000 7f8b72ed3d80 -1 osd.0 39 log_to_monitors {default=true} 2020-11-16T21:31:22.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:31:23.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:31:23.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:31:23.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:31:23.759 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:31:23.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:31:23.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:31:24.179 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 44 up_thru 0 down_at 42 last_clean_interval [5,39) [v2:127.0.0.1:6802/426536,v1:127.0.0.1:6803/426536] [v2:127.0.0.1:6804/426536,v1:127.0.0.1:6805/426536] exists,up 6711bafb-5165-41b9-bed9-ceaa296f93f2 2020-11-16T21:31:24.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:31:24.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:31:24.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:31:24.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1161: TEST_ec_backfill_grow: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T21:31:24.320 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T21:31:24.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1163: TEST_ec_backfill_grow: sleep 2 2020-11-16T21:31:26.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1165: TEST_ec_backfill_grow: wait_for_clean 2020-11-16T21:31:26.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:31:26.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:31:26.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:31:26.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:31:26.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:31:26.332 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:31:26.332 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:31:26.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:31:26.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:31:26.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:31:26.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:31:26.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:31:26.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:31:26.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:31:26.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:31:26.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:31:26.968 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:31:26.968 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:31:26.968 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:31:26.968 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:31:26.968 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:31:26.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:31:26.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:31:26.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:31:27.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=188978561027 2020-11-16T21:31:27.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 188978561027 2020-11-16T21:31:27.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-188978561027' 2020-11-16T21:31:27.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:31:27.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:31:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672996 2020-11-16T21:31:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672996 2020-11-16T21:31:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-188978561027 1-42949672996' 2020-11-16T21:31:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:31:27.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:31:27.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542177 2020-11-16T21:31:27.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542177 2020-11-16T21:31:27.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-188978561027 1-42949672996 2-60129542177' 2020-11-16T21:31:27.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:31:27.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:31:27.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378656 2020-11-16T21:31:27.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378656 2020-11-16T21:31:27.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-188978561027 1-42949672996 2-60129542177 3-81604378656' 2020-11-16T21:31:27.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:31:27.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:31:27.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215134 2020-11-16T21:31:27.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215134 2020-11-16T21:31:27.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-188978561027 1-42949672996 2-60129542177 3-81604378656 4-103079215134' 2020-11-16T21:31:27.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:31:27.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:31:27.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051613 2020-11-16T21:31:27.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051613 2020-11-16T21:31:27.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-188978561027 1-42949672996 2-60129542177 3-81604378656 4-103079215134 5-124554051613' 2020-11-16T21:31:27.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:31:27.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-188978561027 2020-11-16T21:31:27.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:31:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:31:27.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-188978561027 2020-11-16T21:31:27.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:31:27.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=188978561027 2020-11-16T21:31:27.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 188978561027' 2020-11-16T21:31:27.984 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 188978561027 2020-11-16T21:31:27.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:31:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 188978561028 -lt 188978561027 2020-11-16T21:31:28.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:31:28.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672996 2020-11-16T21:31:28.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:31:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:31:28.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672996 2020-11-16T21:31:28.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:31:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672996 2020-11-16T21:31:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2020-11-16T21:31:28.386 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672996 2020-11-16T21:31:28.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:31:28.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672996 -lt 42949672996 2020-11-16T21:31:28.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:31:28.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542177 2020-11-16T21:31:28.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:31:28.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:31:28.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:31:28.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542177 2020-11-16T21:31:28.791 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542177 2020-11-16T21:31:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542177 2020-11-16T21:31:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542177' 2020-11-16T21:31:28.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:31:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542177 -lt 60129542177 2020-11-16T21:31:29.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:31:29.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378656 2020-11-16T21:31:29.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:31:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:31:29.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378656 2020-11-16T21:31:29.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:31:29.211 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378656 2020-11-16T21:31:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378656 2020-11-16T21:31:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378656' 2020-11-16T21:31:29.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:31:29.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378656 -lt 81604378656 2020-11-16T21:31:29.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:31:29.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215134 2020-11-16T21:31:29.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:31:29.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:31:29.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215134 2020-11-16T21:31:29.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:31:29.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215134 2020-11-16T21:31:29.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215134' 2020-11-16T21:31:29.623 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215134 2020-11-16T21:31:29.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:31:30.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215134 -lt 103079215134 2020-11-16T21:31:30.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:31:30.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051613 2020-11-16T21:31:30.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:31:30.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:31:30.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:31:30.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051613 2020-11-16T21:31:30.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051613 2020-11-16T21:31:30.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051613' 2020-11-16T21:31:30.017 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051613 2020-11-16T21:31:30.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:31:30.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051613 -lt 124554051613 2020-11-16T21:31:30.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:31:30.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:30.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:30.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:31:30.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:30.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:30.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:30.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:30.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:30.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:30.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:31.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:31.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:31.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:31.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:31.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:31.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T21:31:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T21:31:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T21:31:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:31.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:31.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:31.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:31.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:31.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:31.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:31.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:31.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:32.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:32.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:32.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:32.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:32.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:32.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:32.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:33.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=128596 2020-11-16T21:31:33.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 128596 '!=' null 2020-11-16T21:31:33.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:33.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:33.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:33.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:33.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:33.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:33.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:33.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:33.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:33.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:33.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:33.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:33.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:33.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:33.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:34.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:34.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:34.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:34.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:34.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:34.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:34.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:34.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:34.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:34.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=108173 2020-11-16T21:31:34.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 108173 '!=' null 2020-11-16T21:31:34.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:34.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:34.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:34.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:34.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:34.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:34.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:34.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:34.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:34.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:34.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:35.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:35.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:35.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:35.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:35.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:35.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:35.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=108173 2020-11-16T21:31:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 108173 '!=' null 2020-11-16T21:31:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:36.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:36.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:36.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:36.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:36.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:36.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:36.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:36.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:36.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:36.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:36.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:36.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:37.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:37.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:37.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:37.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:37.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:37.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:37.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:37.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:37.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:37.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=92136 2020-11-16T21:31:37.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 92136 '!=' null 2020-11-16T21:31:37.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:37.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:37.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:37.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:37.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:37.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:37.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:37.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:37.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:38.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:38.098 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:38.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:38.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:38.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:38.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:38.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:38.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:38.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:38.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:38.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:38.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:38.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:39.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=159704 2020-11-16T21:31:39.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 159704 '!=' null 2020-11-16T21:31:39.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:39.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:39.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:39.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:39.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:39.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:39.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:39.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:39.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:39.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:39.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:39.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:39.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:39.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:39.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:40.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:40.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:40.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:40.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:40.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:40.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:40.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:40.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:40.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:40.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=125920 2020-11-16T21:31:40.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 125920 '!=' null 2020-11-16T21:31:40.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:40.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:40.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:40.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:40.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:40.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:40.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:40.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:40.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:40.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:40.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:40.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:41.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:41.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:41.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:41.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:41.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:41.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:41.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:41.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:41.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:41.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:41.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=125920 2020-11-16T21:31:41.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 125920 '!=' null 2020-11-16T21:31:41.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:41.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:41.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:42.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:42.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:42.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:42.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:42.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:42.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:42.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:42.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:42.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:42.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:42.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:42.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:43.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:43.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:43.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:43.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:43.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:43.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:43.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:43.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:43.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:43.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=196553 2020-11-16T21:31:43.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 196553 '!=' null 2020-11-16T21:31:43.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:43.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:43.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:43.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:43.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:43.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:43.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:43.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:43.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:43.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:43.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:44.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:44.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:44.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:44.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:44.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:44.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:44.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:44.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:44.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:44.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:44.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:44.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:44.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:44.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138199 2020-11-16T21:31:44.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138199 '!=' null 2020-11-16T21:31:44.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:44.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:44.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:45.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:45.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:45.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:45.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:45.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:45.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:45.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:45.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:45.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:45.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:45.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:45.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:45.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:45.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:45.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:46.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138198 2020-11-16T21:31:46.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138198 '!=' null 2020-11-16T21:31:46.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:46.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:46.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:46.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:46.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:46.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:46.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:46.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:46.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:46.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:46.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:46.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:46.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:46.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:46.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:47.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:47.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:47.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:47.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:47.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:47.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:47.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:47.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:47.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:47.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138198 2020-11-16T21:31:47.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138198 '!=' null 2020-11-16T21:31:47.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:47.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:47.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:48.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:48.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:48.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:48.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:48.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:48.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:48.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:48.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:48.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:48.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:48.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:48.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:48.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:48.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:48.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:48.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:48.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:48.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:48.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:48.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=208837 2020-11-16T21:31:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 208837 '!=' null 2020-11-16T21:31:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:49.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:49.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:49.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:49.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:49.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:49.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:49.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:49.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:49.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:49.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:49.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:49.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:49.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:49.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:50.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:50.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:50.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=141274 2020-11-16T21:31:50.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 141274 '!=' null 2020-11-16T21:31:50.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:50.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:50.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:50.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:50.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:50.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:50.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:50.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:50.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:50.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:50.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:51.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:51.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:51.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:51.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:51.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:51.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:51.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:51.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:51.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:51.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:51.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:51.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:51.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:52.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=208843 2020-11-16T21:31:52.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 208843 '!=' null 2020-11-16T21:31:52.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:52.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:52.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:52.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:52.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:52.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:52.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:52.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:52.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:52.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:52.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:52.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:52.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:53.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:53.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:53.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:53.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=208843 2020-11-16T21:31:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 208843 '!=' null 2020-11-16T21:31:53.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:53.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:53.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:53.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:53.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:53.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:53.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:53.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:53.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:53.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:54.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:54.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:54.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:54.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:54.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:54.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:54.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:54.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:54.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:55.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138208 2020-11-16T21:31:55.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138208 '!=' null 2020-11-16T21:31:55.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:55.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:55.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:55.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:55.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:55.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:55.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:55.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:55.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:55.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:55.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:55.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:55.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:55.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:55.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:56.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:56.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:56.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:56.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:56.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:56.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:56.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:56.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:56.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:56.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138208 2020-11-16T21:31:56.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138208 '!=' null 2020-11-16T21:31:56.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:56.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:56.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:56.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:56.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:56.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:56.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:56.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:56.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:56.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:56.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:57.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:57.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:57.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:57.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:57.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:57.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:57.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:57.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:58.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138208 2020-11-16T21:31:58.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138208 '!=' null 2020-11-16T21:31:58.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:58.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:58.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:58.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:58.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:58.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:58.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:58.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:58.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:58.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:58.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:31:58.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:31:58.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:31:58.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:31:58.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:31:59.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:31:59.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:31:59.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:31:59.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:31:59.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:31:59.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:31:59.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:31:59.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:31:59.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:31:59.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=205780 2020-11-16T21:31:59.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 205780 '!=' null 2020-11-16T21:31:59.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:31:59.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:31:59.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:31:59.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:31:59.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:31:59.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:31:59.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:31:59.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:31:59.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:31:59.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:31:59.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:32:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:32:00.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:32:00.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:32:00.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:32:00.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:32:00.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:32:00.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:32:00.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:32:00.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:32:00.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:32:00.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:32:00.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:32:00.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:32:01.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=101352 2020-11-16T21:32:01.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 101352 '!=' null 2020-11-16T21:32:01.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:32:01.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:32:01.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:32:01.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:32:01.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:32:01.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:32:01.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:32:01.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:32:01.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:32:01.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:32:01.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:32:01.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:32:01.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:32:01.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:32:01.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:32:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:32:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:32:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:32:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:32:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:32:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:32:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:32:02.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:32:02.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:32:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138210 2020-11-16T21:32:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138210 '!=' null 2020-11-16T21:32:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:32:02.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:32:02.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:32:02.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:32:02.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:32:02.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:32:02.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:32:02.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:32:02.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:32:02.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:32:02.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:32:02.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:32:02.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:32:02.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:32:02.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:32:03.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:32:03.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:32:03.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:32:03.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1167: TEST_ec_backfill_grow: delete_pool test 2020-11-16T21:32:03.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T21:32:03.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T21:32:04.054 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T21:32:04.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1168: TEST_ec_backfill_grow: kill_daemons td/osd-backfill-space 2020-11-16T21:32:04.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:32:04.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:32:04.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:32:04.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:32:04.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:32:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:32:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2020-11-16T21:32:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:32:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:32:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:32:04.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:32:04.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:32:04.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:32:04.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:32:04.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:32:04.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:32:04.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:32:04.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:32:04.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:32:04.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:32:04.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:32:04.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:32:04.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:32:04.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:32:04.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:32:04.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:32:04.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:32:04.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:32:04.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:32:04.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:32:04.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:04.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:04.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:32:04.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:32:04.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:32:04.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2020-11-16T21:32:04.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2020-11-16T21:32:04.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2020-11-16T21:32:04.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2020-11-16T21:32:04.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:32:04.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:32:04.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:32:04.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:32:04.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:32:04.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:32:04.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:32:04.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:32:04.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:32:04.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:32:04.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:32:04.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:32:04.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:32:04.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:32:04.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:32:04.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:32:04.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:32:04.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:32:04.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:32:04.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:32:04.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:32:04.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:32:04.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:32:04.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:04.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:04.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:32:04.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:32:04.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:32:04.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2020-11-16T21:32:04.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:32:04.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:04.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:04.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137323 2020-11-16T21:32:04.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:32:04.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:32:04.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:32:04.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2020-11-16T21:32:04.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_ec_backfill_multi td/osd-backfill-space 2020-11-16T21:32:04.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:748: TEST_ec_backfill_multi: local dir=td/osd-backfill-space 2020-11-16T21:32:04.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:749: TEST_ec_backfill_multi: local EC= 2020-11-16T21:32:04.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:750: TEST_ec_backfill_multi: local pools=2 2020-11-16T21:32:04.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:751: TEST_ec_backfill_multi: local OSDS=6 2020-11-16T21:32:04.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:752: TEST_ec_backfill_multi: local k=3 2020-11-16T21:32:04.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:753: TEST_ec_backfill_multi: local m=2 2020-11-16T21:32:04.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:754: TEST_ec_backfill_multi: expr 600 / 3 2020-11-16T21:32:04.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:754: TEST_ec_backfill_multi: local ecobjects=200 2020-11-16T21:32:04.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:756: TEST_ec_backfill_multi: run_mon td/osd-backfill-space a 2020-11-16T21:32:04.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2020-11-16T21:32:04.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:32:04.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:32:04.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:32:04.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2020-11-16T21:32:04.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2020-11-16T21:32:04.779 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:32:04.779 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:32:04.779 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:32:04.780 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:32:04.780 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:04.780 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:04.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:04.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:32:04.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:32:04.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:32:04.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:32:04.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:32:04.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:32:04.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:32:04.984 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:32:04.985 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:32:04.985 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:32:04.985 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:32:04.985 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:04.986 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:04.986 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:32:04.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:32:04.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get fsid 2020-11-16T21:32:05.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:32:05.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:32:05.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:32:05.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:32:05.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:32:05.093 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:32:05.093 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:32:05.093 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:32:05.094 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:32:05.094 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:05.094 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:05.094 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:32:05.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:32:05.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get mon_host 2020-11-16T21:32:05.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:757: TEST_ec_backfill_multi: run_mgr td/osd-backfill-space x 2020-11-16T21:32:05.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2020-11-16T21:32:05.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:32:05.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:32:05.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:32:05.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2020-11-16T21:32:05.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:32:05.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:32:05.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:32:05.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:32:05.730 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:32:05.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:05.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:05.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:05.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:32:05.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:32:05.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:758: TEST_ec_backfill_multi: export CEPH_ARGS 2020-11-16T21:32:05.763 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: expr 6 - 1 2020-11-16T21:32:05.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: seq 0 5 2020-11-16T21:32:05.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:32:05.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 0 2020-11-16T21:32:05.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:32:05.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:32:05.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:32:05.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:32:05.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2020-11-16T21:32:05.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:32:05.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:32:05.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:32:05.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:32:05.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2020-11-16T21:32:05.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2020-11-16T21:32:05.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:32:05.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:32:05.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:32:05.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:32:05.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:32:05.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:32:05.771 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:32:05.771 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:05.771 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:05.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:05.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:05.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:32:05.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:32:05.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:32:05.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:32:05.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:32:05.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:32:05.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:32:05.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:05.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:32:05.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:32:05.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2020-11-16T21:32:05.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:32:05.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d9bf3eae-842f-4493-8e61-8960019acec5 2020-11-16T21:32:05.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 d9bf3eae-842f-4493-8e61-8960019acec5' 2020-11-16T21:32:05.777 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 d9bf3eae-842f-4493-8e61-8960019acec5 2020-11-16T21:32:05.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:32:05.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDV77JfOtwTMBAAolJXD36GaUE1CEMnm4VhIg== 2020-11-16T21:32:05.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDV77JfOtwTMBAAolJXD36GaUE1CEMnm4VhIg=="}' 2020-11-16T21:32:05.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d9bf3eae-842f-4493-8e61-8960019acec5 -i td/osd-backfill-space/0/new.json 2020-11-16T21:32:06.129 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:32:06.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2020-11-16T21:32:06.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDV77JfOtwTMBAAolJXD36GaUE1CEMnm4VhIg== --osd-uuid d9bf3eae-842f-4493-8e61-8960019acec5 2020-11-16T21:32:06.179 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:06.186+0000 7fa7709edd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:06.182 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:06.190+0000 7fa7709edd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:06.184 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:06.190+0000 7fa7709edd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:06.184 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:06.190+0000 7fa7709edd80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:32:06.184 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:06.190+0000 7fa7709edd80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:32:06.185 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:06.190+0000 7fa7709edd80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:32:06.237 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:06.242+0000 7fa7709edd80 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2020-11-16T21:32:10.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2020-11-16T21:32:10.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:32:10.778 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:32:10.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:32:10.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:32:11.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:32:11.269 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:32:11.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:11.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:32:11.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:32:11.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:32:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:11.306+0000 7fbf0bc36d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:11.304 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:11.310+0000 7fbf0bc36d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:11.306 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:11.310+0000 7fbf0bc36d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:11.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:32:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:32:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:32:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:32:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:32:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:11.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:32:11.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:11.653 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:32:11.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:32:11.845 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:11.850+0000 7fbf0bc36d80 -1 Falling back to public interface 2020-11-16T21:32:12.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:32:12.674 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:12.678+0000 7fbf0bc36d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:32:13.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:32:13.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:13.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:32:13.033 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:32:13.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:13.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:32:13.427 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/430168,v1:127.0.0.1:6803/430168] [v2:127.0.0.1:6804/430168,v1:127.0.0.1:6805/430168] exists,up d9bf3eae-842f-4493-8e61-8960019acec5 2020-11-16T21:32:13.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:32:13.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:32:13.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:32:13.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:32:13.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 1 2020-11-16T21:32:13.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:32:13.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:32:13.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:32:13.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:32:13.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2020-11-16T21:32:17.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:32:17.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:32:17.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:32:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:32:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2020-11-16T21:32:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2020-11-16T21:32:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:32:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:32:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:32:17.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:32:17.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:32:17.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:32:17.566 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:32:17.567 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:17.567 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:17.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:17.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:17.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:32:17.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:32:17.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:32:17.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:32:17.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:32:17.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:32:17.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:32:17.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:17.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:32:17.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:32:17.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2020-11-16T21:32:17.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:32:17.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7c9dbf94-092b-4432-84b5-9cfe5b143266 2020-11-16T21:32:17.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 7c9dbf94-092b-4432-84b5-9cfe5b143266' 2020-11-16T21:32:17.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:32:17.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDd77Jf9+KeGxAA+3oLWsMH1g/VDl2IN8atWw== 2020-11-16T21:32:17.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDd77Jf9+KeGxAA+3oLWsMH1g/VDl2IN8atWw=="}' 2020-11-16T21:32:17.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7c9dbf94-092b-4432-84b5-9cfe5b143266 -i td/osd-backfill-space/1/new.json 2020-11-16T21:32:17.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2020-11-16T21:32:17.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDd77Jf9+KeGxAA+3oLWsMH1g/VDl2IN8atWw== --osd-uuid 7c9dbf94-092b-4432-84b5-9cfe5b143266 2020-11-16T21:32:17.570 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:14.042+0000 7fa100c71d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:17.570 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:14.046+0000 7fa100c71d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:17.570 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:14.046+0000 7fa100c71d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:17.570 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:14.046+0000 7fa100c71d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:32:17.571 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:14.046+0000 7fa100c71d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:32:17.571 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:14.046+0000 7fa100c71d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:32:17.571 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:14.102+0000 7fa100c71d80 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2020-11-16T21:32:17.571 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 7c9dbf94-092b-4432-84b5-9cfe5b143266 2020-11-16T21:32:17.571 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:32:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2020-11-16T21:32:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:32:18.706 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:32:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:32:18.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:32:19.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:32:19.203 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:32:19.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:32:19.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:19.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:32:19.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:32:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:19.242+0000 7f66883a1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:19.246+0000 7f66883a1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:19.246+0000 7f66883a1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:19.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:32:19.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:32:19.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:32:19.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:32:19.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:32:19.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:19.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:32:19.581 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:32:19.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:19.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:32:19.773 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:19.778+0000 7f66883a1d80 -1 Falling back to public interface 2020-11-16T21:32:19.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:32:20.610 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:20.614+0000 7f66883a1d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:32:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:32:20.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:32:20.976 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:32:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:20.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:32:21.370 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/431162,v1:127.0.0.1:6811/431162] [v2:127.0.0.1:6812/431162,v1:127.0.0.1:6813/431162] exists,up 7c9dbf94-092b-4432-84b5-9cfe5b143266 2020-11-16T21:32:21.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:32:21.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:32:21.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:32:21.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:32:21.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 2 2020-11-16T21:32:21.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:32:21.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:32:21.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:32:21.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:32:21.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2020-11-16T21:32:21.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:32:21.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:32:21.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:32:21.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:32:21.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2020-11-16T21:32:21.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2020-11-16T21:32:21.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:32:21.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:32:21.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:32:21.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:32:21.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:32:21.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:32:21.375 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:32:21.375 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:21.376 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:21.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:21.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:21.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:32:21.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:32:21.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:32:21.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:32:21.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:32:21.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:32:21.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:32:21.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:21.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:32:21.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:32:21.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2020-11-16T21:32:21.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:32:21.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=835d34f6-948a-4afc-b995-a0eb6184c294 2020-11-16T21:32:21.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 835d34f6-948a-4afc-b995-a0eb6184c294' 2020-11-16T21:32:21.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:32:21.380 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 835d34f6-948a-4afc-b995-a0eb6184c294 2020-11-16T21:32:21.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDl77JfOJRLGBAAHZwpLJbmbnRZ/cq9N8DrGw== 2020-11-16T21:32:21.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDl77JfOJRLGBAAHZwpLJbmbnRZ/cq9N8DrGw=="}' 2020-11-16T21:32:21.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 835d34f6-948a-4afc-b995-a0eb6184c294 -i td/osd-backfill-space/2/new.json 2020-11-16T21:32:21.900 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:32:21.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2020-11-16T21:32:21.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDl77JfOJRLGBAAHZwpLJbmbnRZ/cq9N8DrGw== --osd-uuid 835d34f6-948a-4afc-b995-a0eb6184c294 2020-11-16T21:32:21.948 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:21.954+0000 7f9f4b461d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:21.950 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:21.958+0000 7f9f4b461d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:21.952 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:21.958+0000 7f9f4b461d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:21.952 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:21.958+0000 7f9f4b461d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:32:21.953 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:21.958+0000 7f9f4b461d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:32:21.953 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:21.958+0000 7f9f4b461d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:32:22.008 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:22.014+0000 7f9f4b461d80 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2020-11-16T21:32:26.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2020-11-16T21:32:26.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:32:26.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:32:26.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:32:26.338 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:32:26.841 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:32:26.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:32:26.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:26.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:32:26.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:32:26.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:32:26.874 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:26.878+0000 7f707e659d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:26.876 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:26.882+0000 7f707e659d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:26.877 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:26.882+0000 7f707e659d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:27.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:32:27.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:32:27.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:32:27.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:32:27.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:32:27.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:27.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:32:27.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:27.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:32:27.242 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:32:27.429 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:27.434+0000 7f707e659d80 -1 Falling back to public interface 2020-11-16T21:32:27.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:32:28.289 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:28.286+0000 7f707e659d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:32:28.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:32:28.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:28.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:32:28.611 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:32:28.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:28.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:32:29.016 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/432156,v1:127.0.0.1:6819/432156] [v2:127.0.0.1:6820/432156,v1:127.0.0.1:6821/432156] exists,up 835d34f6-948a-4afc-b995-a0eb6184c294 2020-11-16T21:32:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:32:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:32:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:32:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:32:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 3 2020-11-16T21:32:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:32:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:32:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:32:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:32:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/3 2020-11-16T21:32:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:32:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:32:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:32:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:32:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/3' 2020-11-16T21:32:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/3/journal' 2020-11-16T21:32:29.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:32:29.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:32:29.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:32:29.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:32:29.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:32:29.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:32:29.020 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:32:29.021 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:29.021 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:29.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:29.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:29.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:32:29.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:32:29.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:32:29.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:32:29.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:32:29.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:32:29.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:32:29.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:29.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:32:29.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:32:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/3 2020-11-16T21:32:29.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:32:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b3f070dc-78f4-40f6-9f3e-df7f070d6048 2020-11-16T21:32:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 b3f070dc-78f4-40f6-9f3e-df7f070d6048' 2020-11-16T21:32:29.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:32:29.024 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 b3f070dc-78f4-40f6-9f3e-df7f070d6048 2020-11-16T21:32:29.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDt77Jf/vQ4AxAAJiVST5tczxg5eBrFiwNuWg== 2020-11-16T21:32:29.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDt77Jf/vQ4AxAAJiVST5tczxg5eBrFiwNuWg=="}' 2020-11-16T21:32:29.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b3f070dc-78f4-40f6-9f3e-df7f070d6048 -i td/osd-backfill-space/3/new.json 2020-11-16T21:32:29.563 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:32:29.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/3/new.json 2020-11-16T21:32:29.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDt77Jf/vQ4AxAAJiVST5tczxg5eBrFiwNuWg== --osd-uuid b3f070dc-78f4-40f6-9f3e-df7f070d6048 2020-11-16T21:32:29.612 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:29.618+0000 7effa1c96d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:29.616 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:29.622+0000 7effa1c96d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:29.617 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:29.622+0000 7effa1c96d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:29.618 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:29.622+0000 7effa1c96d80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:32:29.619 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:29.626+0000 7effa1c96d80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:32:29.619 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:29.626+0000 7effa1c96d80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:32:29.709 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:29.714+0000 7effa1c96d80 -1 bluestore(td/osd-backfill-space/3) _read_fsid unparsable uuid 2020-11-16T21:32:33.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/3/keyring 2020-11-16T21:32:33.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:32:33.988 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:32:33.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:32:33.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:32:34.494 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:32:34.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:32:34.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:34.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:32:34.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:32:34.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:32:34.526 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:34.530+0000 7fb507491d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:34.528 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:34.534+0000 7fb507491d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:34.529 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:34.534+0000 7fb507491d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:32:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:32:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:32:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:32:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:32:34.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:34.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:32:34.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:34.892 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:32:34.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:32:35.073 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:35.078+0000 7fb507491d80 -1 Falling back to public interface 2020-11-16T21:32:35.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:32:35.983 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:35.982+0000 7fb507491d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:32:36.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:32:36.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:36.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:32:36.286 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:32:36.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:36.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:32:36.749 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/433150,v1:127.0.0.1:6827/433150] [v2:127.0.0.1:6828/433150,v1:127.0.0.1:6829/433150] exists,up b3f070dc-78f4-40f6-9f3e-df7f070d6048 2020-11-16T21:32:36.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:32:36.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:32:36.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:32:36.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:32:36.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 4 2020-11-16T21:32:36.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:32:36.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:32:36.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:32:36.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:32:36.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/4 2020-11-16T21:32:36.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:32:36.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:32:36.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:32:36.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:32:36.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/4' 2020-11-16T21:32:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/4/journal' 2020-11-16T21:32:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:32:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:32:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:32:36.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:32:36.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:32:36.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:32:36.754 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:32:36.754 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:36.754 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:36.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:36.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:32:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:32:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:32:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:32:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:32:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:32:36.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:32:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:32:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:32:36.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/4 2020-11-16T21:32:36.757 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:32:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4aa4aa21-41bb-4e7d-bebf-3a27c608d0f5 2020-11-16T21:32:36.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 4aa4aa21-41bb-4e7d-bebf-3a27c608d0f5' 2020-11-16T21:32:36.757 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:32:36.757 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 4aa4aa21-41bb-4e7d-bebf-3a27c608d0f5 2020-11-16T21:32:36.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD077Jflu7cLhAAM3R2C0ZZHszrmHU4jVQzpw== 2020-11-16T21:32:36.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD077Jflu7cLhAAM3R2C0ZZHszrmHU4jVQzpw=="}' 2020-11-16T21:32:36.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4aa4aa21-41bb-4e7d-bebf-3a27c608d0f5 -i td/osd-backfill-space/4/new.json 2020-11-16T21:32:37.372 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:32:37.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/4/new.json 2020-11-16T21:32:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD077Jflu7cLhAAM3R2C0ZZHszrmHU4jVQzpw== --osd-uuid 4aa4aa21-41bb-4e7d-bebf-3a27c608d0f5 2020-11-16T21:32:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:37.418+0000 7fdeda813d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:37.422+0000 7fdeda813d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:37.426+0000 7fdeda813d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:37.426+0000 7fdeda813d80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:32:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:37.426+0000 7fdeda813d80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:32:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:37.426+0000 7fdeda813d80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:32:37.520 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:37.526+0000 7fdeda813d80 -1 bluestore(td/osd-backfill-space/4) _read_fsid unparsable uuid 2020-11-16T21:32:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/4/keyring 2020-11-16T21:32:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:32:42.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:32:42.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:32:42.080 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:32:42.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:32:42.577 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:32:42.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:42.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:32:42.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:32:42.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:32:42.607 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:42.614+0000 7f5e36aead80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:42.609 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:42.614+0000 7f5e36aead80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:42.618+0000 7f5e36aead80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:42.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:32:42.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:32:42.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:32:42.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:32:42.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:32:42.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:42.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:32:42.962 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:32:42.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:42.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:32:43.153 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:43.158+0000 7f5e36aead80 -1 Falling back to public interface 2020-11-16T21:32:43.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:32:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:44.054+0000 7f5e36aead80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:32:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:32:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:32:44.335 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:32:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:32:44.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:32:45.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:32:45.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:45.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:32:45.737 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:32:45.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:45.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:32:46.135 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/434144,v1:127.0.0.1:6835/434144] [v2:127.0.0.1:6836/434144,v1:127.0.0.1:6837/434144] exists,up 4aa4aa21-41bb-4e7d-bebf-3a27c608d0f5 2020-11-16T21:32:46.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:32:46.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:32:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:32:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:32:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 5 2020-11-16T21:32:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:32:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:32:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T21:32:46.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:32:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/5 2020-11-16T21:32:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:32:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:32:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:32:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:32:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/5' 2020-11-16T21:32:46.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/5/journal' 2020-11-16T21:32:46.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:32:46.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:32:46.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:32:46.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:32:46.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:32:46.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:32:46.140 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:32:46.140 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:32:46.140 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:32:46.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:46.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:32:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:32:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:32:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:32:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:32:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:32:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:32:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:32:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:32:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:32:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/5 2020-11-16T21:32:46.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:32:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9a309ff9-b728-4194-8f0d-20fde8343417 2020-11-16T21:32:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 9a309ff9-b728-4194-8f0d-20fde8343417' 2020-11-16T21:32:46.143 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 9a309ff9-b728-4194-8f0d-20fde8343417 2020-11-16T21:32:46.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:32:46.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD+77JfYPJdChAARt7B/9/FmefS/kVseYhuuA== 2020-11-16T21:32:46.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD+77JfYPJdChAARt7B/9/FmefS/kVseYhuuA=="}' 2020-11-16T21:32:46.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9a309ff9-b728-4194-8f0d-20fde8343417 -i td/osd-backfill-space/5/new.json 2020-11-16T21:32:46.647 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T21:32:46.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/5/new.json 2020-11-16T21:32:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD+77JfYPJdChAARt7B/9/FmefS/kVseYhuuA== --osd-uuid 9a309ff9-b728-4194-8f0d-20fde8343417 2020-11-16T21:32:46.693 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:46.698+0000 7ff792019d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:46.696 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:46.702+0000 7ff792019d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:46.697 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:46.702+0000 7ff792019d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:46.698 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:46.702+0000 7ff792019d80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:32:46.698 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:46.702+0000 7ff792019d80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:32:46.698 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:46.702+0000 7ff792019d80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:32:46.744 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:46.750+0000 7ff792019d80 -1 bluestore(td/osd-backfill-space/5) _read_fsid unparsable uuid 2020-11-16T21:32:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/5/keyring 2020-11-16T21:32:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:32:51.397 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T21:32:51.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T21:32:51.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:32:51.912 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T21:32:51.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T21:32:51.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:32:51.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:32:51.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:32:51.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:32:51.945 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:51.949+0000 7f12b80ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:51.948 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:51.953+0000 7f12b80ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:51.949 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:51.953+0000 7f12b80ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:32:52.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T21:32:52.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:32:52.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T21:32:52.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:32:52.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:32:52.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:52.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:32:52.315 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:32:52.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:52.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:32:52.492 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:52.497+0000 7f12b80ccd80 -1 Falling back to public interface 2020-11-16T21:32:52.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:32:53.335 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:32:53.341+0000 7f12b80ccd80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T21:32:53.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:32:53.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:32:53.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:32:53.711 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:32:53.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:32:53.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:32:54.119 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/435164,v1:127.0.0.1:6843/435164] [v2:127.0.0.1:6844/435164,v1:127.0.0.1:6845/435164] exists,up 9a309ff9-b728-4194-8f0d-20fde8343417 2020-11-16T21:32:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:32:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:32:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:32:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:769: TEST_ec_backfill_multi: ceph osd set-backfillfull-ratio .85 2020-11-16T21:32:54.808 INFO:tasks.workunit.client.0.smithi189.stderr:osd set-backfillfull-ratio 0.85 2020-11-16T21:32:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:771: TEST_ec_backfill_multi: ceph osd set-require-min-compat-client luminous 2020-11-16T21:32:55.670 INFO:tasks.workunit.client.0.smithi189.stderr:set require_min_compat_client to luminous 2020-11-16T21:32:55.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:772: TEST_ec_backfill_multi: create_pool fillpool 1 1 2020-11-16T21:32:55.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create fillpool 1 1 2020-11-16T21:32:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fillpool' already exists 2020-11-16T21:32:56.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:32:57.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:773: TEST_ec_backfill_multi: ceph osd pool set fillpool size 1 --yes-i-really-mean-it 2020-11-16T21:32:58.344 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 1 2020-11-16T21:32:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:781: TEST_ec_backfill_multi: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=9 2020-11-16T21:32:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:9+0 records in 2020-11-16T21:32:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:9+0 records out 2020-11-16T21:32:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:9216 bytes (9.2 kB, 9.0 KiB) copied, 0.00034412 s, 26.8 MB/s 2020-11-16T21:32:58.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: seq 1 200 2020-11-16T21:32:58.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:58.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj1 td/osd-backfill-space/datafile 2020-11-16T21:32:58.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:58.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj2 td/osd-backfill-space/datafile 2020-11-16T21:32:58.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:58.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj3 td/osd-backfill-space/datafile 2020-11-16T21:32:58.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:58.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj4 td/osd-backfill-space/datafile 2020-11-16T21:32:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:58.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj5 td/osd-backfill-space/datafile 2020-11-16T21:32:59.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:59.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj6 td/osd-backfill-space/datafile 2020-11-16T21:32:59.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:59.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj7 td/osd-backfill-space/datafile 2020-11-16T21:32:59.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:59.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj8 td/osd-backfill-space/datafile 2020-11-16T21:32:59.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:59.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj9 td/osd-backfill-space/datafile 2020-11-16T21:32:59.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:59.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj10 td/osd-backfill-space/datafile 2020-11-16T21:32:59.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:59.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj11 td/osd-backfill-space/datafile 2020-11-16T21:32:59.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:59.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj12 td/osd-backfill-space/datafile 2020-11-16T21:32:59.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:59.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj13 td/osd-backfill-space/datafile 2020-11-16T21:32:59.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:32:59.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj14 td/osd-backfill-space/datafile 2020-11-16T21:33:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:00.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj15 td/osd-backfill-space/datafile 2020-11-16T21:33:00.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:00.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj16 td/osd-backfill-space/datafile 2020-11-16T21:33:00.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:00.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj17 td/osd-backfill-space/datafile 2020-11-16T21:33:00.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:00.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj18 td/osd-backfill-space/datafile 2020-11-16T21:33:00.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:00.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj19 td/osd-backfill-space/datafile 2020-11-16T21:33:00.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:00.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj20 td/osd-backfill-space/datafile 2020-11-16T21:33:00.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:00.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj21 td/osd-backfill-space/datafile 2020-11-16T21:33:00.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:00.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj22 td/osd-backfill-space/datafile 2020-11-16T21:33:00.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:00.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj23 td/osd-backfill-space/datafile 2020-11-16T21:33:00.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:00.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj24 td/osd-backfill-space/datafile 2020-11-16T21:33:00.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj25 td/osd-backfill-space/datafile 2020-11-16T21:33:01.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:01.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj26 td/osd-backfill-space/datafile 2020-11-16T21:33:01.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:01.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj27 td/osd-backfill-space/datafile 2020-11-16T21:33:01.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:01.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj28 td/osd-backfill-space/datafile 2020-11-16T21:33:01.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:01.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj29 td/osd-backfill-space/datafile 2020-11-16T21:33:01.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:01.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj30 td/osd-backfill-space/datafile 2020-11-16T21:33:01.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:01.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj31 td/osd-backfill-space/datafile 2020-11-16T21:33:01.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:01.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj32 td/osd-backfill-space/datafile 2020-11-16T21:33:01.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:01.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj33 td/osd-backfill-space/datafile 2020-11-16T21:33:01.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:01.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj34 td/osd-backfill-space/datafile 2020-11-16T21:33:01.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:01.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj35 td/osd-backfill-space/datafile 2020-11-16T21:33:02.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:02.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj36 td/osd-backfill-space/datafile 2020-11-16T21:33:02.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:02.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj37 td/osd-backfill-space/datafile 2020-11-16T21:33:02.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:02.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj38 td/osd-backfill-space/datafile 2020-11-16T21:33:02.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:02.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj39 td/osd-backfill-space/datafile 2020-11-16T21:33:02.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:02.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj40 td/osd-backfill-space/datafile 2020-11-16T21:33:02.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj41 td/osd-backfill-space/datafile 2020-11-16T21:33:02.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:02.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj42 td/osd-backfill-space/datafile 2020-11-16T21:33:02.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:02.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj43 td/osd-backfill-space/datafile 2020-11-16T21:33:02.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:02.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj44 td/osd-backfill-space/datafile 2020-11-16T21:33:02.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:02.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj45 td/osd-backfill-space/datafile 2020-11-16T21:33:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:02.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj46 td/osd-backfill-space/datafile 2020-11-16T21:33:03.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:03.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj47 td/osd-backfill-space/datafile 2020-11-16T21:33:03.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:03.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj48 td/osd-backfill-space/datafile 2020-11-16T21:33:03.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:03.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj49 td/osd-backfill-space/datafile 2020-11-16T21:33:03.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:03.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj50 td/osd-backfill-space/datafile 2020-11-16T21:33:03.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:03.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj51 td/osd-backfill-space/datafile 2020-11-16T21:33:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:03.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj52 td/osd-backfill-space/datafile 2020-11-16T21:33:03.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:03.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj53 td/osd-backfill-space/datafile 2020-11-16T21:33:03.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:03.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj54 td/osd-backfill-space/datafile 2020-11-16T21:33:03.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:03.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj55 td/osd-backfill-space/datafile 2020-11-16T21:33:03.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:03.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj56 td/osd-backfill-space/datafile 2020-11-16T21:33:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj57 td/osd-backfill-space/datafile 2020-11-16T21:33:04.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:04.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj58 td/osd-backfill-space/datafile 2020-11-16T21:33:04.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:04.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj59 td/osd-backfill-space/datafile 2020-11-16T21:33:04.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:04.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj60 td/osd-backfill-space/datafile 2020-11-16T21:33:04.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:04.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj61 td/osd-backfill-space/datafile 2020-11-16T21:33:04.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:04.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj62 td/osd-backfill-space/datafile 2020-11-16T21:33:04.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:04.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj63 td/osd-backfill-space/datafile 2020-11-16T21:33:04.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:04.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj64 td/osd-backfill-space/datafile 2020-11-16T21:33:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:04.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj65 td/osd-backfill-space/datafile 2020-11-16T21:33:04.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:04.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj66 td/osd-backfill-space/datafile 2020-11-16T21:33:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj67 td/osd-backfill-space/datafile 2020-11-16T21:33:05.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:05.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj68 td/osd-backfill-space/datafile 2020-11-16T21:33:05.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:05.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj69 td/osd-backfill-space/datafile 2020-11-16T21:33:05.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:05.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj70 td/osd-backfill-space/datafile 2020-11-16T21:33:05.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:05.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj71 td/osd-backfill-space/datafile 2020-11-16T21:33:05.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:05.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj72 td/osd-backfill-space/datafile 2020-11-16T21:33:05.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:05.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj73 td/osd-backfill-space/datafile 2020-11-16T21:33:05.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:05.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj74 td/osd-backfill-space/datafile 2020-11-16T21:33:05.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:05.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj75 td/osd-backfill-space/datafile 2020-11-16T21:33:05.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:05.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj76 td/osd-backfill-space/datafile 2020-11-16T21:33:05.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj77 td/osd-backfill-space/datafile 2020-11-16T21:33:06.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:06.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj78 td/osd-backfill-space/datafile 2020-11-16T21:33:06.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:06.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj79 td/osd-backfill-space/datafile 2020-11-16T21:33:06.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:06.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj80 td/osd-backfill-space/datafile 2020-11-16T21:33:06.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:06.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj81 td/osd-backfill-space/datafile 2020-11-16T21:33:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj82 td/osd-backfill-space/datafile 2020-11-16T21:33:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:06.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj83 td/osd-backfill-space/datafile 2020-11-16T21:33:06.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:06.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj84 td/osd-backfill-space/datafile 2020-11-16T21:33:06.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:06.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj85 td/osd-backfill-space/datafile 2020-11-16T21:33:06.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:06.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj86 td/osd-backfill-space/datafile 2020-11-16T21:33:06.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:06.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj87 td/osd-backfill-space/datafile 2020-11-16T21:33:07.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:07.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj88 td/osd-backfill-space/datafile 2020-11-16T21:33:07.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:07.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj89 td/osd-backfill-space/datafile 2020-11-16T21:33:07.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:07.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj90 td/osd-backfill-space/datafile 2020-11-16T21:33:07.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:07.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj91 td/osd-backfill-space/datafile 2020-11-16T21:33:07.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:07.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj92 td/osd-backfill-space/datafile 2020-11-16T21:33:07.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:07.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj93 td/osd-backfill-space/datafile 2020-11-16T21:33:07.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:07.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj94 td/osd-backfill-space/datafile 2020-11-16T21:33:07.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:07.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj95 td/osd-backfill-space/datafile 2020-11-16T21:33:07.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:07.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj96 td/osd-backfill-space/datafile 2020-11-16T21:33:07.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:07.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj97 td/osd-backfill-space/datafile 2020-11-16T21:33:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj98 td/osd-backfill-space/datafile 2020-11-16T21:33:08.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:08.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj99 td/osd-backfill-space/datafile 2020-11-16T21:33:08.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:08.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj100 td/osd-backfill-space/datafile 2020-11-16T21:33:08.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:08.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj101 td/osd-backfill-space/datafile 2020-11-16T21:33:08.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:08.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj102 td/osd-backfill-space/datafile 2020-11-16T21:33:08.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:08.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj103 td/osd-backfill-space/datafile 2020-11-16T21:33:08.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:08.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj104 td/osd-backfill-space/datafile 2020-11-16T21:33:08.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:08.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj105 td/osd-backfill-space/datafile 2020-11-16T21:33:08.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:08.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj106 td/osd-backfill-space/datafile 2020-11-16T21:33:08.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:08.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj107 td/osd-backfill-space/datafile 2020-11-16T21:33:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:08.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj108 td/osd-backfill-space/datafile 2020-11-16T21:33:09.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:09.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj109 td/osd-backfill-space/datafile 2020-11-16T21:33:09.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:09.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj110 td/osd-backfill-space/datafile 2020-11-16T21:33:09.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:09.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj111 td/osd-backfill-space/datafile 2020-11-16T21:33:09.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:09.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj112 td/osd-backfill-space/datafile 2020-11-16T21:33:09.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:09.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj113 td/osd-backfill-space/datafile 2020-11-16T21:33:09.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:09.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj114 td/osd-backfill-space/datafile 2020-11-16T21:33:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:09.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj115 td/osd-backfill-space/datafile 2020-11-16T21:33:09.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:09.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj116 td/osd-backfill-space/datafile 2020-11-16T21:33:09.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:09.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj117 td/osd-backfill-space/datafile 2020-11-16T21:33:10.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:10.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj118 td/osd-backfill-space/datafile 2020-11-16T21:33:10.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:10.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj119 td/osd-backfill-space/datafile 2020-11-16T21:33:10.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:10.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj120 td/osd-backfill-space/datafile 2020-11-16T21:33:10.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:10.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj121 td/osd-backfill-space/datafile 2020-11-16T21:33:10.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:10.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj122 td/osd-backfill-space/datafile 2020-11-16T21:33:10.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj123 td/osd-backfill-space/datafile 2020-11-16T21:33:10.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:10.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj124 td/osd-backfill-space/datafile 2020-11-16T21:33:10.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:10.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj125 td/osd-backfill-space/datafile 2020-11-16T21:33:10.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:10.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj126 td/osd-backfill-space/datafile 2020-11-16T21:33:10.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:10.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj127 td/osd-backfill-space/datafile 2020-11-16T21:33:11.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:11.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj128 td/osd-backfill-space/datafile 2020-11-16T21:33:11.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:11.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj129 td/osd-backfill-space/datafile 2020-11-16T21:33:11.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:11.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj130 td/osd-backfill-space/datafile 2020-11-16T21:33:11.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj131 td/osd-backfill-space/datafile 2020-11-16T21:33:11.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj132 td/osd-backfill-space/datafile 2020-11-16T21:33:11.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:11.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj133 td/osd-backfill-space/datafile 2020-11-16T21:33:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:11.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj134 td/osd-backfill-space/datafile 2020-11-16T21:33:11.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:11.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj135 td/osd-backfill-space/datafile 2020-11-16T21:33:11.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:11.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj136 td/osd-backfill-space/datafile 2020-11-16T21:33:11.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:11.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj137 td/osd-backfill-space/datafile 2020-11-16T21:33:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:12.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj138 td/osd-backfill-space/datafile 2020-11-16T21:33:12.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:12.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj139 td/osd-backfill-space/datafile 2020-11-16T21:33:12.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:12.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj140 td/osd-backfill-space/datafile 2020-11-16T21:33:12.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:12.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj141 td/osd-backfill-space/datafile 2020-11-16T21:33:12.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:12.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj142 td/osd-backfill-space/datafile 2020-11-16T21:33:12.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:12.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj143 td/osd-backfill-space/datafile 2020-11-16T21:33:12.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:12.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj144 td/osd-backfill-space/datafile 2020-11-16T21:33:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj145 td/osd-backfill-space/datafile 2020-11-16T21:33:12.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:12.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj146 td/osd-backfill-space/datafile 2020-11-16T21:33:12.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:12.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj147 td/osd-backfill-space/datafile 2020-11-16T21:33:12.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:12.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj148 td/osd-backfill-space/datafile 2020-11-16T21:33:13.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:13.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj149 td/osd-backfill-space/datafile 2020-11-16T21:33:13.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:13.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj150 td/osd-backfill-space/datafile 2020-11-16T21:33:13.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:13.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj151 td/osd-backfill-space/datafile 2020-11-16T21:33:13.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:13.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj152 td/osd-backfill-space/datafile 2020-11-16T21:33:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj153 td/osd-backfill-space/datafile 2020-11-16T21:33:13.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:13.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj154 td/osd-backfill-space/datafile 2020-11-16T21:33:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj155 td/osd-backfill-space/datafile 2020-11-16T21:33:13.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:13.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj156 td/osd-backfill-space/datafile 2020-11-16T21:33:13.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:13.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj157 td/osd-backfill-space/datafile 2020-11-16T21:33:13.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:13.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj158 td/osd-backfill-space/datafile 2020-11-16T21:33:14.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:14.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj159 td/osd-backfill-space/datafile 2020-11-16T21:33:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj160 td/osd-backfill-space/datafile 2020-11-16T21:33:14.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:14.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj161 td/osd-backfill-space/datafile 2020-11-16T21:33:14.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:14.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj162 td/osd-backfill-space/datafile 2020-11-16T21:33:14.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:14.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj163 td/osd-backfill-space/datafile 2020-11-16T21:33:14.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:14.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj164 td/osd-backfill-space/datafile 2020-11-16T21:33:14.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:14.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj165 td/osd-backfill-space/datafile 2020-11-16T21:33:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:14.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj166 td/osd-backfill-space/datafile 2020-11-16T21:33:14.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:14.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj167 td/osd-backfill-space/datafile 2020-11-16T21:33:14.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:14.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj168 td/osd-backfill-space/datafile 2020-11-16T21:33:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj169 td/osd-backfill-space/datafile 2020-11-16T21:33:15.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:15.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj170 td/osd-backfill-space/datafile 2020-11-16T21:33:15.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:15.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj171 td/osd-backfill-space/datafile 2020-11-16T21:33:15.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:15.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj172 td/osd-backfill-space/datafile 2020-11-16T21:33:15.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:15.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj173 td/osd-backfill-space/datafile 2020-11-16T21:33:15.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:15.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj174 td/osd-backfill-space/datafile 2020-11-16T21:33:15.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:15.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj175 td/osd-backfill-space/datafile 2020-11-16T21:33:15.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:15.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj176 td/osd-backfill-space/datafile 2020-11-16T21:33:15.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:15.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj177 td/osd-backfill-space/datafile 2020-11-16T21:33:16.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:16.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj178 td/osd-backfill-space/datafile 2020-11-16T21:33:16.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:16.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj179 td/osd-backfill-space/datafile 2020-11-16T21:33:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj180 td/osd-backfill-space/datafile 2020-11-16T21:33:16.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:16.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj181 td/osd-backfill-space/datafile 2020-11-16T21:33:16.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:16.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj182 td/osd-backfill-space/datafile 2020-11-16T21:33:16.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:16.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj183 td/osd-backfill-space/datafile 2020-11-16T21:33:16.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:16.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj184 td/osd-backfill-space/datafile 2020-11-16T21:33:16.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:16.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj185 td/osd-backfill-space/datafile 2020-11-16T21:33:16.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:16.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj186 td/osd-backfill-space/datafile 2020-11-16T21:33:16.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:16.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj187 td/osd-backfill-space/datafile 2020-11-16T21:33:16.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:16.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj188 td/osd-backfill-space/datafile 2020-11-16T21:33:17.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:17.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj189 td/osd-backfill-space/datafile 2020-11-16T21:33:17.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:17.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj190 td/osd-backfill-space/datafile 2020-11-16T21:33:17.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:17.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj191 td/osd-backfill-space/datafile 2020-11-16T21:33:17.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:17.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj192 td/osd-backfill-space/datafile 2020-11-16T21:33:17.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:17.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj193 td/osd-backfill-space/datafile 2020-11-16T21:33:17.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:17.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj194 td/osd-backfill-space/datafile 2020-11-16T21:33:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj195 td/osd-backfill-space/datafile 2020-11-16T21:33:17.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:17.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj196 td/osd-backfill-space/datafile 2020-11-16T21:33:17.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:17.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj197 td/osd-backfill-space/datafile 2020-11-16T21:33:17.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:17.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj198 td/osd-backfill-space/datafile 2020-11-16T21:33:18.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:18.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj199 td/osd-backfill-space/datafile 2020-11-16T21:33:18.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:18.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj200 td/osd-backfill-space/datafile 2020-11-16T21:33:18.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:787: TEST_ec_backfill_multi: get_primary fillpool obj1 2020-11-16T21:33:18.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=fillpool 2020-11-16T21:33:18.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:33:18.235 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map fillpool obj1 2020-11-16T21:33:18.235 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:33:18.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:787: TEST_ec_backfill_multi: local fillosd=1 2020-11-16T21:33:18.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:788: TEST_ec_backfill_multi: ceph osd erasure-code-profile set ec-profile k=3 m=2 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure 2020-11-16T21:33:19.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:790: TEST_ec_backfill_multi: osdlist 6 1 2020-11-16T21:33:19.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:723: osdlist: local OSDS=6 2020-11-16T21:33:19.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:724: osdlist: local excludeosd=1 2020-11-16T21:33:19.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:726: osdlist: osds= 2020-11-16T21:33:19.203 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: expr 6 - 1 2020-11-16T21:33:19.204 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: seq 0 5 2020-11-16T21:33:19.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:33:19.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 0 = 1 ']' 2020-11-16T21:33:19.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:733: osdlist: '[' -n '' ']' 2020-11-16T21:33:19.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:736: osdlist: osds=0 2020-11-16T21:33:19.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:33:19.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 1 = 1 ']' 2020-11-16T21:33:19.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:731: osdlist: continue 2020-11-16T21:33:19.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:33:19.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 2 = 1 ']' 2020-11-16T21:33:19.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:733: osdlist: '[' -n 0 ']' 2020-11-16T21:33:19.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:734: osdlist: osds='0 ' 2020-11-16T21:33:19.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:736: osdlist: osds='0 2' 2020-11-16T21:33:19.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:33:19.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 3 = 1 ']' 2020-11-16T21:33:19.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:733: osdlist: '[' -n '0 2' ']' 2020-11-16T21:33:19.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:734: osdlist: osds='0 2 ' 2020-11-16T21:33:19.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:736: osdlist: osds='0 2 3' 2020-11-16T21:33:19.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:33:19.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 4 = 1 ']' 2020-11-16T21:33:19.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:733: osdlist: '[' -n '0 2 3' ']' 2020-11-16T21:33:19.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:734: osdlist: osds='0 2 3 ' 2020-11-16T21:33:19.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:736: osdlist: osds='0 2 3 4' 2020-11-16T21:33:19.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:33:19.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 5 = 1 ']' 2020-11-16T21:33:19.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:733: osdlist: '[' -n '0 2 3 4' ']' 2020-11-16T21:33:19.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:734: osdlist: osds='0 2 3 4 ' 2020-11-16T21:33:19.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:736: osdlist: osds='0 2 3 4 5' 2020-11-16T21:33:19.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:738: osdlist: echo 0 2 3 4 5 2020-11-16T21:33:19.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:790: TEST_ec_backfill_multi: nonfillosds='0 2 3 4 5' 2020-11-16T21:33:19.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:792: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:19.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:792: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:19.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:794: TEST_ec_backfill_multi: ceph osd pool create test1 1 1 erasure ec-profile 2020-11-16T21:33:20.038 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' already exists 2020-11-16T21:33:20.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:795: TEST_ec_backfill_multi: expr 1 + 1 2020-11-16T21:33:20.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:795: TEST_ec_backfill_multi: ceph osd pg-upmap 2.0 0 2 3 4 5 2020-11-16T21:33:20.995 INFO:tasks.workunit.client.0.smithi189.stderr:set 2.0 pg_upmap mapping to [0,2,3,4,5] 2020-11-16T21:33:21.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:792: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:21.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:794: TEST_ec_backfill_multi: ceph osd pool create test2 1 1 erasure ec-profile 2020-11-16T21:33:21.955 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' already exists 2020-11-16T21:33:21.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:795: TEST_ec_backfill_multi: expr 2 + 1 2020-11-16T21:33:21.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:795: TEST_ec_backfill_multi: ceph osd pg-upmap 3.0 0 2 3 4 5 2020-11-16T21:33:23.167 INFO:tasks.workunit.client.0.smithi189.stderr:set 3.0 pg_upmap mapping to [0,2,3,4,5] 2020-11-16T21:33:23.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:800: TEST_ec_backfill_multi: sleep 15 2020-11-16T21:33:38.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:802: TEST_ec_backfill_multi: ceph pg dump pgs 2020-11-16T21:33:38.567 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:33:38.567 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:33:24.429846+0000 0'0 50:14 [0,2,3,4,5] 0 [0,2,3,4,5] 0 0'0 2020-11-16T21:33:21.669627+0000 0'0 2020-11-16T21:33:21.669627+0000 0 2020-11-16T21:33:38.568 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 0 0 0 1843200 0 0 100 100 active+clean 2020-11-16T21:32:58.537300+0000 37'200 50:230 [1] 1 [1] 1 0'0 2020-11-16T21:32:56.107196+0000 0'0 2020-11-16T21:32:56.107196+0000 0 2020-11-16T21:33:38.568 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:33:22.241571+0000 0'0 50:20 [0,2,3,4,5] 0 [0,2,3,4,5] 0 0'0 2020-11-16T21:33:19.773069+0000 0'0 2020-11-16T21:33:19.773069+0000 0 2020-11-16T21:33:38.568 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:33:38.568 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:33:38.569 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:33:38.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:804: TEST_ec_backfill_multi: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=12 2020-11-16T21:33:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:12+0 records in 2020-11-16T21:33:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:12+0 records out 2020-11-16T21:33:38.581 INFO:tasks.workunit.client.0.smithi189.stderr:12288 bytes (12 kB, 12 KiB) copied, 0.000439038 s, 28.0 MB/s 2020-11-16T21:33:38.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: seq 1 200 2020-11-16T21:33:38.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:38.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:38.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:38.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj1-1 td/osd-backfill-space/datafile 2020-11-16T21:33:38.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:38.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj1-2 td/osd-backfill-space/datafile 2020-11-16T21:33:39.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:39.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:39.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:39.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj2-1 td/osd-backfill-space/datafile 2020-11-16T21:33:39.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:39.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj2-2 td/osd-backfill-space/datafile 2020-11-16T21:33:39.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:39.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:39.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:39.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj3-1 td/osd-backfill-space/datafile 2020-11-16T21:33:39.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:39.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj3-2 td/osd-backfill-space/datafile 2020-11-16T21:33:40.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:40.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:40.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:40.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj4-1 td/osd-backfill-space/datafile 2020-11-16T21:33:40.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:40.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj4-2 td/osd-backfill-space/datafile 2020-11-16T21:33:40.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:40.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:40.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:40.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj5-1 td/osd-backfill-space/datafile 2020-11-16T21:33:40.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:40.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj5-2 td/osd-backfill-space/datafile 2020-11-16T21:33:41.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:41.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:41.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj6-1 td/osd-backfill-space/datafile 2020-11-16T21:33:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj6-2 td/osd-backfill-space/datafile 2020-11-16T21:33:41.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:41.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:41.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:41.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj7-1 td/osd-backfill-space/datafile 2020-11-16T21:33:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj7-2 td/osd-backfill-space/datafile 2020-11-16T21:33:42.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:42.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:42.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:42.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj8-1 td/osd-backfill-space/datafile 2020-11-16T21:33:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj8-2 td/osd-backfill-space/datafile 2020-11-16T21:33:42.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:42.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj9-1 td/osd-backfill-space/datafile 2020-11-16T21:33:42.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:42.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj9-2 td/osd-backfill-space/datafile 2020-11-16T21:33:43.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:43.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:43.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:43.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj10-1 td/osd-backfill-space/datafile 2020-11-16T21:33:43.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:43.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj10-2 td/osd-backfill-space/datafile 2020-11-16T21:33:43.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:43.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:43.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:43.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj11-1 td/osd-backfill-space/datafile 2020-11-16T21:33:43.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:43.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj11-2 td/osd-backfill-space/datafile 2020-11-16T21:33:44.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:44.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:44.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:44.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj12-1 td/osd-backfill-space/datafile 2020-11-16T21:33:44.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:44.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj12-2 td/osd-backfill-space/datafile 2020-11-16T21:33:44.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:44.707 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:44.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:44.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj13-1 td/osd-backfill-space/datafile 2020-11-16T21:33:44.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:44.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj13-2 td/osd-backfill-space/datafile 2020-11-16T21:33:45.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:45.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:45.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:45.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj14-1 td/osd-backfill-space/datafile 2020-11-16T21:33:45.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:45.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj14-2 td/osd-backfill-space/datafile 2020-11-16T21:33:45.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:45.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:45.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:45.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj15-1 td/osd-backfill-space/datafile 2020-11-16T21:33:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj15-2 td/osd-backfill-space/datafile 2020-11-16T21:33:46.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:46.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:46.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:46.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj16-1 td/osd-backfill-space/datafile 2020-11-16T21:33:46.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:46.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj16-2 td/osd-backfill-space/datafile 2020-11-16T21:33:46.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:46.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:46.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:46.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj17-1 td/osd-backfill-space/datafile 2020-11-16T21:33:47.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:47.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj17-2 td/osd-backfill-space/datafile 2020-11-16T21:33:47.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:47.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:47.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:47.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj18-1 td/osd-backfill-space/datafile 2020-11-16T21:33:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj18-2 td/osd-backfill-space/datafile 2020-11-16T21:33:47.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:47.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:47.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:47.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj19-1 td/osd-backfill-space/datafile 2020-11-16T21:33:48.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:48.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj19-2 td/osd-backfill-space/datafile 2020-11-16T21:33:48.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:48.290 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:48.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:48.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj20-1 td/osd-backfill-space/datafile 2020-11-16T21:33:48.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:48.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj20-2 td/osd-backfill-space/datafile 2020-11-16T21:33:48.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:48.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:48.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:48.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj21-1 td/osd-backfill-space/datafile 2020-11-16T21:33:49.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:49.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj21-2 td/osd-backfill-space/datafile 2020-11-16T21:33:49.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:49.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:49.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:49.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj22-1 td/osd-backfill-space/datafile 2020-11-16T21:33:49.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:49.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj22-2 td/osd-backfill-space/datafile 2020-11-16T21:33:49.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:49.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:49.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:49.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj23-1 td/osd-backfill-space/datafile 2020-11-16T21:33:50.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:50.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj23-2 td/osd-backfill-space/datafile 2020-11-16T21:33:50.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:50.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:50.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:50.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj24-1 td/osd-backfill-space/datafile 2020-11-16T21:33:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj24-2 td/osd-backfill-space/datafile 2020-11-16T21:33:50.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:50.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:50.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:50.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj25-1 td/osd-backfill-space/datafile 2020-11-16T21:33:51.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:51.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj25-2 td/osd-backfill-space/datafile 2020-11-16T21:33:51.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:51.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:51.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:51.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj26-1 td/osd-backfill-space/datafile 2020-11-16T21:33:51.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:51.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj26-2 td/osd-backfill-space/datafile 2020-11-16T21:33:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:51.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj27-1 td/osd-backfill-space/datafile 2020-11-16T21:33:52.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:52.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj27-2 td/osd-backfill-space/datafile 2020-11-16T21:33:52.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:52.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:52.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:52.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj28-1 td/osd-backfill-space/datafile 2020-11-16T21:33:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj28-2 td/osd-backfill-space/datafile 2020-11-16T21:33:52.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:52.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:52.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:52.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj29-1 td/osd-backfill-space/datafile 2020-11-16T21:33:53.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:53.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj29-2 td/osd-backfill-space/datafile 2020-11-16T21:33:53.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:53.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:53.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj30-1 td/osd-backfill-space/datafile 2020-11-16T21:33:53.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:53.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj30-2 td/osd-backfill-space/datafile 2020-11-16T21:33:53.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:53.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:53.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:53.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj31-1 td/osd-backfill-space/datafile 2020-11-16T21:33:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj31-2 td/osd-backfill-space/datafile 2020-11-16T21:33:54.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:54.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:54.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:54.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj32-1 td/osd-backfill-space/datafile 2020-11-16T21:33:54.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:54.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj32-2 td/osd-backfill-space/datafile 2020-11-16T21:33:54.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:54.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:54.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:54.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj33-1 td/osd-backfill-space/datafile 2020-11-16T21:33:55.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:55.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj33-2 td/osd-backfill-space/datafile 2020-11-16T21:33:55.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:55.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:55.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:55.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj34-1 td/osd-backfill-space/datafile 2020-11-16T21:33:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:55.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj34-2 td/osd-backfill-space/datafile 2020-11-16T21:33:55.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:55.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:55.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:55.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj35-1 td/osd-backfill-space/datafile 2020-11-16T21:33:56.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:56.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj35-2 td/osd-backfill-space/datafile 2020-11-16T21:33:56.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:56.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:56.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:56.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj36-1 td/osd-backfill-space/datafile 2020-11-16T21:33:56.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:56.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj36-2 td/osd-backfill-space/datafile 2020-11-16T21:33:56.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:56.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:56.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:56.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj37-1 td/osd-backfill-space/datafile 2020-11-16T21:33:57.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:57.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj37-2 td/osd-backfill-space/datafile 2020-11-16T21:33:57.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:57.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:57.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:57.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj38-1 td/osd-backfill-space/datafile 2020-11-16T21:33:57.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:57.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj38-2 td/osd-backfill-space/datafile 2020-11-16T21:33:57.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:57.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:57.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:57.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj39-1 td/osd-backfill-space/datafile 2020-11-16T21:33:58.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:58.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj39-2 td/osd-backfill-space/datafile 2020-11-16T21:33:58.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:58.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:58.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:58.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj40-1 td/osd-backfill-space/datafile 2020-11-16T21:33:58.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:58.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj40-2 td/osd-backfill-space/datafile 2020-11-16T21:33:58.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:58.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:58.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:58.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj41-1 td/osd-backfill-space/datafile 2020-11-16T21:33:59.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:59.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj41-2 td/osd-backfill-space/datafile 2020-11-16T21:33:59.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:59.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:59.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:59.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj42-1 td/osd-backfill-space/datafile 2020-11-16T21:33:59.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:59.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj42-2 td/osd-backfill-space/datafile 2020-11-16T21:33:59.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:33:59.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:33:59.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:33:59.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj43-1 td/osd-backfill-space/datafile 2020-11-16T21:34:00.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:00.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj43-2 td/osd-backfill-space/datafile 2020-11-16T21:34:00.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:00.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:00.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:00.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj44-1 td/osd-backfill-space/datafile 2020-11-16T21:34:00.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:00.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj44-2 td/osd-backfill-space/datafile 2020-11-16T21:34:00.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:00.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:00.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:00.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj45-1 td/osd-backfill-space/datafile 2020-11-16T21:34:01.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:01.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj45-2 td/osd-backfill-space/datafile 2020-11-16T21:34:01.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:01.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:01.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:01.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj46-1 td/osd-backfill-space/datafile 2020-11-16T21:34:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:01.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj46-2 td/osd-backfill-space/datafile 2020-11-16T21:34:01.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:01.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:01.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:01.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj47-1 td/osd-backfill-space/datafile 2020-11-16T21:34:02.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:02.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj47-2 td/osd-backfill-space/datafile 2020-11-16T21:34:02.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:02.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:02.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:02.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj48-1 td/osd-backfill-space/datafile 2020-11-16T21:34:02.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:02.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj48-2 td/osd-backfill-space/datafile 2020-11-16T21:34:02.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:02.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:02.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:02.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj49-1 td/osd-backfill-space/datafile 2020-11-16T21:34:03.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:03.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj49-2 td/osd-backfill-space/datafile 2020-11-16T21:34:03.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:03.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:03.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:03.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj50-1 td/osd-backfill-space/datafile 2020-11-16T21:34:03.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:03.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj50-2 td/osd-backfill-space/datafile 2020-11-16T21:34:03.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:03.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:03.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:03.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj51-1 td/osd-backfill-space/datafile 2020-11-16T21:34:04.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:04.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj51-2 td/osd-backfill-space/datafile 2020-11-16T21:34:04.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:04.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:04.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:04.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj52-1 td/osd-backfill-space/datafile 2020-11-16T21:34:04.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:04.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj52-2 td/osd-backfill-space/datafile 2020-11-16T21:34:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:04.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj53-1 td/osd-backfill-space/datafile 2020-11-16T21:34:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj53-2 td/osd-backfill-space/datafile 2020-11-16T21:34:05.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:05.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:05.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:05.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj54-1 td/osd-backfill-space/datafile 2020-11-16T21:34:05.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:05.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj54-2 td/osd-backfill-space/datafile 2020-11-16T21:34:05.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:05.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:05.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:05.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj55-1 td/osd-backfill-space/datafile 2020-11-16T21:34:06.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:06.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj55-2 td/osd-backfill-space/datafile 2020-11-16T21:34:06.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:06.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:06.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:06.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj56-1 td/osd-backfill-space/datafile 2020-11-16T21:34:06.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:06.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj56-2 td/osd-backfill-space/datafile 2020-11-16T21:34:06.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:06.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:06.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:06.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj57-1 td/osd-backfill-space/datafile 2020-11-16T21:34:06.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj57-2 td/osd-backfill-space/datafile 2020-11-16T21:34:07.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:07.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:07.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:07.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj58-1 td/osd-backfill-space/datafile 2020-11-16T21:34:07.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:07.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj58-2 td/osd-backfill-space/datafile 2020-11-16T21:34:07.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:07.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:07.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:07.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj59-1 td/osd-backfill-space/datafile 2020-11-16T21:34:08.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:08.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj59-2 td/osd-backfill-space/datafile 2020-11-16T21:34:08.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:08.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:08.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:08.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj60-1 td/osd-backfill-space/datafile 2020-11-16T21:34:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj60-2 td/osd-backfill-space/datafile 2020-11-16T21:34:08.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:08.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:08.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:08.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj61-1 td/osd-backfill-space/datafile 2020-11-16T21:34:09.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:09.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj61-2 td/osd-backfill-space/datafile 2020-11-16T21:34:09.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:09.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:09.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:09.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj62-1 td/osd-backfill-space/datafile 2020-11-16T21:34:09.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:09.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj62-2 td/osd-backfill-space/datafile 2020-11-16T21:34:09.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:09.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:09.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj63-1 td/osd-backfill-space/datafile 2020-11-16T21:34:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj63-2 td/osd-backfill-space/datafile 2020-11-16T21:34:10.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:10.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:10.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:10.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj64-1 td/osd-backfill-space/datafile 2020-11-16T21:34:10.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:10.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj64-2 td/osd-backfill-space/datafile 2020-11-16T21:34:10.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:10.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:10.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:10.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj65-1 td/osd-backfill-space/datafile 2020-11-16T21:34:11.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:11.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj65-2 td/osd-backfill-space/datafile 2020-11-16T21:34:11.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:11.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:11.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:11.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj66-1 td/osd-backfill-space/datafile 2020-11-16T21:34:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj66-2 td/osd-backfill-space/datafile 2020-11-16T21:34:12.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:12.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:12.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:12.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj67-1 td/osd-backfill-space/datafile 2020-11-16T21:34:12.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:12.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj67-2 td/osd-backfill-space/datafile 2020-11-16T21:34:12.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:12.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:12.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:12.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj68-1 td/osd-backfill-space/datafile 2020-11-16T21:34:12.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:12.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj68-2 td/osd-backfill-space/datafile 2020-11-16T21:34:12.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:12.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:12.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:13.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj69-1 td/osd-backfill-space/datafile 2020-11-16T21:34:13.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:13.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj69-2 td/osd-backfill-space/datafile 2020-11-16T21:34:13.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:13.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:13.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:13.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj70-1 td/osd-backfill-space/datafile 2020-11-16T21:34:13.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:13.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj70-2 td/osd-backfill-space/datafile 2020-11-16T21:34:13.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:13.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:13.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:13.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj71-1 td/osd-backfill-space/datafile 2020-11-16T21:34:14.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:14.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj71-2 td/osd-backfill-space/datafile 2020-11-16T21:34:14.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:14.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:14.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:14.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj72-1 td/osd-backfill-space/datafile 2020-11-16T21:34:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:14.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj72-2 td/osd-backfill-space/datafile 2020-11-16T21:34:14.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:14.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:14.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:14.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj73-1 td/osd-backfill-space/datafile 2020-11-16T21:34:15.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:15.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj73-2 td/osd-backfill-space/datafile 2020-11-16T21:34:15.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:15.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:15.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:15.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj74-1 td/osd-backfill-space/datafile 2020-11-16T21:34:15.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:15.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj74-2 td/osd-backfill-space/datafile 2020-11-16T21:34:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:16.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj75-1 td/osd-backfill-space/datafile 2020-11-16T21:34:16.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:16.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj75-2 td/osd-backfill-space/datafile 2020-11-16T21:34:16.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:16.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:16.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:16.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj76-1 td/osd-backfill-space/datafile 2020-11-16T21:34:16.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:16.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj76-2 td/osd-backfill-space/datafile 2020-11-16T21:34:17.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:17.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:17.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:17.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj77-1 td/osd-backfill-space/datafile 2020-11-16T21:34:17.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:17.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj77-2 td/osd-backfill-space/datafile 2020-11-16T21:34:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:17.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj78-1 td/osd-backfill-space/datafile 2020-11-16T21:34:17.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:17.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj78-2 td/osd-backfill-space/datafile 2020-11-16T21:34:18.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:18.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:18.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:18.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj79-1 td/osd-backfill-space/datafile 2020-11-16T21:34:18.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:18.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj79-2 td/osd-backfill-space/datafile 2020-11-16T21:34:18.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:18.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:18.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj80-1 td/osd-backfill-space/datafile 2020-11-16T21:34:18.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:18.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj80-2 td/osd-backfill-space/datafile 2020-11-16T21:34:19.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:19.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:19.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:19.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj81-1 td/osd-backfill-space/datafile 2020-11-16T21:34:19.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:19.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj81-2 td/osd-backfill-space/datafile 2020-11-16T21:34:19.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:19.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:19.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:19.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj82-1 td/osd-backfill-space/datafile 2020-11-16T21:34:19.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:19.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj82-2 td/osd-backfill-space/datafile 2020-11-16T21:34:20.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:20.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:20.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:20.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj83-1 td/osd-backfill-space/datafile 2020-11-16T21:34:20.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:20.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj83-2 td/osd-backfill-space/datafile 2020-11-16T21:34:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:20.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj84-1 td/osd-backfill-space/datafile 2020-11-16T21:34:21.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:21.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj84-2 td/osd-backfill-space/datafile 2020-11-16T21:34:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:21.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj85-1 td/osd-backfill-space/datafile 2020-11-16T21:34:21.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:21.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj85-2 td/osd-backfill-space/datafile 2020-11-16T21:34:21.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:21.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:21.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:21.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj86-1 td/osd-backfill-space/datafile 2020-11-16T21:34:22.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:22.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj86-2 td/osd-backfill-space/datafile 2020-11-16T21:34:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:22.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj87-1 td/osd-backfill-space/datafile 2020-11-16T21:34:22.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:22.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj87-2 td/osd-backfill-space/datafile 2020-11-16T21:34:22.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:22.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:22.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:22.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj88-1 td/osd-backfill-space/datafile 2020-11-16T21:34:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj88-2 td/osd-backfill-space/datafile 2020-11-16T21:34:23.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:23.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:23.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:23.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj89-1 td/osd-backfill-space/datafile 2020-11-16T21:34:23.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:23.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj89-2 td/osd-backfill-space/datafile 2020-11-16T21:34:23.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:23.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:23.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:23.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj90-1 td/osd-backfill-space/datafile 2020-11-16T21:34:24.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:24.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj90-2 td/osd-backfill-space/datafile 2020-11-16T21:34:24.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:24.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:24.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:24.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj91-1 td/osd-backfill-space/datafile 2020-11-16T21:34:24.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:24.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj91-2 td/osd-backfill-space/datafile 2020-11-16T21:34:24.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:24.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:24.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:24.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj92-1 td/osd-backfill-space/datafile 2020-11-16T21:34:25.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:25.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj92-2 td/osd-backfill-space/datafile 2020-11-16T21:34:25.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:25.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:25.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:25.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj93-1 td/osd-backfill-space/datafile 2020-11-16T21:34:25.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:25.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj93-2 td/osd-backfill-space/datafile 2020-11-16T21:34:25.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:25.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:25.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:25.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj94-1 td/osd-backfill-space/datafile 2020-11-16T21:34:26.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:26.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj94-2 td/osd-backfill-space/datafile 2020-11-16T21:34:26.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:26.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:26.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:26.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj95-1 td/osd-backfill-space/datafile 2020-11-16T21:34:26.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:26.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj95-2 td/osd-backfill-space/datafile 2020-11-16T21:34:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:27.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:27.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:27.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj96-1 td/osd-backfill-space/datafile 2020-11-16T21:34:27.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:27.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj96-2 td/osd-backfill-space/datafile 2020-11-16T21:34:27.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:27.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:27.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:27.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj97-1 td/osd-backfill-space/datafile 2020-11-16T21:34:27.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj97-2 td/osd-backfill-space/datafile 2020-11-16T21:34:28.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:28.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:28.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:28.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj98-1 td/osd-backfill-space/datafile 2020-11-16T21:34:28.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:28.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj98-2 td/osd-backfill-space/datafile 2020-11-16T21:34:28.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:28.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:28.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:28.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj99-1 td/osd-backfill-space/datafile 2020-11-16T21:34:28.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:28.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj99-2 td/osd-backfill-space/datafile 2020-11-16T21:34:28.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:28.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:28.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:28.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj100-1 td/osd-backfill-space/datafile 2020-11-16T21:34:29.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:29.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj100-2 td/osd-backfill-space/datafile 2020-11-16T21:34:29.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:29.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:29.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:29.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj101-1 td/osd-backfill-space/datafile 2020-11-16T21:34:29.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:29.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj101-2 td/osd-backfill-space/datafile 2020-11-16T21:34:30.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:30.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:30.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:30.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj102-1 td/osd-backfill-space/datafile 2020-11-16T21:34:30.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:30.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj102-2 td/osd-backfill-space/datafile 2020-11-16T21:34:30.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:30.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:30.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:30.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj103-1 td/osd-backfill-space/datafile 2020-11-16T21:34:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:30.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj103-2 td/osd-backfill-space/datafile 2020-11-16T21:34:31.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:31.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:31.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:31.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj104-1 td/osd-backfill-space/datafile 2020-11-16T21:34:31.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:31.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj104-2 td/osd-backfill-space/datafile 2020-11-16T21:34:31.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:31.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:31.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:31.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj105-1 td/osd-backfill-space/datafile 2020-11-16T21:34:31.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:31.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj105-2 td/osd-backfill-space/datafile 2020-11-16T21:34:32.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:32.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:32.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:32.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj106-1 td/osd-backfill-space/datafile 2020-11-16T21:34:32.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:32.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj106-2 td/osd-backfill-space/datafile 2020-11-16T21:34:32.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:32.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:32.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:32.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj107-1 td/osd-backfill-space/datafile 2020-11-16T21:34:32.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:32.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj107-2 td/osd-backfill-space/datafile 2020-11-16T21:34:33.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:33.230 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:33.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:33.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj108-1 td/osd-backfill-space/datafile 2020-11-16T21:34:33.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:33.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj108-2 td/osd-backfill-space/datafile 2020-11-16T21:34:33.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:33.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:33.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:33.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj109-1 td/osd-backfill-space/datafile 2020-11-16T21:34:34.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj109-2 td/osd-backfill-space/datafile 2020-11-16T21:34:34.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:34.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:34.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:34.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj110-1 td/osd-backfill-space/datafile 2020-11-16T21:34:34.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:34.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj110-2 td/osd-backfill-space/datafile 2020-11-16T21:34:34.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:34.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:34.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:34.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj111-1 td/osd-backfill-space/datafile 2020-11-16T21:34:35.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:35.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj111-2 td/osd-backfill-space/datafile 2020-11-16T21:34:35.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:35.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:35.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:35.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj112-1 td/osd-backfill-space/datafile 2020-11-16T21:34:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj112-2 td/osd-backfill-space/datafile 2020-11-16T21:34:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:35.940 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj113-1 td/osd-backfill-space/datafile 2020-11-16T21:34:36.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:36.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj113-2 td/osd-backfill-space/datafile 2020-11-16T21:34:36.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:36.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:36.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj114-1 td/osd-backfill-space/datafile 2020-11-16T21:34:36.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:36.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj114-2 td/osd-backfill-space/datafile 2020-11-16T21:34:36.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:36.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:36.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:36.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj115-1 td/osd-backfill-space/datafile 2020-11-16T21:34:37.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:37.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj115-2 td/osd-backfill-space/datafile 2020-11-16T21:34:37.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:37.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:37.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:37.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj116-1 td/osd-backfill-space/datafile 2020-11-16T21:34:37.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:37.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj116-2 td/osd-backfill-space/datafile 2020-11-16T21:34:38.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:38.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:38.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:38.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj117-1 td/osd-backfill-space/datafile 2020-11-16T21:34:38.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:38.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj117-2 td/osd-backfill-space/datafile 2020-11-16T21:34:38.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:38.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:38.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:38.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj118-1 td/osd-backfill-space/datafile 2020-11-16T21:34:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj118-2 td/osd-backfill-space/datafile 2020-11-16T21:34:39.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:39.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj119-1 td/osd-backfill-space/datafile 2020-11-16T21:34:39.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:39.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj119-2 td/osd-backfill-space/datafile 2020-11-16T21:34:39.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:39.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:39.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:39.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj120-1 td/osd-backfill-space/datafile 2020-11-16T21:34:40.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:40.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj120-2 td/osd-backfill-space/datafile 2020-11-16T21:34:40.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:40.307 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:40.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:40.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj121-1 td/osd-backfill-space/datafile 2020-11-16T21:34:40.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:40.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj121-2 td/osd-backfill-space/datafile 2020-11-16T21:34:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:40.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:40.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:40.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj122-1 td/osd-backfill-space/datafile 2020-11-16T21:34:41.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:41.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj122-2 td/osd-backfill-space/datafile 2020-11-16T21:34:41.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:41.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:41.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:41.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj123-1 td/osd-backfill-space/datafile 2020-11-16T21:34:41.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:41.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj123-2 td/osd-backfill-space/datafile 2020-11-16T21:34:42.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:42.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:42.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:42.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj124-1 td/osd-backfill-space/datafile 2020-11-16T21:34:42.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:42.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj124-2 td/osd-backfill-space/datafile 2020-11-16T21:34:42.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:42.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:42.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:42.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj125-1 td/osd-backfill-space/datafile 2020-11-16T21:34:43.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:43.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj125-2 td/osd-backfill-space/datafile 2020-11-16T21:34:43.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:43.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:43.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:43.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj126-1 td/osd-backfill-space/datafile 2020-11-16T21:34:43.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:43.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj126-2 td/osd-backfill-space/datafile 2020-11-16T21:34:43.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:43.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:43.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj127-1 td/osd-backfill-space/datafile 2020-11-16T21:34:44.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:44.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj127-2 td/osd-backfill-space/datafile 2020-11-16T21:34:44.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:44.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:44.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:44.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj128-1 td/osd-backfill-space/datafile 2020-11-16T21:34:44.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:44.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj128-2 td/osd-backfill-space/datafile 2020-11-16T21:34:44.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:44.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:44.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:44.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj129-1 td/osd-backfill-space/datafile 2020-11-16T21:34:45.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:45.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj129-2 td/osd-backfill-space/datafile 2020-11-16T21:34:45.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:45.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj130-1 td/osd-backfill-space/datafile 2020-11-16T21:34:45.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:45.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj130-2 td/osd-backfill-space/datafile 2020-11-16T21:34:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:46.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:46.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj131-1 td/osd-backfill-space/datafile 2020-11-16T21:34:46.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:46.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj131-2 td/osd-backfill-space/datafile 2020-11-16T21:34:46.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:46.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:46.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:46.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj132-1 td/osd-backfill-space/datafile 2020-11-16T21:34:46.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:46.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj132-2 td/osd-backfill-space/datafile 2020-11-16T21:34:47.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:47.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:47.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:47.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj133-1 td/osd-backfill-space/datafile 2020-11-16T21:34:47.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:47.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj133-2 td/osd-backfill-space/datafile 2020-11-16T21:34:47.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:47.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:47.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj134-1 td/osd-backfill-space/datafile 2020-11-16T21:34:48.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:48.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj134-2 td/osd-backfill-space/datafile 2020-11-16T21:34:48.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:48.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:48.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:48.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj135-1 td/osd-backfill-space/datafile 2020-11-16T21:34:48.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:48.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj135-2 td/osd-backfill-space/datafile 2020-11-16T21:34:48.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:48.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:48.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:48.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj136-1 td/osd-backfill-space/datafile 2020-11-16T21:34:49.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:49.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj136-2 td/osd-backfill-space/datafile 2020-11-16T21:34:49.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:49.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:49.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:49.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj137-1 td/osd-backfill-space/datafile 2020-11-16T21:34:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj137-2 td/osd-backfill-space/datafile 2020-11-16T21:34:50.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:50.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:50.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:50.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj138-1 td/osd-backfill-space/datafile 2020-11-16T21:34:50.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:50.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj138-2 td/osd-backfill-space/datafile 2020-11-16T21:34:50.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:50.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:50.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:50.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj139-1 td/osd-backfill-space/datafile 2020-11-16T21:34:50.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:50.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj139-2 td/osd-backfill-space/datafile 2020-11-16T21:34:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:51.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj140-1 td/osd-backfill-space/datafile 2020-11-16T21:34:51.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:51.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj140-2 td/osd-backfill-space/datafile 2020-11-16T21:34:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:51.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj141-1 td/osd-backfill-space/datafile 2020-11-16T21:34:52.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:52.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj141-2 td/osd-backfill-space/datafile 2020-11-16T21:34:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:52.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj142-1 td/osd-backfill-space/datafile 2020-11-16T21:34:52.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:52.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj142-2 td/osd-backfill-space/datafile 2020-11-16T21:34:52.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:52.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:52.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:52.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj143-1 td/osd-backfill-space/datafile 2020-11-16T21:34:53.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:53.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj143-2 td/osd-backfill-space/datafile 2020-11-16T21:34:53.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:53.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:53.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:53.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj144-1 td/osd-backfill-space/datafile 2020-11-16T21:34:53.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:53.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj144-2 td/osd-backfill-space/datafile 2020-11-16T21:34:54.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:54.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:54.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:54.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj145-1 td/osd-backfill-space/datafile 2020-11-16T21:34:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj145-2 td/osd-backfill-space/datafile 2020-11-16T21:34:54.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:54.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj146-1 td/osd-backfill-space/datafile 2020-11-16T21:34:55.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:55.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj146-2 td/osd-backfill-space/datafile 2020-11-16T21:34:55.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:55.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:55.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj147-1 td/osd-backfill-space/datafile 2020-11-16T21:34:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj147-2 td/osd-backfill-space/datafile 2020-11-16T21:34:55.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:55.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:55.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:55.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj148-1 td/osd-backfill-space/datafile 2020-11-16T21:34:56.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:56.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj148-2 td/osd-backfill-space/datafile 2020-11-16T21:34:56.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:56.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:56.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:56.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj149-1 td/osd-backfill-space/datafile 2020-11-16T21:34:56.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:56.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj149-2 td/osd-backfill-space/datafile 2020-11-16T21:34:56.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:56.972 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:56.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:56.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj150-1 td/osd-backfill-space/datafile 2020-11-16T21:34:57.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:57.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj150-2 td/osd-backfill-space/datafile 2020-11-16T21:34:57.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:57.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:57.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:57.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj151-1 td/osd-backfill-space/datafile 2020-11-16T21:34:57.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:57.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj151-2 td/osd-backfill-space/datafile 2020-11-16T21:34:58.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:58.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:58.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj152-1 td/osd-backfill-space/datafile 2020-11-16T21:34:58.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:58.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj152-2 td/osd-backfill-space/datafile 2020-11-16T21:34:58.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:58.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:58.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:58.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj153-1 td/osd-backfill-space/datafile 2020-11-16T21:34:58.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj153-2 td/osd-backfill-space/datafile 2020-11-16T21:34:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:59.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj154-1 td/osd-backfill-space/datafile 2020-11-16T21:34:59.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:59.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj154-2 td/osd-backfill-space/datafile 2020-11-16T21:34:59.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:34:59.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:34:59.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:34:59.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj155-1 td/osd-backfill-space/datafile 2020-11-16T21:35:00.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:00.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj155-2 td/osd-backfill-space/datafile 2020-11-16T21:35:00.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:00.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:00.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:00.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj156-1 td/osd-backfill-space/datafile 2020-11-16T21:35:00.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:00.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj156-2 td/osd-backfill-space/datafile 2020-11-16T21:35:00.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:00.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:00.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:00.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj157-1 td/osd-backfill-space/datafile 2020-11-16T21:35:01.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:01.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj157-2 td/osd-backfill-space/datafile 2020-11-16T21:35:01.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:01.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:01.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:01.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj158-1 td/osd-backfill-space/datafile 2020-11-16T21:35:01.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:01.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj158-2 td/osd-backfill-space/datafile 2020-11-16T21:35:02.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:02.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:02.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:02.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj159-1 td/osd-backfill-space/datafile 2020-11-16T21:35:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:02.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj159-2 td/osd-backfill-space/datafile 2020-11-16T21:35:02.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:02.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:02.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:02.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj160-1 td/osd-backfill-space/datafile 2020-11-16T21:35:02.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:02.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj160-2 td/osd-backfill-space/datafile 2020-11-16T21:35:03.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:03.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:03.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:03.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj161-1 td/osd-backfill-space/datafile 2020-11-16T21:35:03.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:03.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj161-2 td/osd-backfill-space/datafile 2020-11-16T21:35:03.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:03.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:03.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:03.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj162-1 td/osd-backfill-space/datafile 2020-11-16T21:35:04.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:04.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj162-2 td/osd-backfill-space/datafile 2020-11-16T21:35:04.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:04.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:04.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:04.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj163-1 td/osd-backfill-space/datafile 2020-11-16T21:35:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj163-2 td/osd-backfill-space/datafile 2020-11-16T21:35:04.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:04.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:04.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:04.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj164-1 td/osd-backfill-space/datafile 2020-11-16T21:35:05.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:05.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj164-2 td/osd-backfill-space/datafile 2020-11-16T21:35:05.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:09.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:09.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj165-1 td/osd-backfill-space/datafile 2020-11-16T21:35:09.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj165-2 td/osd-backfill-space/datafile 2020-11-16T21:35:09.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:09.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:09.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj166-1 td/osd-backfill-space/datafile 2020-11-16T21:35:09.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj166-2 td/osd-backfill-space/datafile 2020-11-16T21:35:09.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:09.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:09.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj167-1 td/osd-backfill-space/datafile 2020-11-16T21:35:09.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj167-2 td/osd-backfill-space/datafile 2020-11-16T21:35:09.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:09.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:09.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj168-1 td/osd-backfill-space/datafile 2020-11-16T21:35:09.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj168-2 td/osd-backfill-space/datafile 2020-11-16T21:35:09.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:09.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:09.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj169-1 td/osd-backfill-space/datafile 2020-11-16T21:35:09.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj169-2 td/osd-backfill-space/datafile 2020-11-16T21:35:09.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:09.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:09.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj170-1 td/osd-backfill-space/datafile 2020-11-16T21:35:09.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj170-2 td/osd-backfill-space/datafile 2020-11-16T21:35:09.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:09.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:09.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj171-1 td/osd-backfill-space/datafile 2020-11-16T21:35:09.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj171-2 td/osd-backfill-space/datafile 2020-11-16T21:35:09.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:09.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:09.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj172-1 td/osd-backfill-space/datafile 2020-11-16T21:35:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj172-2 td/osd-backfill-space/datafile 2020-11-16T21:35:09.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:09.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:09.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:09.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj173-1 td/osd-backfill-space/datafile 2020-11-16T21:35:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj173-2 td/osd-backfill-space/datafile 2020-11-16T21:35:10.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:10.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:10.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:10.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj174-1 td/osd-backfill-space/datafile 2020-11-16T21:35:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj174-2 td/osd-backfill-space/datafile 2020-11-16T21:35:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:10.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj175-1 td/osd-backfill-space/datafile 2020-11-16T21:35:11.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:11.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj175-2 td/osd-backfill-space/datafile 2020-11-16T21:35:11.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:11.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:11.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:11.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj176-1 td/osd-backfill-space/datafile 2020-11-16T21:35:11.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:11.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj176-2 td/osd-backfill-space/datafile 2020-11-16T21:35:11.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:11.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:11.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:11.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj177-1 td/osd-backfill-space/datafile 2020-11-16T21:35:12.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:12.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj177-2 td/osd-backfill-space/datafile 2020-11-16T21:35:12.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:12.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:12.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:12.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj178-1 td/osd-backfill-space/datafile 2020-11-16T21:35:12.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:12.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj178-2 td/osd-backfill-space/datafile 2020-11-16T21:35:13.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:13.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:13.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:13.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj179-1 td/osd-backfill-space/datafile 2020-11-16T21:35:13.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:13.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj179-2 td/osd-backfill-space/datafile 2020-11-16T21:35:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:13.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:13.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:13.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj180-1 td/osd-backfill-space/datafile 2020-11-16T21:35:13.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:13.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj180-2 td/osd-backfill-space/datafile 2020-11-16T21:35:14.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:14.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:14.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:14.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj181-1 td/osd-backfill-space/datafile 2020-11-16T21:35:14.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:14.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj181-2 td/osd-backfill-space/datafile 2020-11-16T21:35:14.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:14.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:14.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:14.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj182-1 td/osd-backfill-space/datafile 2020-11-16T21:35:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj182-2 td/osd-backfill-space/datafile 2020-11-16T21:35:15.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:15.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:15.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:15.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj183-1 td/osd-backfill-space/datafile 2020-11-16T21:35:15.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:15.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj183-2 td/osd-backfill-space/datafile 2020-11-16T21:35:15.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:15.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:15.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:15.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj184-1 td/osd-backfill-space/datafile 2020-11-16T21:35:16.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:16.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj184-2 td/osd-backfill-space/datafile 2020-11-16T21:35:16.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:16.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:16.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:16.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj185-1 td/osd-backfill-space/datafile 2020-11-16T21:35:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj185-2 td/osd-backfill-space/datafile 2020-11-16T21:35:17.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:17.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:17.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:17.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj186-1 td/osd-backfill-space/datafile 2020-11-16T21:35:17.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:17.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj186-2 td/osd-backfill-space/datafile 2020-11-16T21:35:17.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:17.698 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:17.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:17.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj187-1 td/osd-backfill-space/datafile 2020-11-16T21:35:18.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:18.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj187-2 td/osd-backfill-space/datafile 2020-11-16T21:35:18.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:18.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:18.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:18.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj188-1 td/osd-backfill-space/datafile 2020-11-16T21:35:18.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:18.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj188-2 td/osd-backfill-space/datafile 2020-11-16T21:35:18.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:18.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:18.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:18.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj189-1 td/osd-backfill-space/datafile 2020-11-16T21:35:19.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:19.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj189-2 td/osd-backfill-space/datafile 2020-11-16T21:35:19.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:19.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:19.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:19.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj190-1 td/osd-backfill-space/datafile 2020-11-16T21:35:19.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:19.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj190-2 td/osd-backfill-space/datafile 2020-11-16T21:35:19.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:19.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:19.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:19.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj191-1 td/osd-backfill-space/datafile 2020-11-16T21:35:20.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:20.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj191-2 td/osd-backfill-space/datafile 2020-11-16T21:35:20.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:20.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj192-1 td/osd-backfill-space/datafile 2020-11-16T21:35:20.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:20.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj192-2 td/osd-backfill-space/datafile 2020-11-16T21:35:21.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:21.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:21.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:21.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj193-1 td/osd-backfill-space/datafile 2020-11-16T21:35:21.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:21.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj193-2 td/osd-backfill-space/datafile 2020-11-16T21:35:21.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:21.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:21.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:21.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj194-1 td/osd-backfill-space/datafile 2020-11-16T21:35:22.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:22.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj194-2 td/osd-backfill-space/datafile 2020-11-16T21:35:22.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:22.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:22.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:22.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj195-1 td/osd-backfill-space/datafile 2020-11-16T21:35:22.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:22.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj195-2 td/osd-backfill-space/datafile 2020-11-16T21:35:23.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:23.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:23.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:23.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj196-1 td/osd-backfill-space/datafile 2020-11-16T21:35:23.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:23.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj196-2 td/osd-backfill-space/datafile 2020-11-16T21:35:23.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:23.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:23.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:23.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj197-1 td/osd-backfill-space/datafile 2020-11-16T21:35:24.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:24.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj197-2 td/osd-backfill-space/datafile 2020-11-16T21:35:24.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:24.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:24.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:24.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj198-1 td/osd-backfill-space/datafile 2020-11-16T21:35:25.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:25.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj198-2 td/osd-backfill-space/datafile 2020-11-16T21:35:25.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:25.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj199-1 td/osd-backfill-space/datafile 2020-11-16T21:35:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj199-2 td/osd-backfill-space/datafile 2020-11-16T21:35:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2020-11-16T21:35:26.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:26.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:26.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj200-1 td/osd-backfill-space/datafile 2020-11-16T21:35:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj200-2 td/osd-backfill-space/datafile 2020-11-16T21:35:26.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:813: TEST_ec_backfill_multi: ceph pg dump pgs 2020-11-16T21:35:26.967 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:35:26.967 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 196 0 0 0 0 2408448 0 0 96 96 active+clean 2020-11-16T21:33:24.429846+0000 53'196 53:212 [0,2,3,4,5] 0 [0,2,3,4,5] 0 0'0 2020-11-16T21:33:21.669627+0000 0'0 2020-11-16T21:33:21.669627+0000 0 2020-11-16T21:35:26.967 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 0 0 0 1843200 0 0 100 100 active+clean 2020-11-16T21:32:58.537300+0000 37'200 50:230 [1] 1 [1] 1 0'0 2020-11-16T21:32:56.107196+0000 0'0 2020-11-16T21:32:56.107196+0000 0 2020-11-16T21:35:26.967 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 196 0 0 0 0 2408448 0 0 96 96 active+clean 2020-11-16T21:33:22.241571+0000 52'196 52:218 [0,2,3,4,5] 0 [0,2,3,4,5] 0 0'0 2020-11-16T21:33:19.773069+0000 0'0 2020-11-16T21:33:19.773069+0000 0 2020-11-16T21:35:26.968 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:35:26.968 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:35:26.969 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:35:26.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:815: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:35:26.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:815: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:26.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:817: TEST_ec_backfill_multi: expr 1 + 1 2020-11-16T21:35:26.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:817: TEST_ec_backfill_multi: ceph osd pg-upmap 2.0 0 2 3 4 1 2020-11-16T21:35:27.896 INFO:tasks.workunit.client.0.smithi189.stderr:set 2.0 pg_upmap mapping to [0,2,3,4,1] 2020-11-16T21:35:27.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:815: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2020-11-16T21:35:27.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:817: TEST_ec_backfill_multi: expr 2 + 1 2020-11-16T21:35:27.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:817: TEST_ec_backfill_multi: ceph osd pg-upmap 3.0 0 2 3 4 1 2020-11-16T21:35:28.994 INFO:tasks.workunit.client.0.smithi189.stderr:set 3.0 pg_upmap mapping to [0,2,3,4,1] 2020-11-16T21:35:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:820: TEST_ec_backfill_multi: sleep 30 2020-11-16T21:35:59.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:822: TEST_ec_backfill_multi: wait_for_not_backfilling 240 2020-11-16T21:35:59.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2020-11-16T21:35:59.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2020-11-16T21:35:59.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2020-11-16T21:35:59.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:35:59.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:35:59.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:35:59.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2020-11-16T21:35:59.015 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:35:59.015 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:35:59.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:35:59.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:35:59.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:35:59.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:35:59.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:35:59.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:35:59.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:35:59.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:35:59.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:35:59.730 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:35:59.730 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:35:59.730 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:35:59.730 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:35:59.730 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:35:59.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:35:59.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:35:59.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:35:59.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836529 2020-11-16T21:35:59.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836529 2020-11-16T21:35:59.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836529' 2020-11-16T21:35:59.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:35:59.926 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:36:00.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673006 2020-11-16T21:36:00.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673006 2020-11-16T21:36:00.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836529 1-42949673006' 2020-11-16T21:36:00.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:36:00.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:36:00.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542189 2020-11-16T21:36:00.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542189 2020-11-16T21:36:00.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-60129542189' 2020-11-16T21:36:00.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:36:00.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:36:00.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411371 2020-11-16T21:36:00.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411371 2020-11-16T21:36:00.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-60129542189 3-77309411371' 2020-11-16T21:36:00.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:36:00.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:36:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247850 2020-11-16T21:36:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247850 2020-11-16T21:36:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-60129542189 3-77309411371 4-98784247850' 2020-11-16T21:36:00.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:36:00.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:36:00.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084328 2020-11-16T21:36:00.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084328 2020-11-16T21:36:00.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-60129542189 3-77309411371 4-98784247850 5-120259084328' 2020-11-16T21:36:00.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:00.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836529 2020-11-16T21:36:00.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:00.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:36:00.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836529 2020-11-16T21:36:00.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:00.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836529 2020-11-16T21:36:00.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2020-11-16T21:36:00.769 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836529 2020-11-16T21:36:00.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:36:01.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836527 -lt 21474836529 2020-11-16T21:36:01.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:36:02.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:36:02.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:36:02.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836529 -lt 21474836529 2020-11-16T21:36:02.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:02.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673006 2020-11-16T21:36:02.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:02.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:36:02.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673006 2020-11-16T21:36:02.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:02.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673006 2020-11-16T21:36:02.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2020-11-16T21:36:02.564 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673006 2020-11-16T21:36:02.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:36:02.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673006 -lt 42949673006 2020-11-16T21:36:02.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:02.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542189 2020-11-16T21:36:02.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:02.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:36:02.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542189 2020-11-16T21:36:02.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:02.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542189 2020-11-16T21:36:02.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542189' 2020-11-16T21:36:02.962 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542189 2020-11-16T21:36:02.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:36:03.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542189 -lt 60129542189 2020-11-16T21:36:03.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:03.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-77309411371 2020-11-16T21:36:03.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:03.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:36:03.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411371 2020-11-16T21:36:03.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:03.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411371 2020-11-16T21:36:03.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 77309411371' 2020-11-16T21:36:03.388 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 77309411371 2020-11-16T21:36:03.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:36:03.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411372 -lt 77309411371 2020-11-16T21:36:03.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:03.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247850 2020-11-16T21:36:03.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:36:03.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247850 2020-11-16T21:36:03.788 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247850 2020-11-16T21:36:03.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247850' 2020-11-16T21:36:03.790 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247850 2020-11-16T21:36:03.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:36:04.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247850 -lt 98784247850 2020-11-16T21:36:04.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:04.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084328 2020-11-16T21:36:04.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:04.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:36:04.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084328 2020-11-16T21:36:04.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:04.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084328 2020-11-16T21:36:04.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084328' 2020-11-16T21:36:04.171 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 120259084328 2020-11-16T21:36:04.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:36:04.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084328 -lt 120259084328 2020-11-16T21:36:04.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:36:04.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:36:04.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:36:05.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 3 == 0 2020-11-16T21:36:05.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:36:05.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:36:05.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:36:05.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:36:05.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:36:05.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:36:05.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:36:05.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:36:05.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:36:05.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' -1 2020-11-16T21:36:05.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:68: wait_for_not_state: loop=0 2020-11-16T21:36:05.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:69: wait_for_not_state: num_in_state=1 2020-11-16T21:36:05.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 5 2020-11-16T21:36:10.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:36:10.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:36:10.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:36:10.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:36:10.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:36:10.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:36:10.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:36:10.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:36:10.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2020-11-16T21:36:10.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2020-11-16T21:36:10.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2020-11-16T21:36:10.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 1 >= 17 )) 2020-11-16T21:36:10.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 10 2020-11-16T21:36:20.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2020-11-16T21:36:20.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:36:20.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:36:20.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:36:20.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:36:20.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:36:20.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:36:20.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:36:21.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:36:21.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:36:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:36:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:36:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:823: TEST_ec_backfill_multi: wait_for_not_activating 60 2020-11-16T21:36:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2020-11-16T21:36:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2020-11-16T21:36:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2020-11-16T21:36:21.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:36:21.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:36:21.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:36:21.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2020-11-16T21:36:21.248 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:36:21.248 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:36:21.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:36:21.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:36:21.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:36:21.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:36:21.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:36:21.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:36:21.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:36:21.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:36:21.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:36:21.732 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:36:21.732 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:36:21.733 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:36:21.733 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:36:21.733 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:36:21.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:36:21.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:36:21.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:36:21.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836535 2020-11-16T21:36:21.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836535 2020-11-16T21:36:21.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836535' 2020-11-16T21:36:21.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:36:21.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:36:22.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673013 2020-11-16T21:36:22.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673013 2020-11-16T21:36:22.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836535 1-42949673013' 2020-11-16T21:36:22.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:36:22.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:36:22.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542195 2020-11-16T21:36:22.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542195 2020-11-16T21:36:22.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-60129542195' 2020-11-16T21:36:22.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:36:22.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:36:22.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411378 2020-11-16T21:36:22.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411378 2020-11-16T21:36:22.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-60129542195 3-77309411378' 2020-11-16T21:36:22.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:36:22.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:36:22.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247856 2020-11-16T21:36:22.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247856 2020-11-16T21:36:22.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-60129542195 3-77309411378 4-98784247856' 2020-11-16T21:36:22.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:36:22.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:36:22.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084334 2020-11-16T21:36:22.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084334 2020-11-16T21:36:22.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-60129542195 3-77309411378 4-98784247856 5-120259084334' 2020-11-16T21:36:22.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:22.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836535 2020-11-16T21:36:22.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:22.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:36:22.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836535 2020-11-16T21:36:22.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:22.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836535 2020-11-16T21:36:22.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2020-11-16T21:36:22.766 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836535 2020-11-16T21:36:22.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:36:23.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836533 -lt 21474836535 2020-11-16T21:36:23.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:36:24.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:36:24.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:36:24.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836535 -lt 21474836535 2020-11-16T21:36:24.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:24.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673013 2020-11-16T21:36:24.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:24.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:36:24.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:24.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673013 2020-11-16T21:36:24.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673013 2020-11-16T21:36:24.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673013' 2020-11-16T21:36:24.559 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673013 2020-11-16T21:36:24.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:36:24.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673013 -lt 42949673013 2020-11-16T21:36:24.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:24.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542195 2020-11-16T21:36:24.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:36:24.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542195 2020-11-16T21:36:24.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542195 2020-11-16T21:36:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542195' 2020-11-16T21:36:24.975 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542195 2020-11-16T21:36:24.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:36:25.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542195 -lt 60129542195 2020-11-16T21:36:25.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:25.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-77309411378 2020-11-16T21:36:25.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:25.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:36:25.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411378 2020-11-16T21:36:25.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411378 2020-11-16T21:36:25.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 77309411378' 2020-11-16T21:36:25.359 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 77309411378 2020-11-16T21:36:25.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:36:25.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411378 -lt 77309411378 2020-11-16T21:36:25.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:25.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247856 2020-11-16T21:36:25.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:25.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:36:25.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247856 2020-11-16T21:36:25.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:25.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247856 2020-11-16T21:36:25.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247856' 2020-11-16T21:36:25.753 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247856 2020-11-16T21:36:25.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:36:26.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247856 -lt 98784247856 2020-11-16T21:36:26.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:36:26.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084334 2020-11-16T21:36:26.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:36:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:36:26.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084334 2020-11-16T21:36:26.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:36:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084334 2020-11-16T21:36:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084334' 2020-11-16T21:36:26.148 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 120259084334 2020-11-16T21:36:26.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:36:26.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084335 -lt 120259084334 2020-11-16T21:36:26.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:36:26.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:36:26.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:36:26.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 3 == 0 2020-11-16T21:36:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:36:26.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2020-11-16T21:36:26.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2020-11-16T21:36:26.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:36:26.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2020-11-16T21:36:26.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:36:26.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2020-11-16T21:36:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:36:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:36:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:36:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:36:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:825: TEST_ec_backfill_multi: ceph pg dump pgs 2020-11-16T21:36:27.742 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:36:27.743 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 200 0 0 200 0 2457600 0 0 100 100 active+remapped+backfill_toofull 2020-11-16T21:36:00.350326+0000 54'200 64:244 [0,2,3,4,1] 0 [0,2,3,4,5] 0 0'0 2020-11-16T21:33:21.669627+0000 0'0 2020-11-16T21:33:21.669627+0000 0 2020-11-16T21:36:27.743 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 0 0 0 1843200 0 0 100 100 active+clean 2020-11-16T21:32:58.537300+0000 37'200 64:242 [1] 1 [1] 1 0'0 2020-11-16T21:32:56.107196+0000 0'0 2020-11-16T21:32:56.107196+0000 0 2020-11-16T21:36:27.743 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 200 0 0 0 0 2457600 0 0 100 100 active+clean 2020-11-16T21:36:11.628529+0000 54'200 65:659 [0,2,3,4,1] 0 [0,2,3,4,1] 0 0'0 2020-11-16T21:33:19.773069+0000 0'0 2020-11-16T21:33:19.773069+0000 0 2020-11-16T21:36:27.743 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:36:27.743 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:36:27.745 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:36:27.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:827: TEST_ec_backfill_multi: ERRORS=0 2020-11-16T21:36:27.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:828: TEST_ec_backfill_multi: ceph pg dump pgs 2020-11-16T21:36:27.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:828: TEST_ec_backfill_multi: grep -v '^1.0' 2020-11-16T21:36:27.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:828: TEST_ec_backfill_multi: grep +backfill_toofull 2020-11-16T21:36:27.757 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:828: TEST_ec_backfill_multi: wc -l 2020-11-16T21:36:28.128 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:36:28.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:828: TEST_ec_backfill_multi: '[' 1 '!=' 1 ']' 2020-11-16T21:36:28.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:834: TEST_ec_backfill_multi: ceph pg dump pgs 2020-11-16T21:36:28.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:834: TEST_ec_backfill_multi: grep -v '^1.0' 2020-11-16T21:36:28.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:834: TEST_ec_backfill_multi: grep active+clean 2020-11-16T21:36:28.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:834: TEST_ec_backfill_multi: wc -l 2020-11-16T21:36:28.486 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:36:28.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:834: TEST_ec_backfill_multi: '[' 1 '!=' 1 ']' 2020-11-16T21:36:28.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:840: TEST_ec_backfill_multi: '[' 0 '!=' 0 ']' 2020-11-16T21:36:28.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:845: TEST_ec_backfill_multi: delete_pool fillpool 2020-11-16T21:36:28.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=fillpool 2020-11-16T21:36:28.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete fillpool fillpool --yes-i-really-really-mean-it 2020-11-16T21:36:29.009 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fillpool' does not exist 2020-11-16T21:36:29.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:846: TEST_ec_backfill_multi: seq 1 2 2020-11-16T21:36:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:846: TEST_ec_backfill_multi: for i in $(seq 1 $pools) 2020-11-16T21:36:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:848: TEST_ec_backfill_multi: delete_pool test1 2020-11-16T21:36:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2020-11-16T21:36:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-11-16T21:36:29.568 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' does not exist 2020-11-16T21:36:29.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:846: TEST_ec_backfill_multi: for i in $(seq 1 $pools) 2020-11-16T21:36:29.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:848: TEST_ec_backfill_multi: delete_pool test2 2020-11-16T21:36:29.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2020-11-16T21:36:29.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-11-16T21:36:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' does not exist 2020-11-16T21:36:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:850: TEST_ec_backfill_multi: kill_daemons td/osd-backfill-space 2020-11-16T21:36:30.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:36:30.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:36:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:36:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:36:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:36:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:36:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2020-11-16T21:36:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:36:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:36:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:36:30.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:36:30.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:36:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:36:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:36:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:36:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:36:30.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:36:30.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:36:30.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:36:30.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:36:30.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:36:30.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:36:30.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:36:30.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:36:30.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:36:30.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:36:30.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:36:30.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:36:30.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:36:30.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:36:30.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:36:30.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:36:30.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:36:30.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:36:30.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:36:30.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2020-11-16T21:36:30.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2020-11-16T21:36:30.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2020-11-16T21:36:30.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2020-11-16T21:36:30.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:36:30.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:36:30.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:36:30.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:36:30.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:36:30.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:36:30.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:36:30.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:36:30.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:36:30.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:36:30.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:36:30.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:36:30.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:36:30.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:36:30.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:36:30.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:36:30.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:36:30.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:36:30.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:36:30.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:36:30.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:36:30.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:36:30.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:36:30.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:36:30.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:36:30.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:36:30.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:36:30.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:36:30.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2020-11-16T21:36:30.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:36:30.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:36:30.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:36:30.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137323 2020-11-16T21:36:30.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:36:30.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:36:30.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:36:30.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2020-11-16T21:36:30.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_ec_backfill_simple td/osd-backfill-space 2020-11-16T21:36:30.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:623: TEST_ec_backfill_simple: local dir=td/osd-backfill-space 2020-11-16T21:36:30.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:624: TEST_ec_backfill_simple: local EC= 2020-11-16T21:36:30.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:625: TEST_ec_backfill_simple: local pools=1 2020-11-16T21:36:30.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:626: TEST_ec_backfill_simple: local OSDS=6 2020-11-16T21:36:30.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:627: TEST_ec_backfill_simple: local k=3 2020-11-16T21:36:30.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:628: TEST_ec_backfill_simple: local m=2 2020-11-16T21:36:30.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:629: TEST_ec_backfill_simple: expr 600 / 3 2020-11-16T21:36:30.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:629: TEST_ec_backfill_simple: local ecobjects=200 2020-11-16T21:36:30.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:631: TEST_ec_backfill_simple: run_mon td/osd-backfill-space a 2020-11-16T21:36:30.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2020-11-16T21:36:30.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:36:30.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:36:30.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:36:30.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2020-11-16T21:36:30.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2020-11-16T21:36:30.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:36:30.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:36:30.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:36:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:36:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:36:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:36:30.737 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:36:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:36:30.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:36:30.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:36:30.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:36:30.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:36:30.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:36:30.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:36:30.952 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:36:30.952 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:36:30.952 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:36:30.953 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:36:30.953 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:36:30.953 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:36:30.953 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:36:30.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:36:30.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get fsid 2020-11-16T21:36:31.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:36:31.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:36:31.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:36:31.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:36:31.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:36:31.060 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:36:31.061 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:36:31.061 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:36:31.061 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:36:31.061 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:36:31.061 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:36:31.062 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137323/ceph-mon.a.asok 2020-11-16T21:36:31.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:36:31.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.137323/ceph-mon.a.asok config get mon_host 2020-11-16T21:36:31.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:632: TEST_ec_backfill_simple: run_mgr td/osd-backfill-space x 2020-11-16T21:36:31.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2020-11-16T21:36:31.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:36:31.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:36:31.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:36:31.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2020-11-16T21:36:31.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:36:31.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:36:31.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:36:31.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:36:31.715 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:36:31.715 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:36:31.715 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:36:31.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:36:31.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:36:31.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:36:31.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:633: TEST_ec_backfill_simple: export CEPH_ARGS 2020-11-16T21:36:31.744 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: expr 6 - 1 2020-11-16T21:36:31.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: seq 0 5 2020-11-16T21:36:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:36:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 0 2020-11-16T21:36:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:36:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:36:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:36:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:36:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2020-11-16T21:36:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:36:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:36:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:36:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:36:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2020-11-16T21:36:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2020-11-16T21:36:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:36:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:36:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:36:31.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:36:31.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:36:31.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:36:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:36:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:36:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:36:31.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:36:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:36:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:36:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:36:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:36:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:36:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:36:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:36:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:36:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:36:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:36:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:36:31.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2020-11-16T21:36:31.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:36:31.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6afb27f0-47cf-4cfc-aa55-6bcb8e0e0d56 2020-11-16T21:36:31.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 6afb27f0-47cf-4cfc-aa55-6bcb8e0e0d56' 2020-11-16T21:36:31.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:36:31.755 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 6afb27f0-47cf-4cfc-aa55-6bcb8e0e0d56 2020-11-16T21:36:31.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDf8LJfKseqLhAADlnaDyFZO9XoaDN5vdlnYw== 2020-11-16T21:36:31.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDf8LJfKseqLhAADlnaDyFZO9XoaDN5vdlnYw=="}' 2020-11-16T21:36:31.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6afb27f0-47cf-4cfc-aa55-6bcb8e0e0d56 -i td/osd-backfill-space/0/new.json 2020-11-16T21:36:32.112 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:36:32.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2020-11-16T21:36:32.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDf8LJfKseqLhAADlnaDyFZO9XoaDN5vdlnYw== --osd-uuid 6afb27f0-47cf-4cfc-aa55-6bcb8e0e0d56 2020-11-16T21:36:32.160 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:32.165+0000 7f506492cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:32.163 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:32.169+0000 7f506492cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:32.164 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:32.169+0000 7f506492cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:32.164 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:32.169+0000 7f506492cd80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:36:32.165 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:32.169+0000 7f506492cd80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:36:32.165 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:32.169+0000 7f506492cd80 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2020-11-16T21:36:32.214 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:32.217+0000 7f506492cd80 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2020-11-16T21:36:36.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2020-11-16T21:36:36.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:36:36.627 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:36:36.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:36:36.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:36:37.124 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:36:37.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:36:37.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:36:37.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:36:37.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:36:37.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:36:37.155 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:37.161+0000 7fe503241d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:37.157 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:37.161+0000 7fe503241d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:37.158 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:37.161+0000 7fe503241d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:37.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:36:37.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:36:37.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:36:37.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:36:37.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:36:37.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:36:37.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:36:37.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:36:37.531 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:36:37.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:36:37.700 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:37.705+0000 7fe503241d80 -1 Falling back to public interface 2020-11-16T21:36:37.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:36:38.523 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:38.525+0000 7fe503241d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:36:38.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:36:38.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:36:38.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:36:38.901 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:36:38.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:36:38.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:36:39.283 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/450376,v1:127.0.0.1:6803/450376] [v2:127.0.0.1:6804/450376,v1:127.0.0.1:6805/450376] exists,up 6afb27f0-47cf-4cfc-aa55-6bcb8e0e0d56 2020-11-16T21:36:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:36:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:36:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:36:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:36:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 1 2020-11-16T21:36:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:36:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:36:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:36:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:36:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2020-11-16T21:36:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:36:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:36:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:36:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:36:39.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2020-11-16T21:36:39.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2020-11-16T21:36:39.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:36:39.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:36:39.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:36:39.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:36:39.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:36:39.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:36:39.288 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:36:39.288 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:36:39.288 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:36:39.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:36:39.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:36:39.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:36:39.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:36:39.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:36:39.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:36:39.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:36:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:36:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:36:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:36:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:36:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:36:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2020-11-16T21:36:39.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:36:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fe40f1f9-16be-4f31-bcd2-e87705feeab2 2020-11-16T21:36:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 fe40f1f9-16be-4f31-bcd2-e87705feeab2' 2020-11-16T21:36:39.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:36:39.292 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 fe40f1f9-16be-4f31-bcd2-e87705feeab2 2020-11-16T21:36:39.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDn8LJfhJsuExAALcL2OB5fdiTaks47r7N2kg== 2020-11-16T21:36:39.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDn8LJfhJsuExAALcL2OB5fdiTaks47r7N2kg=="}' 2020-11-16T21:36:39.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fe40f1f9-16be-4f31-bcd2-e87705feeab2 -i td/osd-backfill-space/1/new.json 2020-11-16T21:36:39.901 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:36:39.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2020-11-16T21:36:39.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDn8LJfhJsuExAALcL2OB5fdiTaks47r7N2kg== --osd-uuid fe40f1f9-16be-4f31-bcd2-e87705feeab2 2020-11-16T21:36:39.946 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:39.953+0000 7f1a29de5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:39.949 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:39.953+0000 7f1a29de5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:39.951 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:39.957+0000 7f1a29de5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:39.951 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:39.957+0000 7f1a29de5d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:36:39.951 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:39.957+0000 7f1a29de5d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:36:39.951 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:39.957+0000 7f1a29de5d80 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2020-11-16T21:36:40.001 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:40.005+0000 7f1a29de5d80 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2020-11-16T21:36:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2020-11-16T21:36:44.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:36:44.615 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:36:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:36:44.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:36:45.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:36:45.120 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:36:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:36:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:36:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:36:45.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:36:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:45.157+0000 7f185f919d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:45.161+0000 7f185f919d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:45.161+0000 7f185f919d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:45.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:36:45.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:36:45.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:36:45.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:36:45.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:36:45.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:36:45.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:36:45.539 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:36:45.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:36:45.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:36:45.712 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:45.717+0000 7f185f919d80 -1 Falling back to public interface 2020-11-16T21:36:45.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:36:46.568 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:46.573+0000 7f185f919d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:36:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:36:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:36:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:36:46.916 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:36:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:36:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:36:47.307 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/451370,v1:127.0.0.1:6811/451370] [v2:127.0.0.1:6812/451370,v1:127.0.0.1:6813/451370] exists,up fe40f1f9-16be-4f31-bcd2-e87705feeab2 2020-11-16T21:36:47.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:36:47.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:36:47.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:36:47.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:36:47.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 2 2020-11-16T21:36:47.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:36:47.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:36:47.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:36:47.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:36:47.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2020-11-16T21:36:47.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:36:47.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:36:47.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:36:47.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:36:47.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2020-11-16T21:36:47.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2020-11-16T21:36:47.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:36:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:36:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:36:47.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:36:47.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:36:47.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:36:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:36:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:36:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:36:47.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:36:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:36:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:36:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:36:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:36:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:36:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:36:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:36:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:36:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:36:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:36:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:36:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2020-11-16T21:36:47.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:36:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3539ef3c-2656-4615-92ee-3f1db1042778 2020-11-16T21:36:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 3539ef3c-2656-4615-92ee-3f1db1042778' 2020-11-16T21:36:47.316 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 3539ef3c-2656-4615-92ee-3f1db1042778 2020-11-16T21:36:47.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:36:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDv8LJfezRQFBAAuPCQTV1YBRedQH8JNxDFCg== 2020-11-16T21:36:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDv8LJfezRQFBAAuPCQTV1YBRedQH8JNxDFCg=="}' 2020-11-16T21:36:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3539ef3c-2656-4615-92ee-3f1db1042778 -i td/osd-backfill-space/2/new.json 2020-11-16T21:36:47.817 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:36:47.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2020-11-16T21:36:47.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDv8LJfezRQFBAAuPCQTV1YBRedQH8JNxDFCg== --osd-uuid 3539ef3c-2656-4615-92ee-3f1db1042778 2020-11-16T21:36:47.860 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:47.865+0000 7f7910db0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:47.864 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:47.869+0000 7f7910db0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:47.865 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:47.869+0000 7f7910db0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:47.865 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:47.869+0000 7f7910db0d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:36:47.865 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:47.869+0000 7f7910db0d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:36:47.866 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:47.869+0000 7f7910db0d80 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2020-11-16T21:36:47.903 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:47.909+0000 7f7910db0d80 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2020-11-16T21:36:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2020-11-16T21:36:52.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:36:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:36:52.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:36:52.289 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:36:52.791 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:36:52.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:36:52.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:36:52.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:36:52.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:36:52.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:36:52.822 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:52.829+0000 7f3c23849d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:52.824 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:52.829+0000 7f3c23849d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:52.826 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:52.833+0000 7f3c23849d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:53.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:36:53.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:36:53.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:36:53.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:36:53.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:36:53.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:36:53.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:36:53.178 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:36:53.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:36:53.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:36:53.368 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:53.373+0000 7f3c23849d80 -1 Falling back to public interface 2020-11-16T21:36:53.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:36:54.201 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:54.205+0000 7f3c23849d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:36:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:36:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:36:54.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:36:54.553 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:36:54.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:36:54.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:36:54.993 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/452364,v1:127.0.0.1:6819/452364] [v2:127.0.0.1:6820/452364,v1:127.0.0.1:6821/452364] exists,up 3539ef3c-2656-4615-92ee-3f1db1042778 2020-11-16T21:36:54.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:36:54.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:36:54.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:36:54.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:36:54.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 3 2020-11-16T21:36:54.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:36:54.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:36:54.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:36:54.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:36:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/3 2020-11-16T21:36:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:36:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:36:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:36:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:36:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/3' 2020-11-16T21:36:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/3/journal' 2020-11-16T21:36:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:36:54.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:36:54.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:36:54.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:36:54.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:36:54.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:36:54.997 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:36:54.998 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:36:54.998 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:36:54.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:36:54.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:36:54.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:36:54.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:36:54.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:36:54.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:36:54.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:36:54.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:36:54.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:36:54.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:36:55.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:36:55.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:36:55.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/3 2020-11-16T21:36:55.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:36:55.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=491e6423-abf3-4691-a3c4-ac0151e1983d 2020-11-16T21:36:55.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 491e6423-abf3-4691-a3c4-ac0151e1983d' 2020-11-16T21:36:55.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:36:55.001 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 491e6423-abf3-4691-a3c4-ac0151e1983d 2020-11-16T21:36:55.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD38LJfd3nRARAACNDdvaihXRoDEeOUjMSdJA== 2020-11-16T21:36:55.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD38LJfd3nRARAACNDdvaihXRoDEeOUjMSdJA=="}' 2020-11-16T21:36:55.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 491e6423-abf3-4691-a3c4-ac0151e1983d -i td/osd-backfill-space/3/new.json 2020-11-16T21:36:55.566 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:36:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/3/new.json 2020-11-16T21:36:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD38LJfd3nRARAACNDdvaihXRoDEeOUjMSdJA== --osd-uuid 491e6423-abf3-4691-a3c4-ac0151e1983d 2020-11-16T21:36:55.612 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:55.617+0000 7fa2130e4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:55.614 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:55.621+0000 7fa2130e4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:55.616 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:55.621+0000 7fa2130e4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:36:55.617 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:55.621+0000 7fa2130e4d80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:36:55.617 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:55.621+0000 7fa2130e4d80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:36:55.617 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:55.621+0000 7fa2130e4d80 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2020-11-16T21:36:55.654 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:36:55.661+0000 7fa2130e4d80 -1 bluestore(td/osd-backfill-space/3) _read_fsid unparsable uuid 2020-11-16T21:36:59.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/3/keyring 2020-11-16T21:36:59.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:37:03.964 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:37:03.965 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:37:03.965 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:37:03.965 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:37:03.965 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:37:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:37:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:37:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:37:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:37:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:37:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:37:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:37:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:00.485+0000 7f0d6ab09d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:00.489+0000 7f0d6ab09d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:00.489+0000 7f0d6ab09d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:37:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:37:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:37:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:37:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:37:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:37:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:37:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:37:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:37:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:01.025+0000 7f0d6ab09d80 -1 Falling back to public interface 2020-11-16T21:37:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:37:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:01.977+0000 7f0d6ab09d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:37:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:37:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:37:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:37:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:37:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:37:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:37:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:37:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:37:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:37:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:37:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:37:04.069 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/453358,v1:127.0.0.1:6827/453358] [v2:127.0.0.1:6828/453358,v1:127.0.0.1:6829/453358] exists,up 491e6423-abf3-4691-a3c4-ac0151e1983d 2020-11-16T21:37:04.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:37:04.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:37:04.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:37:04.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:37:04.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 4 2020-11-16T21:37:04.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:37:04.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:37:04.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:37:04.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:37:04.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/4 2020-11-16T21:37:04.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:37:04.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:37:04.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:37:04.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:37:04.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/4' 2020-11-16T21:37:04.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/4/journal' 2020-11-16T21:37:04.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:37:04.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:37:04.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:37:04.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:37:04.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:37:04.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:37:04.075 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:37:04.075 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:37:04.075 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:37:04.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:37:04.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:37:04.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:37:04.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:37:04.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:37:04.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:37:04.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:37:04.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:37:04.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:37:04.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:37:04.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:37:04.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:37:04.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/4 2020-11-16T21:37:04.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:37:04.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3ba527bd-4372-4948-89e6-8c9ebcf7bbbf 2020-11-16T21:37:04.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 3ba527bd-4372-4948-89e6-8c9ebcf7bbbf' 2020-11-16T21:37:04.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:37:04.078 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 3ba527bd-4372-4948-89e6-8c9ebcf7bbbf 2020-11-16T21:37:04.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAA8bJfubwpBhAAGM7D/GHDtcvbYvAMDkaCvw== 2020-11-16T21:37:04.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAA8bJfubwpBhAAGM7D/GHDtcvbYvAMDkaCvw=="}' 2020-11-16T21:37:04.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3ba527bd-4372-4948-89e6-8c9ebcf7bbbf -i td/osd-backfill-space/4/new.json 2020-11-16T21:37:04.565 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:37:04.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/4/new.json 2020-11-16T21:37:04.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAA8bJfubwpBhAAGM7D/GHDtcvbYvAMDkaCvw== --osd-uuid 3ba527bd-4372-4948-89e6-8c9ebcf7bbbf 2020-11-16T21:37:04.614 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:04.616+0000 7f95854d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:04.638 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:04.640+0000 7f95854d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:04.639 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:04.644+0000 7f95854d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:04.639 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:04.644+0000 7f95854d6d80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:37:04.640 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:04.644+0000 7f95854d6d80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:37:04.640 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:04.644+0000 7f95854d6d80 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2020-11-16T21:37:04.694 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:04.696+0000 7f95854d6d80 -1 bluestore(td/osd-backfill-space/4) _read_fsid unparsable uuid 2020-11-16T21:37:09.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/4/keyring 2020-11-16T21:37:09.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:37:09.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:37:09.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:37:09.276 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:37:09.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:37:09.775 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:37:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:37:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:37:09.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:37:09.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:37:09.806 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:09.808+0000 7fb24e94dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:09.809 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:09.812+0000 7fb24e94dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:09.810 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:09.816+0000 7fb24e94dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:10.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:37:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:37:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:37:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:37:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:37:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:37:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:37:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:37:10.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:37:10.147 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:37:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:10.352+0000 7fb24e94dd80 -1 Falling back to public interface 2020-11-16T21:37:10.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:37:11.174 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:11.180+0000 7fb24e94dd80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:37:11.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:37:11.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:37:11.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:37:11.554 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:37:11.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:37:11.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:37:11.974 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/454378,v1:127.0.0.1:6835/454378] [v2:127.0.0.1:6836/454378,v1:127.0.0.1:6837/454378] exists,up 3ba527bd-4372-4948-89e6-8c9ebcf7bbbf 2020-11-16T21:37:11.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:37:11.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:37:11.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:37:11.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T21:37:11.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 5 2020-11-16T21:37:11.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2020-11-16T21:37:11.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:37:11.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T21:37:11.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:37:11.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/5 2020-11-16T21:37:11.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:37:11.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:37:11.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:37:11.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:37:11.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/5' 2020-11-16T21:37:11.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/5/journal' 2020-11-16T21:37:11.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:37:11.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:37:11.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:37:11.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:37:11.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:37:11.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:37:11.979 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:37:11.979 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:37:11.979 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:37:11.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:37:11.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:37:11.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:37:11.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:37:11.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:37:11.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:37:11.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:37:11.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:37:11.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:37:11.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:37:11.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:37:11.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:37:11.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/5 2020-11-16T21:37:11.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:37:11.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5a13bf55-8383-4e96-9d49-021d14edc255 2020-11-16T21:37:11.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 5a13bf55-8383-4e96-9d49-021d14edc255' 2020-11-16T21:37:11.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:37:11.982 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 5a13bf55-8383-4e96-9d49-021d14edc255 2020-11-16T21:37:11.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAI8bJf3zpbABAAvoYBxcHTsncgYDDccHhI/Q== 2020-11-16T21:37:11.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAI8bJf3zpbABAAvoYBxcHTsncgYDDccHhI/Q=="}' 2020-11-16T21:37:11.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5a13bf55-8383-4e96-9d49-021d14edc255 -i td/osd-backfill-space/5/new.json 2020-11-16T21:37:12.569 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T21:37:12.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/5/new.json 2020-11-16T21:37:12.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAI8bJf3zpbABAAvoYBxcHTsncgYDDccHhI/Q== --osd-uuid 5a13bf55-8383-4e96-9d49-021d14edc255 2020-11-16T21:37:12.622 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:12.624+0000 7ff933d8bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:12.640+0000 7ff933d8bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:12.636 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:12.640+0000 7ff933d8bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:12.640+0000 7ff933d8bd80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:37:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:12.640+0000 7ff933d8bd80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:37:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:12.640+0000 7ff933d8bd80 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2020-11-16T21:37:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:12.712+0000 7ff933d8bd80 -1 bluestore(td/osd-backfill-space/5) _read_fsid unparsable uuid 2020-11-16T21:37:17.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/5/keyring 2020-11-16T21:37:17.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:37:17.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T21:37:17.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:37:17.375 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T21:37:17.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T21:37:17.884 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T21:37:17.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:37:17.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:37:17.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:37:17.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:37:17.915 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:17.920+0000 7feca42d5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:17.917 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:17.920+0000 7feca42d5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:17.918 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:17.924+0000 7feca42d5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:37:18.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T21:37:18.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:37:18.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T21:37:18.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:37:18.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:37:18.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:37:18.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:37:18.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:37:18.275 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:37:18.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:37:18.468 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:18.472+0000 7feca42d5d80 -1 Falling back to public interface 2020-11-16T21:37:18.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:37:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:37:19.340+0000 7feca42d5d80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T21:37:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:37:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:37:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:37:19.663 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:37:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:37:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:37:20.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:37:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:37:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:37:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:37:21.035 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:37:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:37:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:37:21.430 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/455372,v1:127.0.0.1:6843/455372] [v2:127.0.0.1:6844/455372,v1:127.0.0.1:6845/455372] exists,up 5a13bf55-8383-4e96-9d49-021d14edc255 2020-11-16T21:37:21.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:37:21.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:37:21.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:37:21.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:640: TEST_ec_backfill_simple: ceph osd set-backfillfull-ratio .85 2020-11-16T21:37:22.186 INFO:tasks.workunit.client.0.smithi189.stderr:osd set-backfillfull-ratio 0.85 2020-11-16T21:37:22.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:641: TEST_ec_backfill_simple: create_pool fillpool 1 1 2020-11-16T21:37:22.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create fillpool 1 1 2020-11-16T21:37:22.798 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fillpool' already exists 2020-11-16T21:37:22.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:37:23.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:642: TEST_ec_backfill_simple: ceph osd pool set fillpool size 1 --yes-i-really-mean-it 2020-11-16T21:37:24.553 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 1 2020-11-16T21:37:24.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:651: TEST_ec_backfill_simple: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=13 2020-11-16T21:37:24.564 INFO:tasks.workunit.client.0.smithi189.stderr:13+0 records in 2020-11-16T21:37:24.565 INFO:tasks.workunit.client.0.smithi189.stderr:13+0 records out 2020-11-16T21:37:24.565 INFO:tasks.workunit.client.0.smithi189.stderr:13312 bytes (13 kB, 13 KiB) copied, 0.000387017 s, 34.4 MB/s 2020-11-16T21:37:24.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: seq 1 200 2020-11-16T21:37:24.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:24.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj1 td/osd-backfill-space/datafile 2020-11-16T21:37:25.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:25.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj2 td/osd-backfill-space/datafile 2020-11-16T21:37:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj3 td/osd-backfill-space/datafile 2020-11-16T21:37:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj4 td/osd-backfill-space/datafile 2020-11-16T21:37:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj5 td/osd-backfill-space/datafile 2020-11-16T21:37:25.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:25.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj6 td/osd-backfill-space/datafile 2020-11-16T21:37:25.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:25.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj7 td/osd-backfill-space/datafile 2020-11-16T21:37:25.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:25.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj8 td/osd-backfill-space/datafile 2020-11-16T21:37:25.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:25.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj9 td/osd-backfill-space/datafile 2020-11-16T21:37:26.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:26.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj10 td/osd-backfill-space/datafile 2020-11-16T21:37:26.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:26.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj11 td/osd-backfill-space/datafile 2020-11-16T21:37:26.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:26.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj12 td/osd-backfill-space/datafile 2020-11-16T21:37:26.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:26.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj13 td/osd-backfill-space/datafile 2020-11-16T21:37:26.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:26.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj14 td/osd-backfill-space/datafile 2020-11-16T21:37:26.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:26.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj15 td/osd-backfill-space/datafile 2020-11-16T21:37:26.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:26.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj16 td/osd-backfill-space/datafile 2020-11-16T21:37:26.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:26.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj17 td/osd-backfill-space/datafile 2020-11-16T21:37:26.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:26.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj18 td/osd-backfill-space/datafile 2020-11-16T21:37:26.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:26.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj19 td/osd-backfill-space/datafile 2020-11-16T21:37:27.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:27.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj20 td/osd-backfill-space/datafile 2020-11-16T21:37:27.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:27.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj21 td/osd-backfill-space/datafile 2020-11-16T21:37:27.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:27.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj22 td/osd-backfill-space/datafile 2020-11-16T21:37:27.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:27.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj23 td/osd-backfill-space/datafile 2020-11-16T21:37:27.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:27.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj24 td/osd-backfill-space/datafile 2020-11-16T21:37:27.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:27.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj25 td/osd-backfill-space/datafile 2020-11-16T21:37:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj26 td/osd-backfill-space/datafile 2020-11-16T21:37:27.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:27.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj27 td/osd-backfill-space/datafile 2020-11-16T21:37:27.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:27.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj28 td/osd-backfill-space/datafile 2020-11-16T21:37:27.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:27.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj29 td/osd-backfill-space/datafile 2020-11-16T21:37:28.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:28.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj30 td/osd-backfill-space/datafile 2020-11-16T21:37:28.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:28.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj31 td/osd-backfill-space/datafile 2020-11-16T21:37:28.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:28.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj32 td/osd-backfill-space/datafile 2020-11-16T21:37:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj33 td/osd-backfill-space/datafile 2020-11-16T21:37:28.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:28.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj34 td/osd-backfill-space/datafile 2020-11-16T21:37:28.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:28.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj35 td/osd-backfill-space/datafile 2020-11-16T21:37:28.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:28.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj36 td/osd-backfill-space/datafile 2020-11-16T21:37:28.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:28.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj37 td/osd-backfill-space/datafile 2020-11-16T21:37:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj38 td/osd-backfill-space/datafile 2020-11-16T21:37:28.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:28.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj39 td/osd-backfill-space/datafile 2020-11-16T21:37:29.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:29.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj40 td/osd-backfill-space/datafile 2020-11-16T21:37:29.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:29.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj41 td/osd-backfill-space/datafile 2020-11-16T21:37:29.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:29.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj42 td/osd-backfill-space/datafile 2020-11-16T21:37:29.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:29.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj43 td/osd-backfill-space/datafile 2020-11-16T21:37:33.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj44 td/osd-backfill-space/datafile 2020-11-16T21:37:33.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj45 td/osd-backfill-space/datafile 2020-11-16T21:37:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj46 td/osd-backfill-space/datafile 2020-11-16T21:37:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj47 td/osd-backfill-space/datafile 2020-11-16T21:37:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj48 td/osd-backfill-space/datafile 2020-11-16T21:37:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj49 td/osd-backfill-space/datafile 2020-11-16T21:37:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj50 td/osd-backfill-space/datafile 2020-11-16T21:37:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj51 td/osd-backfill-space/datafile 2020-11-16T21:37:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj52 td/osd-backfill-space/datafile 2020-11-16T21:37:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj53 td/osd-backfill-space/datafile 2020-11-16T21:37:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj54 td/osd-backfill-space/datafile 2020-11-16T21:37:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj55 td/osd-backfill-space/datafile 2020-11-16T21:37:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj56 td/osd-backfill-space/datafile 2020-11-16T21:37:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj57 td/osd-backfill-space/datafile 2020-11-16T21:37:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj58 td/osd-backfill-space/datafile 2020-11-16T21:37:33.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj59 td/osd-backfill-space/datafile 2020-11-16T21:37:33.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj60 td/osd-backfill-space/datafile 2020-11-16T21:37:33.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj61 td/osd-backfill-space/datafile 2020-11-16T21:37:33.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj62 td/osd-backfill-space/datafile 2020-11-16T21:37:33.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj63 td/osd-backfill-space/datafile 2020-11-16T21:37:33.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj64 td/osd-backfill-space/datafile 2020-11-16T21:37:33.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj65 td/osd-backfill-space/datafile 2020-11-16T21:37:33.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj66 td/osd-backfill-space/datafile 2020-11-16T21:37:33.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj67 td/osd-backfill-space/datafile 2020-11-16T21:37:33.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj68 td/osd-backfill-space/datafile 2020-11-16T21:37:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj69 td/osd-backfill-space/datafile 2020-11-16T21:37:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj70 td/osd-backfill-space/datafile 2020-11-16T21:37:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj71 td/osd-backfill-space/datafile 2020-11-16T21:37:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj72 td/osd-backfill-space/datafile 2020-11-16T21:37:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj73 td/osd-backfill-space/datafile 2020-11-16T21:37:33.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj74 td/osd-backfill-space/datafile 2020-11-16T21:37:33.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj75 td/osd-backfill-space/datafile 2020-11-16T21:37:33.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj76 td/osd-backfill-space/datafile 2020-11-16T21:37:33.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj77 td/osd-backfill-space/datafile 2020-11-16T21:37:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj78 td/osd-backfill-space/datafile 2020-11-16T21:37:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj79 td/osd-backfill-space/datafile 2020-11-16T21:37:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj80 td/osd-backfill-space/datafile 2020-11-16T21:37:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj81 td/osd-backfill-space/datafile 2020-11-16T21:37:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj82 td/osd-backfill-space/datafile 2020-11-16T21:37:33.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj83 td/osd-backfill-space/datafile 2020-11-16T21:37:33.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj84 td/osd-backfill-space/datafile 2020-11-16T21:37:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj85 td/osd-backfill-space/datafile 2020-11-16T21:37:33.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj86 td/osd-backfill-space/datafile 2020-11-16T21:37:33.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj87 td/osd-backfill-space/datafile 2020-11-16T21:37:33.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj88 td/osd-backfill-space/datafile 2020-11-16T21:37:33.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj89 td/osd-backfill-space/datafile 2020-11-16T21:37:33.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:33.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj90 td/osd-backfill-space/datafile 2020-11-16T21:37:34.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:34.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj91 td/osd-backfill-space/datafile 2020-11-16T21:37:34.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:34.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj92 td/osd-backfill-space/datafile 2020-11-16T21:37:34.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:34.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj93 td/osd-backfill-space/datafile 2020-11-16T21:37:34.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:34.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj94 td/osd-backfill-space/datafile 2020-11-16T21:37:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj95 td/osd-backfill-space/datafile 2020-11-16T21:37:34.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:34.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj96 td/osd-backfill-space/datafile 2020-11-16T21:37:34.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:34.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj97 td/osd-backfill-space/datafile 2020-11-16T21:37:34.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:34.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj98 td/osd-backfill-space/datafile 2020-11-16T21:37:34.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:34.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj99 td/osd-backfill-space/datafile 2020-11-16T21:37:34.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj100 td/osd-backfill-space/datafile 2020-11-16T21:37:34.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:34.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj101 td/osd-backfill-space/datafile 2020-11-16T21:37:35.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:35.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj102 td/osd-backfill-space/datafile 2020-11-16T21:37:35.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:35.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj103 td/osd-backfill-space/datafile 2020-11-16T21:37:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:35.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj104 td/osd-backfill-space/datafile 2020-11-16T21:37:35.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:35.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj105 td/osd-backfill-space/datafile 2020-11-16T21:37:35.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:35.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj106 td/osd-backfill-space/datafile 2020-11-16T21:37:35.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:35.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj107 td/osd-backfill-space/datafile 2020-11-16T21:37:35.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:35.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj108 td/osd-backfill-space/datafile 2020-11-16T21:37:35.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:35.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj109 td/osd-backfill-space/datafile 2020-11-16T21:37:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj110 td/osd-backfill-space/datafile 2020-11-16T21:37:35.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:35.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj111 td/osd-backfill-space/datafile 2020-11-16T21:37:36.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:36.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj112 td/osd-backfill-space/datafile 2020-11-16T21:37:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj113 td/osd-backfill-space/datafile 2020-11-16T21:37:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj114 td/osd-backfill-space/datafile 2020-11-16T21:37:36.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:36.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj115 td/osd-backfill-space/datafile 2020-11-16T21:37:36.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:36.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj116 td/osd-backfill-space/datafile 2020-11-16T21:37:36.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:36.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj117 td/osd-backfill-space/datafile 2020-11-16T21:37:36.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:36.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj118 td/osd-backfill-space/datafile 2020-11-16T21:37:36.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:36.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj119 td/osd-backfill-space/datafile 2020-11-16T21:37:36.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj120 td/osd-backfill-space/datafile 2020-11-16T21:37:37.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:37.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj121 td/osd-backfill-space/datafile 2020-11-16T21:37:37.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:37.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj122 td/osd-backfill-space/datafile 2020-11-16T21:37:37.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:37.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj123 td/osd-backfill-space/datafile 2020-11-16T21:37:37.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:37.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj124 td/osd-backfill-space/datafile 2020-11-16T21:37:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj125 td/osd-backfill-space/datafile 2020-11-16T21:37:37.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:37.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj126 td/osd-backfill-space/datafile 2020-11-16T21:37:37.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:37.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj127 td/osd-backfill-space/datafile 2020-11-16T21:37:37.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:37.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj128 td/osd-backfill-space/datafile 2020-11-16T21:37:37.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:37.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj129 td/osd-backfill-space/datafile 2020-11-16T21:37:37.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:37.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj130 td/osd-backfill-space/datafile 2020-11-16T21:37:38.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:38.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj131 td/osd-backfill-space/datafile 2020-11-16T21:37:38.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:38.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj132 td/osd-backfill-space/datafile 2020-11-16T21:37:38.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:38.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj133 td/osd-backfill-space/datafile 2020-11-16T21:37:38.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:38.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj134 td/osd-backfill-space/datafile 2020-11-16T21:37:38.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:38.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj135 td/osd-backfill-space/datafile 2020-11-16T21:37:38.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:38.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj136 td/osd-backfill-space/datafile 2020-11-16T21:37:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:38.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj137 td/osd-backfill-space/datafile 2020-11-16T21:37:38.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:38.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj138 td/osd-backfill-space/datafile 2020-11-16T21:37:38.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:38.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj139 td/osd-backfill-space/datafile 2020-11-16T21:37:38.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:38.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj140 td/osd-backfill-space/datafile 2020-11-16T21:37:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:38.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj141 td/osd-backfill-space/datafile 2020-11-16T21:37:39.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:39.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj142 td/osd-backfill-space/datafile 2020-11-16T21:37:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj143 td/osd-backfill-space/datafile 2020-11-16T21:37:39.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:39.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj144 td/osd-backfill-space/datafile 2020-11-16T21:37:39.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:39.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj145 td/osd-backfill-space/datafile 2020-11-16T21:37:39.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:39.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj146 td/osd-backfill-space/datafile 2020-11-16T21:37:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj147 td/osd-backfill-space/datafile 2020-11-16T21:37:39.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:39.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj148 td/osd-backfill-space/datafile 2020-11-16T21:37:39.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:39.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj149 td/osd-backfill-space/datafile 2020-11-16T21:37:39.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:39.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj150 td/osd-backfill-space/datafile 2020-11-16T21:37:39.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:39.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj151 td/osd-backfill-space/datafile 2020-11-16T21:37:40.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:40.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj152 td/osd-backfill-space/datafile 2020-11-16T21:37:40.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:40.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj153 td/osd-backfill-space/datafile 2020-11-16T21:37:40.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:40.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj154 td/osd-backfill-space/datafile 2020-11-16T21:37:40.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:40.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj155 td/osd-backfill-space/datafile 2020-11-16T21:37:40.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:40.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj156 td/osd-backfill-space/datafile 2020-11-16T21:37:40.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:40.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj157 td/osd-backfill-space/datafile 2020-11-16T21:37:40.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:40.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj158 td/osd-backfill-space/datafile 2020-11-16T21:37:40.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:40.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj159 td/osd-backfill-space/datafile 2020-11-16T21:37:40.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:40.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj160 td/osd-backfill-space/datafile 2020-11-16T21:37:40.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:40.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj161 td/osd-backfill-space/datafile 2020-11-16T21:37:40.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:40.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj162 td/osd-backfill-space/datafile 2020-11-16T21:37:41.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj163 td/osd-backfill-space/datafile 2020-11-16T21:37:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj164 td/osd-backfill-space/datafile 2020-11-16T21:37:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj165 td/osd-backfill-space/datafile 2020-11-16T21:37:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj166 td/osd-backfill-space/datafile 2020-11-16T21:37:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj167 td/osd-backfill-space/datafile 2020-11-16T21:37:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj168 td/osd-backfill-space/datafile 2020-11-16T21:37:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj169 td/osd-backfill-space/datafile 2020-11-16T21:37:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj170 td/osd-backfill-space/datafile 2020-11-16T21:37:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj171 td/osd-backfill-space/datafile 2020-11-16T21:37:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj172 td/osd-backfill-space/datafile 2020-11-16T21:37:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj173 td/osd-backfill-space/datafile 2020-11-16T21:37:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj174 td/osd-backfill-space/datafile 2020-11-16T21:37:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj175 td/osd-backfill-space/datafile 2020-11-16T21:37:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj176 td/osd-backfill-space/datafile 2020-11-16T21:37:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj177 td/osd-backfill-space/datafile 2020-11-16T21:37:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj178 td/osd-backfill-space/datafile 2020-11-16T21:37:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj179 td/osd-backfill-space/datafile 2020-11-16T21:37:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj180 td/osd-backfill-space/datafile 2020-11-16T21:37:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj181 td/osd-backfill-space/datafile 2020-11-16T21:37:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj182 td/osd-backfill-space/datafile 2020-11-16T21:37:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj183 td/osd-backfill-space/datafile 2020-11-16T21:37:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj184 td/osd-backfill-space/datafile 2020-11-16T21:37:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj185 td/osd-backfill-space/datafile 2020-11-16T21:37:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj186 td/osd-backfill-space/datafile 2020-11-16T21:37:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj187 td/osd-backfill-space/datafile 2020-11-16T21:37:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj188 td/osd-backfill-space/datafile 2020-11-16T21:37:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj189 td/osd-backfill-space/datafile 2020-11-16T21:37:45.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj190 td/osd-backfill-space/datafile 2020-11-16T21:37:45.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj191 td/osd-backfill-space/datafile 2020-11-16T21:37:45.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj192 td/osd-backfill-space/datafile 2020-11-16T21:37:45.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj193 td/osd-backfill-space/datafile 2020-11-16T21:37:45.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj194 td/osd-backfill-space/datafile 2020-11-16T21:37:45.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj195 td/osd-backfill-space/datafile 2020-11-16T21:37:45.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj196 td/osd-backfill-space/datafile 2020-11-16T21:37:45.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj197 td/osd-backfill-space/datafile 2020-11-16T21:37:45.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj198 td/osd-backfill-space/datafile 2020-11-16T21:37:45.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj199 td/osd-backfill-space/datafile 2020-11-16T21:37:45.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:45.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj200 td/osd-backfill-space/datafile 2020-11-16T21:37:45.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:657: TEST_ec_backfill_simple: get_primary fillpool obj1 2020-11-16T21:37:45.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=fillpool 2020-11-16T21:37:45.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:37:45.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map fillpool obj1 2020-11-16T21:37:45.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:37:45.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:657: TEST_ec_backfill_simple: local fillosd=1 2020-11-16T21:37:45.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:658: TEST_ec_backfill_simple: expr 1 + 1 2020-11-16T21:37:45.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:658: TEST_ec_backfill_simple: osd=2 2020-11-16T21:37:45.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:659: TEST_ec_backfill_simple: '[' 2 = 6 ']' 2020-11-16T21:37:45.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:663: TEST_ec_backfill_simple: sleep 5 2020-11-16T21:37:50.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:664: TEST_ec_backfill_simple: kill_daemon td/osd-backfill-space/osd.1.pid TERM 2020-11-16T21:37:50.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/osd-backfill-space/osd.1.pid 2020-11-16T21:37:50.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=451370 2020-11-16T21:37:50.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2020-11-16T21:37:50.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2020-11-16T21:37:50.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2020-11-16T21:37:50.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2020-11-16T21:37:50.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 451370 2020-11-16T21:37:50.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2020-11-16T21:37:50.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2020-11-16T21:37:50.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2020-11-16T21:37:50.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2020-11-16T21:37:50.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 451370 2020-11-16T21:37:50.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2020-11-16T21:37:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2020-11-16T21:37:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2020-11-16T21:37:50.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:665: TEST_ec_backfill_simple: ceph osd out osd.1 2020-11-16T21:37:50.839 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. 2020-11-16T21:37:50.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:666: TEST_ec_backfill_simple: sleep 2 2020-11-16T21:37:52.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:667: TEST_ec_backfill_simple: ceph osd erasure-code-profile set ec-profile k=3 m=2 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure 2020-11-16T21:37:53.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:669: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:37:53.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:669: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:37:53.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:671: TEST_ec_backfill_simple: ceph osd pool create test1 1 1 erasure ec-profile 2020-11-16T21:37:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' already exists 2020-11-16T21:37:54.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:676: TEST_ec_backfill_simple: sleep 5 2020-11-16T21:37:59.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:678: TEST_ec_backfill_simple: ceph pg dump pgs 2020-11-16T21:37:59.510 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:37:59.510 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T21:37:55.712764+0000 0'0 44:12 [3,5,4,0,2] 3 [3,5,4,0,2] 3 0'0 2020-11-16T21:37:53.935898+0000 0'0 2020-11-16T21:37:53.935898+0000 0 2020-11-16T21:37:59.510 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 0 0 0 2662400 0 0 100 100 stale+active+clean 2020-11-16T21:37:24.960033+0000 38'200 38:219 [1] 1 [1] 1 0'0 2020-11-16T21:37:22.573305+0000 0'0 2020-11-16T21:37:22.573305+0000 0 2020-11-16T21:37:59.511 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:37:59.511 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:37:59.512 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:37:59.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:680: TEST_ec_backfill_simple: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=18 2020-11-16T21:37:59.528 INFO:tasks.workunit.client.0.smithi189.stderr:18+0 records in 2020-11-16T21:37:59.528 INFO:tasks.workunit.client.0.smithi189.stderr:18+0 records out 2020-11-16T21:37:59.528 INFO:tasks.workunit.client.0.smithi189.stderr:18432 bytes (18 kB, 18 KiB) copied, 0.000296386 s, 62.2 MB/s 2020-11-16T21:37:59.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: seq 1 200 2020-11-16T21:37:59.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:59.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:37:59.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:37:59.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj1 td/osd-backfill-space/datafile 2020-11-16T21:37:59.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:59.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:37:59.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:37:59.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj2 td/osd-backfill-space/datafile 2020-11-16T21:37:59.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:37:59.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:37:59.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:37:59.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj3 td/osd-backfill-space/datafile 2020-11-16T21:38:00.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:00.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:00.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:00.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj4 td/osd-backfill-space/datafile 2020-11-16T21:38:00.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:00.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:00.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:00.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj5 td/osd-backfill-space/datafile 2020-11-16T21:38:00.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:00.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:00.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:00.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj6 td/osd-backfill-space/datafile 2020-11-16T21:38:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:00.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:00.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:00.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj7 td/osd-backfill-space/datafile 2020-11-16T21:38:01.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:01.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:01.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:01.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj8 td/osd-backfill-space/datafile 2020-11-16T21:38:01.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:01.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:01.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:01.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj9 td/osd-backfill-space/datafile 2020-11-16T21:38:01.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:01.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:01.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:01.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj10 td/osd-backfill-space/datafile 2020-11-16T21:38:01.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:01.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:01.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:01.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj11 td/osd-backfill-space/datafile 2020-11-16T21:38:02.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:02.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:02.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:02.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj12 td/osd-backfill-space/datafile 2020-11-16T21:38:02.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:02.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:02.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:02.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj13 td/osd-backfill-space/datafile 2020-11-16T21:38:02.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:02.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:02.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:02.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj14 td/osd-backfill-space/datafile 2020-11-16T21:38:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:02.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:02.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:02.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj15 td/osd-backfill-space/datafile 2020-11-16T21:38:03.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:03.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:03.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:03.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj16 td/osd-backfill-space/datafile 2020-11-16T21:38:03.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:03.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:03.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:03.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj17 td/osd-backfill-space/datafile 2020-11-16T21:38:03.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:03.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:03.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:03.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj18 td/osd-backfill-space/datafile 2020-11-16T21:38:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:03.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj19 td/osd-backfill-space/datafile 2020-11-16T21:38:04.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:04.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:04.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:04.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj20 td/osd-backfill-space/datafile 2020-11-16T21:38:04.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:04.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:04.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:04.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj21 td/osd-backfill-space/datafile 2020-11-16T21:38:04.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:04.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:04.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:04.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj22 td/osd-backfill-space/datafile 2020-11-16T21:38:04.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:04.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:04.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:04.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj23 td/osd-backfill-space/datafile 2020-11-16T21:38:05.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:05.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:05.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:05.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj24 td/osd-backfill-space/datafile 2020-11-16T21:38:05.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:05.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:05.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:05.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj25 td/osd-backfill-space/datafile 2020-11-16T21:38:05.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:05.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:05.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:05.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj26 td/osd-backfill-space/datafile 2020-11-16T21:38:05.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:05.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:05.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:05.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj27 td/osd-backfill-space/datafile 2020-11-16T21:38:06.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:06.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:06.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:06.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj28 td/osd-backfill-space/datafile 2020-11-16T21:38:06.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:06.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:06.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:06.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj29 td/osd-backfill-space/datafile 2020-11-16T21:38:06.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:06.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:06.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:06.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj30 td/osd-backfill-space/datafile 2020-11-16T21:38:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:07.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj31 td/osd-backfill-space/datafile 2020-11-16T21:38:07.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:07.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:07.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:07.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj32 td/osd-backfill-space/datafile 2020-11-16T21:38:07.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:07.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:07.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:07.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj33 td/osd-backfill-space/datafile 2020-11-16T21:38:07.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:07.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:07.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:07.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj34 td/osd-backfill-space/datafile 2020-11-16T21:38:08.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:08.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:08.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:08.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj35 td/osd-backfill-space/datafile 2020-11-16T21:38:08.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:08.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:08.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:08.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj36 td/osd-backfill-space/datafile 2020-11-16T21:38:08.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:08.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:08.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:08.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj37 td/osd-backfill-space/datafile 2020-11-16T21:38:08.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:08.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:08.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:08.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj38 td/osd-backfill-space/datafile 2020-11-16T21:38:09.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:09.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:09.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:09.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj39 td/osd-backfill-space/datafile 2020-11-16T21:38:09.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:09.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:09.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:09.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj40 td/osd-backfill-space/datafile 2020-11-16T21:38:09.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:09.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj41 td/osd-backfill-space/datafile 2020-11-16T21:38:09.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:09.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:09.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:09.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj42 td/osd-backfill-space/datafile 2020-11-16T21:38:10.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:10.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:10.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:10.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj43 td/osd-backfill-space/datafile 2020-11-16T21:38:10.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:10.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:10.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:10.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj44 td/osd-backfill-space/datafile 2020-11-16T21:38:10.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:10.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:10.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:10.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj45 td/osd-backfill-space/datafile 2020-11-16T21:38:10.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:10.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:10.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj46 td/osd-backfill-space/datafile 2020-11-16T21:38:11.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:11.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:11.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:11.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj47 td/osd-backfill-space/datafile 2020-11-16T21:38:11.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:11.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:11.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:11.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj48 td/osd-backfill-space/datafile 2020-11-16T21:38:11.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:11.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:11.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:11.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj49 td/osd-backfill-space/datafile 2020-11-16T21:38:11.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:11.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:11.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:11.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj50 td/osd-backfill-space/datafile 2020-11-16T21:38:12.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:12.185 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:12.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:12.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj51 td/osd-backfill-space/datafile 2020-11-16T21:38:12.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:12.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:12.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:12.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj52 td/osd-backfill-space/datafile 2020-11-16T21:38:12.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:12.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:12.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj53 td/osd-backfill-space/datafile 2020-11-16T21:38:12.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:12.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:12.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:12.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj54 td/osd-backfill-space/datafile 2020-11-16T21:38:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:13.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:13.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:13.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj55 td/osd-backfill-space/datafile 2020-11-16T21:38:13.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:13.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:13.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:13.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj56 td/osd-backfill-space/datafile 2020-11-16T21:38:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:13.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj57 td/osd-backfill-space/datafile 2020-11-16T21:38:13.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:13.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:13.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:13.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj58 td/osd-backfill-space/datafile 2020-11-16T21:38:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:14.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj59 td/osd-backfill-space/datafile 2020-11-16T21:38:14.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:14.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:14.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:14.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj60 td/osd-backfill-space/datafile 2020-11-16T21:38:14.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:14.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj61 td/osd-backfill-space/datafile 2020-11-16T21:38:14.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:14.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:14.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:14.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj62 td/osd-backfill-space/datafile 2020-11-16T21:38:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:15.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj63 td/osd-backfill-space/datafile 2020-11-16T21:38:15.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:15.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:15.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:15.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj64 td/osd-backfill-space/datafile 2020-11-16T21:38:15.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:15.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:15.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:15.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj65 td/osd-backfill-space/datafile 2020-11-16T21:38:15.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:15.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:15.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:15.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj66 td/osd-backfill-space/datafile 2020-11-16T21:38:16.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:16.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:16.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj67 td/osd-backfill-space/datafile 2020-11-16T21:38:16.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:16.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:16.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:16.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj68 td/osd-backfill-space/datafile 2020-11-16T21:38:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:16.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj69 td/osd-backfill-space/datafile 2020-11-16T21:38:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:16.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj70 td/osd-backfill-space/datafile 2020-11-16T21:38:17.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:17.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:17.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:17.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj71 td/osd-backfill-space/datafile 2020-11-16T21:38:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:17.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:17.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:17.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj72 td/osd-backfill-space/datafile 2020-11-16T21:38:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:17.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj73 td/osd-backfill-space/datafile 2020-11-16T21:38:17.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:17.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:17.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:17.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj74 td/osd-backfill-space/datafile 2020-11-16T21:38:18.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:18.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:18.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:18.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj75 td/osd-backfill-space/datafile 2020-11-16T21:38:18.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:18.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:18.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:18.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj76 td/osd-backfill-space/datafile 2020-11-16T21:38:18.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:18.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:18.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:18.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj77 td/osd-backfill-space/datafile 2020-11-16T21:38:18.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:18.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:18.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:18.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj78 td/osd-backfill-space/datafile 2020-11-16T21:38:18.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:18.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:18.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:18.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj79 td/osd-backfill-space/datafile 2020-11-16T21:38:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:19.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj80 td/osd-backfill-space/datafile 2020-11-16T21:38:19.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:19.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:19.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:19.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj81 td/osd-backfill-space/datafile 2020-11-16T21:38:19.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:19.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:19.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:19.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj82 td/osd-backfill-space/datafile 2020-11-16T21:38:19.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:19.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:19.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:19.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj83 td/osd-backfill-space/datafile 2020-11-16T21:38:20.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:20.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:20.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:20.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj84 td/osd-backfill-space/datafile 2020-11-16T21:38:20.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:20.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:20.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:20.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj85 td/osd-backfill-space/datafile 2020-11-16T21:38:20.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:20.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:20.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:20.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj86 td/osd-backfill-space/datafile 2020-11-16T21:38:20.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:20.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:20.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:20.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj87 td/osd-backfill-space/datafile 2020-11-16T21:38:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:21.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj88 td/osd-backfill-space/datafile 2020-11-16T21:38:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj89 td/osd-backfill-space/datafile 2020-11-16T21:38:25.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj90 td/osd-backfill-space/datafile 2020-11-16T21:38:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj91 td/osd-backfill-space/datafile 2020-11-16T21:38:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj92 td/osd-backfill-space/datafile 2020-11-16T21:38:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj93 td/osd-backfill-space/datafile 2020-11-16T21:38:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj94 td/osd-backfill-space/datafile 2020-11-16T21:38:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj95 td/osd-backfill-space/datafile 2020-11-16T21:38:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj96 td/osd-backfill-space/datafile 2020-11-16T21:38:25.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj97 td/osd-backfill-space/datafile 2020-11-16T21:38:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj98 td/osd-backfill-space/datafile 2020-11-16T21:38:25.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj99 td/osd-backfill-space/datafile 2020-11-16T21:38:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj100 td/osd-backfill-space/datafile 2020-11-16T21:38:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj101 td/osd-backfill-space/datafile 2020-11-16T21:38:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj102 td/osd-backfill-space/datafile 2020-11-16T21:38:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj103 td/osd-backfill-space/datafile 2020-11-16T21:38:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj104 td/osd-backfill-space/datafile 2020-11-16T21:38:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj105 td/osd-backfill-space/datafile 2020-11-16T21:38:25.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj106 td/osd-backfill-space/datafile 2020-11-16T21:38:25.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:25.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:25.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:25.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj107 td/osd-backfill-space/datafile 2020-11-16T21:38:26.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:26.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:26.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj108 td/osd-backfill-space/datafile 2020-11-16T21:38:26.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:26.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:26.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:26.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj109 td/osd-backfill-space/datafile 2020-11-16T21:38:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:26.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:26.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:26.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj110 td/osd-backfill-space/datafile 2020-11-16T21:38:26.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:26.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:26.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:26.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj111 td/osd-backfill-space/datafile 2020-11-16T21:38:27.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:27.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:27.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:27.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj112 td/osd-backfill-space/datafile 2020-11-16T21:38:27.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:27.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:27.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:27.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj113 td/osd-backfill-space/datafile 2020-11-16T21:38:27.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:27.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:27.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:27.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj114 td/osd-backfill-space/datafile 2020-11-16T21:38:27.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:27.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:27.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:27.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj115 td/osd-backfill-space/datafile 2020-11-16T21:38:28.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:28.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:28.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:28.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj116 td/osd-backfill-space/datafile 2020-11-16T21:38:28.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:28.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:28.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:28.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj117 td/osd-backfill-space/datafile 2020-11-16T21:38:28.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:28.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:28.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:28.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj118 td/osd-backfill-space/datafile 2020-11-16T21:38:28.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:28.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:28.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:28.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj119 td/osd-backfill-space/datafile 2020-11-16T21:38:29.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:29.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:29.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:29.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj120 td/osd-backfill-space/datafile 2020-11-16T21:38:29.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:29.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:29.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:29.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj121 td/osd-backfill-space/datafile 2020-11-16T21:38:29.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:29.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:29.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:29.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj122 td/osd-backfill-space/datafile 2020-11-16T21:38:29.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:29.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:29.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:29.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj123 td/osd-backfill-space/datafile 2020-11-16T21:38:30.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:30.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:30.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:30.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj124 td/osd-backfill-space/datafile 2020-11-16T21:38:30.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:30.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj125 td/osd-backfill-space/datafile 2020-11-16T21:38:30.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:30.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:30.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:30.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj126 td/osd-backfill-space/datafile 2020-11-16T21:38:30.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:30.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:30.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:30.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj127 td/osd-backfill-space/datafile 2020-11-16T21:38:31.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:31.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:31.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:31.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj128 td/osd-backfill-space/datafile 2020-11-16T21:38:31.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:31.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:31.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:31.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj129 td/osd-backfill-space/datafile 2020-11-16T21:38:31.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:31.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:31.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:31.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj130 td/osd-backfill-space/datafile 2020-11-16T21:38:31.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:31.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:31.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:31.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj131 td/osd-backfill-space/datafile 2020-11-16T21:38:32.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:32.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:32.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:32.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj132 td/osd-backfill-space/datafile 2020-11-16T21:38:32.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:32.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj133 td/osd-backfill-space/datafile 2020-11-16T21:38:32.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:32.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:32.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:32.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj134 td/osd-backfill-space/datafile 2020-11-16T21:38:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:32.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:32.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:32.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj135 td/osd-backfill-space/datafile 2020-11-16T21:38:33.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:33.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:33.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:33.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj136 td/osd-backfill-space/datafile 2020-11-16T21:38:33.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:33.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:33.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:33.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj137 td/osd-backfill-space/datafile 2020-11-16T21:38:33.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:33.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:33.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:33.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj138 td/osd-backfill-space/datafile 2020-11-16T21:38:33.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:33.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:33.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:33.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj139 td/osd-backfill-space/datafile 2020-11-16T21:38:34.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:34.078 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:34.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:34.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj140 td/osd-backfill-space/datafile 2020-11-16T21:38:34.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:34.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj141 td/osd-backfill-space/datafile 2020-11-16T21:38:34.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:34.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:34.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:34.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj142 td/osd-backfill-space/datafile 2020-11-16T21:38:34.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:34.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:34.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:34.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj143 td/osd-backfill-space/datafile 2020-11-16T21:38:35.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:35.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:35.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:35.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj144 td/osd-backfill-space/datafile 2020-11-16T21:38:35.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:35.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:35.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:35.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj145 td/osd-backfill-space/datafile 2020-11-16T21:38:35.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:35.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:35.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:35.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj146 td/osd-backfill-space/datafile 2020-11-16T21:38:35.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:35.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:35.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj147 td/osd-backfill-space/datafile 2020-11-16T21:38:36.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:36.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:36.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:36.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj148 td/osd-backfill-space/datafile 2020-11-16T21:38:36.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:36.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:36.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:36.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj149 td/osd-backfill-space/datafile 2020-11-16T21:38:36.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:36.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:36.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:36.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj150 td/osd-backfill-space/datafile 2020-11-16T21:38:36.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:36.778 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:36.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:36.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj151 td/osd-backfill-space/datafile 2020-11-16T21:38:37.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:37.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:37.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:37.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj152 td/osd-backfill-space/datafile 2020-11-16T21:38:37.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:37.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:37.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:37.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj153 td/osd-backfill-space/datafile 2020-11-16T21:38:37.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:37.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:37.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:37.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj154 td/osd-backfill-space/datafile 2020-11-16T21:38:37.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:37.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:37.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:37.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj155 td/osd-backfill-space/datafile 2020-11-16T21:38:38.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:38.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:38.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:38.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj156 td/osd-backfill-space/datafile 2020-11-16T21:38:38.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj157 td/osd-backfill-space/datafile 2020-11-16T21:38:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj158 td/osd-backfill-space/datafile 2020-11-16T21:38:42.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj159 td/osd-backfill-space/datafile 2020-11-16T21:38:42.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj160 td/osd-backfill-space/datafile 2020-11-16T21:38:42.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj161 td/osd-backfill-space/datafile 2020-11-16T21:38:42.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj162 td/osd-backfill-space/datafile 2020-11-16T21:38:42.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj163 td/osd-backfill-space/datafile 2020-11-16T21:38:42.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj164 td/osd-backfill-space/datafile 2020-11-16T21:38:42.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj165 td/osd-backfill-space/datafile 2020-11-16T21:38:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj166 td/osd-backfill-space/datafile 2020-11-16T21:38:42.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj167 td/osd-backfill-space/datafile 2020-11-16T21:38:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj168 td/osd-backfill-space/datafile 2020-11-16T21:38:42.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.602 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj169 td/osd-backfill-space/datafile 2020-11-16T21:38:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj170 td/osd-backfill-space/datafile 2020-11-16T21:38:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj171 td/osd-backfill-space/datafile 2020-11-16T21:38:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj172 td/osd-backfill-space/datafile 2020-11-16T21:38:42.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj173 td/osd-backfill-space/datafile 2020-11-16T21:38:42.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj174 td/osd-backfill-space/datafile 2020-11-16T21:38:42.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:42.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:42.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:42.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj175 td/osd-backfill-space/datafile 2020-11-16T21:38:43.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:43.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:43.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:43.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj176 td/osd-backfill-space/datafile 2020-11-16T21:38:43.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:43.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:43.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:43.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj177 td/osd-backfill-space/datafile 2020-11-16T21:38:43.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:43.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:43.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:43.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj178 td/osd-backfill-space/datafile 2020-11-16T21:38:43.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:43.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj179 td/osd-backfill-space/datafile 2020-11-16T21:38:44.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:44.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:44.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:44.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj180 td/osd-backfill-space/datafile 2020-11-16T21:38:44.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:44.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:44.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:44.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj181 td/osd-backfill-space/datafile 2020-11-16T21:38:44.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:44.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:44.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:44.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj182 td/osd-backfill-space/datafile 2020-11-16T21:38:44.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:44.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:44.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:44.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj183 td/osd-backfill-space/datafile 2020-11-16T21:38:45.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:45.098 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:45.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:45.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj184 td/osd-backfill-space/datafile 2020-11-16T21:38:45.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:45.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:45.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:45.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj185 td/osd-backfill-space/datafile 2020-11-16T21:38:45.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:45.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:45.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:45.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj186 td/osd-backfill-space/datafile 2020-11-16T21:38:45.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:45.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:45.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:45.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj187 td/osd-backfill-space/datafile 2020-11-16T21:38:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:46.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:46.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:46.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj188 td/osd-backfill-space/datafile 2020-11-16T21:38:46.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:46.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:46.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:46.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj189 td/osd-backfill-space/datafile 2020-11-16T21:38:46.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:46.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:46.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:46.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj190 td/osd-backfill-space/datafile 2020-11-16T21:38:46.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:46.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj191 td/osd-backfill-space/datafile 2020-11-16T21:38:47.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:47.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:47.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:47.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj192 td/osd-backfill-space/datafile 2020-11-16T21:38:47.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:47.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:47.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj193 td/osd-backfill-space/datafile 2020-11-16T21:38:47.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:47.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:47.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:47.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj194 td/osd-backfill-space/datafile 2020-11-16T21:38:47.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:47.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:47.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:47.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj195 td/osd-backfill-space/datafile 2020-11-16T21:38:48.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:48.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:48.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:48.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj196 td/osd-backfill-space/datafile 2020-11-16T21:38:48.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:48.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:48.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:48.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj197 td/osd-backfill-space/datafile 2020-11-16T21:38:48.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:48.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:48.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:48.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj198 td/osd-backfill-space/datafile 2020-11-16T21:38:48.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:48.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:48.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:48.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj199 td/osd-backfill-space/datafile 2020-11-16T21:38:48.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2020-11-16T21:38:48.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:38:48.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2020-11-16T21:38:48.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj200 td/osd-backfill-space/datafile 2020-11-16T21:38:49.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:689: TEST_ec_backfill_simple: kill_daemon td/osd-backfill-space/osd.2.pid TERM 2020-11-16T21:38:49.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/osd-backfill-space/osd.2.pid 2020-11-16T21:38:49.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=452364 2020-11-16T21:38:49.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2020-11-16T21:38:49.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2020-11-16T21:38:49.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2020-11-16T21:38:49.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2020-11-16T21:38:49.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 452364 2020-11-16T21:38:49.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2020-11-16T21:38:49.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2020-11-16T21:38:49.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2020-11-16T21:38:49.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2020-11-16T21:38:49.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 452364 2020-11-16T21:38:49.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2020-11-16T21:38:49.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2020-11-16T21:38:49.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2020-11-16T21:38:49.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2020-11-16T21:38:49.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 452364 2020-11-16T21:38:49.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2020-11-16T21:38:49.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2020-11-16T21:38:49.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2020-11-16T21:38:49.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:690: TEST_ec_backfill_simple: ceph osd out osd.2 2020-11-16T21:38:50.182 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already out. 2020-11-16T21:38:50.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:692: TEST_ec_backfill_simple: activate_osd td/osd-backfill-space 1 2020-11-16T21:38:50.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2020-11-16T21:38:50.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T21:38:50.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T21:38:50.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T21:38:50.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/1 2020-11-16T21:38:50.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2020-11-16T21:38:50.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:38:50.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:38:50.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:38:50.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2020-11-16T21:38:50.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2020-11-16T21:38:50.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T21:38:50.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T21:38:50.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2020-11-16T21:38:50.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T21:38:50.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:38:50.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:38:50.195 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:38:50.195 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:38:50.195 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:38:50.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:38:50.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' 2020-11-16T21:38:50.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:38:50.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2020-11-16T21:38:50.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2020-11-16T21:38:50.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:38:50.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:38:50.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:38:50.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T21:38:50.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T21:38:50.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/1 2020-11-16T21:38:50.197 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:38:50.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T21:38:50.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=f5b90eb9-f99a-41dc-aff8-50e583582af1 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.137323/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:38:50.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/1/whoami 2020-11-16T21:38:50.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T21:38:50.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T21:38:50.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T21:38:50.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T21:38:50.229 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:38:50.234+0000 7f1dfeb0ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:38:50.231 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:38:50.238+0000 7f1dfeb0ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:38:50.232 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:38:50.238+0000 7f1dfeb0ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:38:50.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T21:38:50.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:38:50.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:38:50.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:38:50.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:38:50.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:38:50.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:38:50.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:38:50.625 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:38:50.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:38:50.785 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:38:50.790+0000 7f1dfeb0ed80 -1 Falling back to public interface 2020-11-16T21:38:51.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:38:52.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:38:52.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:38:52.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:38:52.036 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:38:52.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:38:52.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:38:52.320 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:38:52.326+0000 7f1dfeb0ed80 -1 osd.1 38 log_to_monitors {default=true} 2020-11-16T21:38:52.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:38:53.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:38:53.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:38:53.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:38:53.458 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:38:53.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:38:53.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:38:53.867 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up out weight 0 up_from 49 up_thru 50 down_at 39 last_clean_interval [10,38) [v2:127.0.0.1:6810/464235,v1:127.0.0.1:6811/464235] [v2:127.0.0.1:6812/464235,v1:127.0.0.1:6813/464235] exists,up fe40f1f9-16be-4f31-bcd2-e87705feeab2 2020-11-16T21:38:53.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:38:53.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:38:53.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:38:53.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:693: TEST_ec_backfill_simple: ceph osd in osd.1 2020-11-16T21:38:54.437 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already in. 2020-11-16T21:38:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:694: TEST_ec_backfill_simple: sleep 30 2020-11-16T21:39:24.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:696: TEST_ec_backfill_simple: ceph pg dump pgs 2020-11-16T21:39:24.827 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:39:24.827 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 200 0 200 200 0 3686400 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2020-11-16T21:38:55.866954+0000 45'200 55:244 [3,5,1,0,4] 3 [3,5,4,0,NONE] 3 0'0 2020-11-16T21:37:53.935898+0000 0'0 2020-11-16T21:37:53.935898+0000 0 2020-11-16T21:39:24.828 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 0 0 0 2662400 0 0 100 100 active+clean 2020-11-16T21:38:55.129118+0000 38'200 54:239 [1] 1 [1] 1 0'0 2020-11-16T21:37:22.573305+0000 0'0 2020-11-16T21:37:22.573305+0000 0 2020-11-16T21:39:24.828 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:39:24.828 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:39:24.830 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:39:24.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:698: TEST_ec_backfill_simple: wait_for_not_backfilling 240 2020-11-16T21:39:24.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2020-11-16T21:39:24.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2020-11-16T21:39:24.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2020-11-16T21:39:24.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:39:24.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:39:24.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:39:24.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2020-11-16T21:39:24.842 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:39:24.843 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:39:24.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:39:24.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:39:24.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:39:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:39:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:39:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:39:25.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:39:25.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:39:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:39:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:39:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:39:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:39:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:39:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:39:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:39:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:25.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:39:25.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836516 2020-11-16T21:39:25.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836516 2020-11-16T21:39:25.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836516' 2020-11-16T21:39:25.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:25.707 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:39:25.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397513 2020-11-16T21:39:25.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397513 2020-11-16T21:39:25.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836516 1-210453397513' 2020-11-16T21:39:25.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:25.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:39:26.022 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.2 2020-11-16T21:39:26.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T21:39:26.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T21:39:26.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T21:39:26.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:26.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:39:26.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378655 2020-11-16T21:39:26.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378655 2020-11-16T21:39:26.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836516 1-210453397513 3-81604378655' 2020-11-16T21:39:26.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:26.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:39:26.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215133 2020-11-16T21:39:26.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215133 2020-11-16T21:39:26.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836516 1-210453397513 3-81604378655 4-103079215133' 2020-11-16T21:39:26.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:26.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:39:26.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051612 2020-11-16T21:39:26.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051612 2020-11-16T21:39:26.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836516 1-210453397513 3-81604378655 4-103079215133 5-124554051612' 2020-11-16T21:39:26.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:39:26.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836516 2020-11-16T21:39:26.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:39:26.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:39:26.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836516 2020-11-16T21:39:26.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:39:26.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836516 2020-11-16T21:39:26.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2020-11-16T21:39:26.541 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836516 2020-11-16T21:39:26.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:39:26.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836514 -lt 21474836516 2020-11-16T21:39:26.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:39:27.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:39:27.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:39:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836516 -lt 21474836516 2020-11-16T21:39:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:39:28.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397513 2020-11-16T21:39:28.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:39:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:39:28.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397513 2020-11-16T21:39:28.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:39:28.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397513 2020-11-16T21:39:28.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397513' 2020-11-16T21:39:28.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:39:28.349 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 210453397513 2020-11-16T21:39:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397514 -lt 210453397513 2020-11-16T21:39:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:39:28.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:39:28.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378655 2020-11-16T21:39:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:39:28.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378655 2020-11-16T21:39:28.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:39:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378655 2020-11-16T21:39:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378655' 2020-11-16T21:39:28.732 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378655 2020-11-16T21:39:28.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:39:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378655 -lt 81604378655 2020-11-16T21:39:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:39:29.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215133 2020-11-16T21:39:29.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:39:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:39:29.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215133 2020-11-16T21:39:29.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:39:29.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215133 2020-11-16T21:39:29.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215133' 2020-11-16T21:39:29.119 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215133 2020-11-16T21:39:29.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:39:29.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215133 -lt 103079215133 2020-11-16T21:39:29.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:39:29.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051612 2020-11-16T21:39:29.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:39:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:39:29.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051612 2020-11-16T21:39:29.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:39:29.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051612 2020-11-16T21:39:29.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051612' 2020-11-16T21:39:29.495 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051612 2020-11-16T21:39:29.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:39:29.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051612 -lt 124554051612 2020-11-16T21:39:29.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:39:29.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:39:29.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:39:30.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2020-11-16T21:39:30.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:39:30.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2020-11-16T21:39:30.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2020-11-16T21:39:30.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:39:30.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2020-11-16T21:39:30.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:39:30.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2020-11-16T21:39:30.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:39:30.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:39:30.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:39:30.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:39:30.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:699: TEST_ec_backfill_simple: wait_for_not_activating 60 2020-11-16T21:39:30.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2020-11-16T21:39:30.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2020-11-16T21:39:30.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2020-11-16T21:39:30.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2020-11-16T21:39:30.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2020-11-16T21:39:30.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2020-11-16T21:39:30.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2020-11-16T21:39:30.775 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:39:30.775 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:39:30.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:39:30.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:39:30.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:39:30.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2020-11-16T21:39:30.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2020-11-16T21:39:30.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2020-11-16T21:39:30.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:39:30.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:39:31.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:39:31.251 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:39:31.252 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:39:31.252 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:39:31.252 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:39:31.252 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:39:31.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:39:31.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:31.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:39:31.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836519 2020-11-16T21:39:31.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836519 2020-11-16T21:39:31.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519' 2020-11-16T21:39:31.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:31.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:39:31.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397516 2020-11-16T21:39:31.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397516 2020-11-16T21:39:31.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-210453397516' 2020-11-16T21:39:31.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:31.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:39:31.706 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.2 2020-11-16T21:39:31.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T21:39:31.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T21:39:31.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T21:39:31.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:31.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:39:31.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378658 2020-11-16T21:39:31.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378658 2020-11-16T21:39:31.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-210453397516 3-81604378658' 2020-11-16T21:39:31.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:31.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:39:32.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215136 2020-11-16T21:39:32.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215136 2020-11-16T21:39:32.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-210453397516 3-81604378658 4-103079215136' 2020-11-16T21:39:32.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:39:32.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:39:32.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051615 2020-11-16T21:39:32.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051615 2020-11-16T21:39:32.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-210453397516 3-81604378658 4-103079215136 5-124554051615' 2020-11-16T21:39:32.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:39:32.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836519 2020-11-16T21:39:32.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:39:32.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:39:32.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836519 2020-11-16T21:39:32.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:39:32.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836519 2020-11-16T21:39:32.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2020-11-16T21:39:32.251 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836519 2020-11-16T21:39:32.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:39:32.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836519 -lt 21474836519 2020-11-16T21:39:32.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:39:32.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:39:32.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397516 2020-11-16T21:39:32.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:39:32.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397516 2020-11-16T21:39:32.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:39:32.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397516 2020-11-16T21:39:32.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397516' 2020-11-16T21:39:32.656 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 210453397516 2020-11-16T21:39:32.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:39:33.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397514 -lt 210453397516 2020-11-16T21:39:33.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:39:34.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:39:34.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:39:34.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397517 -lt 210453397516 2020-11-16T21:39:34.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:39:34.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378658 2020-11-16T21:39:34.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:39:34.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:39:34.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378658 2020-11-16T21:39:34.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:39:34.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378658 2020-11-16T21:39:34.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378658' 2020-11-16T21:39:34.462 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378658 2020-11-16T21:39:34.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:39:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378659 -lt 81604378658 2020-11-16T21:39:34.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:39:34.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215136 2020-11-16T21:39:34.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:39:34.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:39:34.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215136 2020-11-16T21:39:34.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:39:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215136 2020-11-16T21:39:34.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215136' 2020-11-16T21:39:34.849 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215136 2020-11-16T21:39:34.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:39:35.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215137 -lt 103079215136 2020-11-16T21:39:35.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:39:35.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051615 2020-11-16T21:39:35.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:39:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:39:35.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051615 2020-11-16T21:39:35.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:39:35.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051615 2020-11-16T21:39:35.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051615' 2020-11-16T21:39:35.209 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051615 2020-11-16T21:39:35.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:39:35.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051615 -lt 124554051615 2020-11-16T21:39:35.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2020-11-16T21:39:35.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:39:35.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:39:36.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2020-11-16T21:39:36.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2020-11-16T21:39:36.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2020-11-16T21:39:36.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2020-11-16T21:39:36.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2020-11-16T21:39:36.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2020-11-16T21:39:36.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2020-11-16T21:39:36.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2020-11-16T21:39:36.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2020-11-16T21:39:36.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2020-11-16T21:39:36.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2020-11-16T21:39:36.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2020-11-16T21:39:36.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:701: TEST_ec_backfill_simple: ceph pg dump pgs 2020-11-16T21:39:36.865 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:39:36.865 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 200 0 200 200 0 3686400 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2020-11-16T21:39:25.869062+0000 45'200 55:248 [3,5,1,0,4] 3 [3,5,4,0,NONE] 3 0'0 2020-11-16T21:37:53.935898+0000 0'0 2020-11-16T21:37:53.935898+0000 0 2020-11-16T21:39:36.865 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 0 0 0 2662400 0 0 100 100 active+clean 2020-11-16T21:38:55.129118+0000 38'200 54:239 [1] 1 [1] 1 0'0 2020-11-16T21:37:22.573305+0000 0'0 2020-11-16T21:37:22.573305+0000 0 2020-11-16T21:39:36.865 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:39:36.865 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:39:36.867 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:39:36.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:703: TEST_ec_backfill_simple: ERRORS=0 2020-11-16T21:39:36.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:704: TEST_ec_backfill_simple: ceph pg dump pgs 2020-11-16T21:39:36.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:704: TEST_ec_backfill_simple: grep -v '^1.0' 2020-11-16T21:39:36.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:704: TEST_ec_backfill_simple: grep +backfill_toofull 2020-11-16T21:39:36.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:704: TEST_ec_backfill_simple: wc -l 2020-11-16T21:39:37.223 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:39:37.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:704: TEST_ec_backfill_simple: '[' 1 '!=' 1 ']' 2020-11-16T21:39:37.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:709: TEST_ec_backfill_simple: '[' 0 '!=' 0 ']' 2020-11-16T21:39:37.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:714: TEST_ec_backfill_simple: delete_pool fillpool 2020-11-16T21:39:37.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=fillpool 2020-11-16T21:39:37.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete fillpool fillpool --yes-i-really-really-mean-it 2020-11-16T21:39:37.748 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fillpool' does not exist 2020-11-16T21:39:37.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:715: TEST_ec_backfill_simple: seq 1 1 2020-11-16T21:39:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:715: TEST_ec_backfill_simple: for i in $(seq 1 $pools) 2020-11-16T21:39:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:717: TEST_ec_backfill_simple: delete_pool test1 2020-11-16T21:39:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2020-11-16T21:39:37.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-11-16T21:39:38.371 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' does not exist 2020-11-16T21:39:38.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:719: TEST_ec_backfill_simple: kill_daemons td/osd-backfill-space 2020-11-16T21:39:38.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:39:38.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:39:38.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:39:38.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:39:38.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:39:38.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:39:38.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2020-11-16T21:39:38.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:39:38.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:39:38.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:39:38.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:39:38.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:39:38.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:39:38.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:39:38.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:39:38.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:39:38.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:39:38.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:39:38.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:39:38.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:39:38.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:39:38.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:39:38.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:39:38.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:39:38.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:39:38.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:39:38.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:39:38.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:39:38.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:39:38.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:39:38.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:39:38.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:39:38.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:39:38.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:39:38.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:39:38.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T21:39:38.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-backfill-space 0 2020-11-16T21:39:38.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2020-11-16T21:39:38.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T21:39:42.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2020-11-16T21:39:42.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:39:42.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:39:42.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:39:42.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:39:42.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:39:42.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:39:42.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:39:42.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:39:42.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:39:42.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:39:42.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:39:42.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:39:42.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:39:42.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:39:42.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:39:42.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:39:42.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:39:42.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T21:39:42.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2020-11-16T21:39:42.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:39:42.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:39:42.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137323 2020-11-16T21:39:42.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137323 2020-11-16T21:39:42.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:39:42.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:39:42.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T21:39:42.657 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T21:39:42.657 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T21:39:42.672 INFO:tasks.workunit:Running workunit osd/osd-backfill-stats.sh... 2020-11-16T21:39:42.673 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-backfill-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh 2020-11-16T21:39:42.732 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T21:39:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T21:39:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T21:39:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T21:39:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T21:39:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T21:39:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T21:39:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T21:39:42.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T21:39:42.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T21:39:42.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-backfill-stats 2020-11-16T21:39:42.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:21: run: local dir=td/osd-backfill-stats 2020-11-16T21:39:42.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:22: run: shift 2020-11-16T21:39:42.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-11-16T21:39:42.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-11-16T21:39:42.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:26: run: export CEPH_ARGS 2020-11-16T21:39:42.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:27: run: uuidgen 2020-11-16T21:39:42.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:27: run: CEPH_ARGS+='--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none ' 2020-11-16T21:39:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 ' 2020-11-16T21:39:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:39:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:30: run: export margin=10 2020-11-16T21:39:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:30: run: margin=10 2020-11-16T21:39:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:31: run: export objects=200 2020-11-16T21:39:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:31: run: objects=200 2020-11-16T21:39:42.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:32: run: export poolname=test 2020-11-16T21:39:42.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:32: run: poolname=test 2020-11-16T21:39:42.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:34: run: set 2020-11-16T21:39:42.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T21:39:42.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:34: run: local 'funcs=TEST_backfill_down_out 2020-11-16T21:39:42.755 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_ec_all_out 2020-11-16T21:39:42.755 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_ec_down_all_out 2020-11-16T21:39:42.756 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_ec_down_out 2020-11-16T21:39:42.756 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_ec_prim_out 2020-11-16T21:39:42.756 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_out 2020-11-16T21:39:42.756 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_out2 2020-11-16T21:39:42.756 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_remapped 2020-11-16T21:39:42.756 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_sizeup 2020-11-16T21:39:42.756 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_sizeup4_allout 2020-11-16T21:39:42.757 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_backfill_sizeup_out' 2020-11-16T21:39:42.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2020-11-16T21:39:42.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2020-11-16T21:39:42.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2020-11-16T21:39:42.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2020-11-16T21:39:42.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:39:42.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:39:42.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:39:42.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:39:42.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:39:42.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:39:42.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:39:42.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:39:42.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:39:42.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:39:42.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:39:42.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:39:42.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:39:42.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:39:42.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:39:42.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:39:42.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:39:42.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:39:42.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:39:42.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:39:42.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:39:42.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:39:42.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:39:42.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:39:42.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:39:42.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:39:42.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:39:42.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:39:42.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2020-11-16T21:39:42.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:39:42.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:39:42.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:39:42.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.465868 2020-11-16T21:39:42.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:39:42.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T21:39:42.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T21:39:42.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:39:42.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2020-11-16T21:39:42.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_down_out td/osd-backfill-stats 2020-11-16T21:39:42.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:278: TEST_backfill_down_out: local dir=td/osd-backfill-stats 2020-11-16T21:39:42.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:280: TEST_backfill_down_out: run_mon td/osd-backfill-stats a 2020-11-16T21:39:42.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2020-11-16T21:39:42.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:39:42.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:39:42.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:39:42.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2020-11-16T21:39:42.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2020-11-16T21:39:43.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:39:43.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:39:43.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:39:43.084 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:39:43.084 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:39:43.084 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:39:43.084 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:39:43.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:39:43.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:39:43.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:39:43.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:39:43.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:39:43.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:39:43.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:39:43.290 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:39:43.290 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:39:43.290 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:39:43.291 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:39:43.291 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:39:43.291 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:39:43.291 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:39:43.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:39:43.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get fsid 2020-11-16T21:39:43.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:39:43.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:39:43.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:39:43.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:39:43.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:39:43.405 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:39:43.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:39:43.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:39:43.406 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:39:43.406 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:39:43.406 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:39:43.407 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:39:43.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:39:43.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get mon_host 2020-11-16T21:39:43.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:281: TEST_backfill_down_out: run_mgr td/osd-backfill-stats x 2020-11-16T21:39:43.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2020-11-16T21:39:43.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:39:43.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:39:43.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:39:43.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2020-11-16T21:39:43.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:39:44.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:39:44.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:39:44.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:39:44.062 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:39:44.062 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:39:44.062 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:39:44.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:39:44.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:39:44.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:39:44.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:282: TEST_backfill_down_out: run_osd td/osd-backfill-stats 0 2020-11-16T21:39:44.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:39:44.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:39:44.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:39:44.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:39:44.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2020-11-16T21:39:44.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:39:44.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:39:44.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:39:44.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:39:44.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2020-11-16T21:39:44.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2020-11-16T21:39:44.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:39:44.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:39:44.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:39:44.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:39:44.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:39:44.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:39:44.100 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:39:44.100 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:39:44.100 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:39:44.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:39:44.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:39:44.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:39:44.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:39:44.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:39:44.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:39:44.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:39:44.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:39:44.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:39:44.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:39:44.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:39:44.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:39:44.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2020-11-16T21:39:44.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:39:44.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=188945dd-6460-4a49-9916-02f5c867400c 2020-11-16T21:39:44.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 188945dd-6460-4a49-9916-02f5c867400c' 2020-11-16T21:39:44.102 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 188945dd-6460-4a49-9916-02f5c867400c 2020-11-16T21:39:44.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:39:44.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCg8bJfKhBjBxAAfcUXqq2eMLFpWlvSo6zOzQ== 2020-11-16T21:39:44.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCg8bJfKhBjBxAAfcUXqq2eMLFpWlvSo6zOzQ=="}' 2020-11-16T21:39:44.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 188945dd-6460-4a49-9916-02f5c867400c -i td/osd-backfill-stats/0/new.json 2020-11-16T21:39:44.452 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:39:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2020-11-16T21:39:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCg8bJfKhBjBxAAfcUXqq2eMLFpWlvSo6zOzQ== --osd-uuid 188945dd-6460-4a49-9916-02f5c867400c 2020-11-16T21:39:44.497 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:44.501+0000 7f32d8f1ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:44.500 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:44.505+0000 7f32d8f1ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:44.502 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:44.505+0000 7f32d8f1ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:44.502 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:44.509+0000 7f32d8f1ed80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:39:44.503 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:44.509+0000 7f32d8f1ed80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:39:44.503 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:44.509+0000 7f32d8f1ed80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:39:44.566 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:44.569+0000 7f32d8f1ed80 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2020-11-16T21:39:49.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2020-11-16T21:39:49.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:39:49.174 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:39:49.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:39:49.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:39:49.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:39:49.677 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:39:49.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:39:49.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:39:49.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:39:49.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:39:49.707 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:49.713+0000 7f1f3062bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:49.709 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:49.713+0000 7f1f3062bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:49.710 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:49.717+0000 7f1f3062bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:50.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:39:50.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:39:50.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:39:50.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:39:50.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:39:50.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:39:50.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:39:50.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:39:50.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:39:50.070 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:39:50.263 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:50.269+0000 7f1f3062bd80 -1 Falling back to public interface 2020-11-16T21:39:50.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:39:50.906 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:50.909+0000 7f1f3062bd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:39:51.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:39:51.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:39:51.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:39:51.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:39:51.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:39:51.458 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:39:51.882 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/466721,v1:127.0.0.1:6803/466721] [v2:127.0.0.1:6804/466721,v1:127.0.0.1:6805/466721] exists,up 188945dd-6460-4a49-9916-02f5c867400c 2020-11-16T21:39:51.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:39:51.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:39:51.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:39:51.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:283: TEST_backfill_down_out: run_osd td/osd-backfill-stats 1 2020-11-16T21:39:51.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:39:51.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:39:51.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:39:51.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:39:51.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2020-11-16T21:39:51.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:39:51.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:39:51.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:39:51.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:39:51.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2020-11-16T21:39:51.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2020-11-16T21:39:51.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:39:51.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:39:51.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:39:51.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:39:51.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:39:51.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:39:51.887 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:39:51.887 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:39:51.888 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:39:51.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:39:51.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:39:51.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:39:51.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:39:51.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:39:51.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:39:51.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:39:51.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:39:51.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:39:51.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:39:51.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:39:51.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:39:51.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2020-11-16T21:39:51.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:39:51.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ba57606d-b86e-4bf6-bc03-0fddbb644c57 2020-11-16T21:39:51.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 ba57606d-b86e-4bf6-bc03-0fddbb644c57' 2020-11-16T21:39:51.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:39:51.890 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 ba57606d-b86e-4bf6-bc03-0fddbb644c57 2020-11-16T21:39:51.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCn8bJf2fugNhAAMMAXoFmG3Sbbr2czbgcAtw== 2020-11-16T21:39:51.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCn8bJf2fugNhAAMMAXoFmG3Sbbr2czbgcAtw=="}' 2020-11-16T21:39:51.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ba57606d-b86e-4bf6-bc03-0fddbb644c57 -i td/osd-backfill-stats/1/new.json 2020-11-16T21:39:52.452 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:39:52.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2020-11-16T21:39:52.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCn8bJf2fugNhAAMMAXoFmG3Sbbr2czbgcAtw== --osd-uuid ba57606d-b86e-4bf6-bc03-0fddbb644c57 2020-11-16T21:39:52.493 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:52.497+0000 7f93628ded80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:52.496 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:52.501+0000 7f93628ded80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:52.497 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:52.501+0000 7f93628ded80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:52.498 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:52.501+0000 7f93628ded80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:39:52.498 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:52.501+0000 7f93628ded80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:39:52.498 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:52.501+0000 7f93628ded80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:39:52.557 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:52.561+0000 7f93628ded80 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2020-11-16T21:39:56.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2020-11-16T21:39:56.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:39:56.919 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:39:56.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:39:56.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:39:57.424 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:39:57.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:39:57.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:39:57.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:39:57.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:39:57.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:39:57.454 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:57.461+0000 7fe1b8963d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:57.457 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:57.461+0000 7fe1b8963d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:57.459 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:57.465+0000 7fe1b8963d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:39:57.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:39:57.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:39:57.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:39:57.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:39:57.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:39:57.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:39:57.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:39:57.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:39:57.808 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:39:57.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:39:58.000 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:58.005+0000 7fe1b8963d80 -1 Falling back to public interface 2020-11-16T21:39:58.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:39:58.832 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:39:58.833+0000 7fe1b8963d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:39:59.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:39:59.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:39:59.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:39:59.214 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:39:59.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:39:59.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:39:59.614 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/467715,v1:127.0.0.1:6811/467715] [v2:127.0.0.1:6812/467715,v1:127.0.0.1:6813/467715] exists,up ba57606d-b86e-4bf6-bc03-0fddbb644c57 2020-11-16T21:39:59.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:39:59.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:39:59.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:39:59.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:284: TEST_backfill_down_out: run_osd td/osd-backfill-stats 2 2020-11-16T21:39:59.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:39:59.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:39:59.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:39:59.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:39:59.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2020-11-16T21:39:59.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:39:59.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:39:59.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:39:59.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:39:59.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2020-11-16T21:39:59.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2020-11-16T21:39:59.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:39:59.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:39:59.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:39:59.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:39:59.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:39:59.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:39:59.619 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:39:59.619 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:39:59.619 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:39:59.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:39:59.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:39:59.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:39:59.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:39:59.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:39:59.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:39:59.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:39:59.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:39:59.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:39:59.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:39:59.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:39:59.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:39:59.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2020-11-16T21:39:59.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:39:59.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=08e9664d-1742-4fdc-990b-596b0e024d0e 2020-11-16T21:39:59.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 08e9664d-1742-4fdc-990b-596b0e024d0e' 2020-11-16T21:39:59.622 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 08e9664d-1742-4fdc-990b-596b0e024d0e 2020-11-16T21:39:59.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:39:59.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCv8bJfA6nhJhAAI25AXd/ktEfR4pBrLS3QYw== 2020-11-16T21:39:59.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCv8bJfA6nhJhAAI25AXd/ktEfR4pBrLS3QYw=="}' 2020-11-16T21:39:59.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 08e9664d-1742-4fdc-990b-596b0e024d0e -i td/osd-backfill-stats/2/new.json 2020-11-16T21:40:00.227 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:40:00.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2020-11-16T21:40:00.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCv8bJfA6nhJhAAI25AXd/ktEfR4pBrLS3QYw== --osd-uuid 08e9664d-1742-4fdc-990b-596b0e024d0e 2020-11-16T21:40:00.277 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:00.281+0000 7fb58ddd4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:00.280 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:00.285+0000 7fb58ddd4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:00.282 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:00.289+0000 7fb58ddd4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:00.283 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:00.289+0000 7fb58ddd4d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:40:00.283 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:00.289+0000 7fb58ddd4d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:40:00.283 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:00.289+0000 7fb58ddd4d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:40:00.335 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:00.341+0000 7fb58ddd4d80 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2020-11-16T21:40:04.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2020-11-16T21:40:04.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:40:04.973 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:40:04.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:40:04.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:40:05.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:40:05.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:40:05.475 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:40:05.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:40:05.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:40:05.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:40:05.495 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:05.501+0000 7f0e85fd5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:05.497 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:05.501+0000 7f0e85fd5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:05.499 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:05.505+0000 7f0e85fd5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:05.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:40:05.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:40:05.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:40:05.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:40:05.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:40:05.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:40:05.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:40:05.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:40:05.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:40:05.848 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:40:06.040 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:06.045+0000 7f0e85fd5d80 -1 Falling back to public interface 2020-11-16T21:40:06.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:40:06.882 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:06.885+0000 7f0e85fd5d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:40:07.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:40:07.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:40:07.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:40:07.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:40:07.240 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:40:07.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:40:07.651 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/468710,v1:127.0.0.1:6819/468710] [v2:127.0.0.1:6820/468710,v1:127.0.0.1:6821/468710] exists,up 08e9664d-1742-4fdc-990b-596b0e024d0e 2020-11-16T21:40:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:40:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:40:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:40:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:285: TEST_backfill_down_out: run_osd td/osd-backfill-stats 3 2020-11-16T21:40:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:40:07.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:40:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:40:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:40:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2020-11-16T21:40:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:40:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:40:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:40:07.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:40:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2020-11-16T21:40:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2020-11-16T21:40:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:40:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:40:07.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:40:07.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:40:07.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:40:07.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:40:07.655 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:40:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:40:07.656 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:40:07.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:40:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:40:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:40:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:40:07.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:40:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:40:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:40:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:40:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:40:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:40:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:40:07.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:40:07.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2020-11-16T21:40:07.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:40:07.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8baf3de2-2ab2-4b3f-a531-d37ffc2009d4 2020-11-16T21:40:07.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 8baf3de2-2ab2-4b3f-a531-d37ffc2009d4' 2020-11-16T21:40:07.660 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 8baf3de2-2ab2-4b3f-a531-d37ffc2009d4 2020-11-16T21:40:07.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:40:07.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC38bJf7mLgKBAAryl56lBNEwWzNGUTkVEHDA== 2020-11-16T21:40:07.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC38bJf7mLgKBAAryl56lBNEwWzNGUTkVEHDA=="}' 2020-11-16T21:40:07.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8baf3de2-2ab2-4b3f-a531-d37ffc2009d4 -i td/osd-backfill-stats/3/new.json 2020-11-16T21:40:08.263 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:40:08.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2020-11-16T21:40:08.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC38bJf7mLgKBAAryl56lBNEwWzNGUTkVEHDA== --osd-uuid 8baf3de2-2ab2-4b3f-a531-d37ffc2009d4 2020-11-16T21:40:08.306 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:08.313+0000 7fba1802fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:08.309 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:08.313+0000 7fba1802fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:08.310 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:08.317+0000 7fba1802fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:08.311 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:08.317+0000 7fba1802fd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:40:08.311 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:08.317+0000 7fba1802fd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:40:08.311 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:08.317+0000 7fba1802fd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:40:08.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:08.389+0000 7fba1802fd80 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2020-11-16T21:40:13.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2020-11-16T21:40:13.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:40:13.086 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:40:13.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:40:13.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:40:13.572 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:40:13.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:40:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:40:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:40:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:40:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:40:13.601 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:13.605+0000 7fd983923d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:13.603 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:13.609+0000 7fd983923d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:13.609+0000 7fd983923d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:13.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:40:13.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:40:13.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:40:13.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:40:13.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:40:13.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:40:13.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:40:13.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:40:13.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:40:13.978 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:40:14.164 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:14.169+0000 7fd983923d80 -1 Falling back to public interface 2020-11-16T21:40:14.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:40:14.779 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:14.785+0000 7fd983923d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:40:15.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:40:15.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:40:15.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:40:15.394 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:40:15.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:40:15.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:40:15.805 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/469704,v1:127.0.0.1:6827/469704] [v2:127.0.0.1:6828/469704,v1:127.0.0.1:6829/469704] exists,up 8baf3de2-2ab2-4b3f-a531-d37ffc2009d4 2020-11-16T21:40:15.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:40:15.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:40:15.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:40:15.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:286: TEST_backfill_down_out: run_osd td/osd-backfill-stats 4 2020-11-16T21:40:15.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:40:15.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:40:15.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:40:15.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:40:15.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2020-11-16T21:40:15.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:40:15.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:40:15.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:40:15.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:40:15.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2020-11-16T21:40:15.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2020-11-16T21:40:15.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:40:15.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:40:15.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:40:15.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:40:15.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:40:15.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:40:15.810 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:40:15.810 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:40:15.810 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:40:15.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:40:15.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:40:15.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:40:15.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:40:15.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:40:15.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:40:15.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:40:15.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:40:15.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:40:15.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:40:15.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:40:15.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:40:15.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2020-11-16T21:40:15.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:40:15.814 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 cf1e6f1d-a31a-4b4f-8c1a-cf95a24544b8 2020-11-16T21:40:15.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=cf1e6f1d-a31a-4b4f-8c1a-cf95a24544b8 2020-11-16T21:40:15.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 cf1e6f1d-a31a-4b4f-8c1a-cf95a24544b8' 2020-11-16T21:40:15.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:40:15.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC/8bJfxJMxMhAAly7TGW/SKCCbN6PlC4q4Rg== 2020-11-16T21:40:15.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC/8bJfxJMxMhAAly7TGW/SKCCbN6PlC4q4Rg=="}' 2020-11-16T21:40:15.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new cf1e6f1d-a31a-4b4f-8c1a-cf95a24544b8 -i td/osd-backfill-stats/4/new.json 2020-11-16T21:40:16.323 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:40:16.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2020-11-16T21:40:16.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC/8bJfxJMxMhAAly7TGW/SKCCbN6PlC4q4Rg== --osd-uuid cf1e6f1d-a31a-4b4f-8c1a-cf95a24544b8 2020-11-16T21:40:16.371 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:16.377+0000 7fd371359d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:16.375 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:16.381+0000 7fd371359d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:16.377 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:16.381+0000 7fd371359d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:16.377 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:16.381+0000 7fd371359d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:40:16.378 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:16.381+0000 7fd371359d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:40:16.378 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:16.381+0000 7fd371359d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:40:16.436 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:16.445+0000 7fd371359d80 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2020-11-16T21:40:21.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2020-11-16T21:40:21.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:40:21.066 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:40:21.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:40:21.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:40:21.572 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:40:21.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:40:21.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:40:21.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:40:21.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:40:21.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:40:21.600 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:21.605+0000 7f2eff61ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:21.603 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:21.609+0000 7f2eff61ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:21.604 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:21.609+0000 7f2eff61ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:21.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:40:21.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:40:21.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:40:21.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:40:21.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:40:21.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:40:21.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:40:21.960 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:40:21.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:40:21.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:40:22.156 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:22.161+0000 7f2eff61ad80 -1 Falling back to public interface 2020-11-16T21:40:22.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:40:23.034 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:23.037+0000 7f2eff61ad80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:40:23.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:40:23.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:40:23.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:40:23.371 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:40:23.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:40:23.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:40:23.829 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/470698,v1:127.0.0.1:6835/470698] [v2:127.0.0.1:6836/470698,v1:127.0.0.1:6837/470698] exists,up cf1e6f1d-a31a-4b4f-8c1a-cf95a24544b8 2020-11-16T21:40:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:40:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:40:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:40:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:287: TEST_backfill_down_out: run_osd td/osd-backfill-stats 5 2020-11-16T21:40:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:40:23.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:40:23.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T21:40:23.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:40:23.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2020-11-16T21:40:23.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:40:23.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:40:23.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:40:23.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:40:23.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2020-11-16T21:40:23.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2020-11-16T21:40:23.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:40:23.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:40:23.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:40:23.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:40:23.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:40:23.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:40:23.834 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:40:23.835 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:40:23.835 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:40:23.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:40:23.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:40:23.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:40:23.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:40:23.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:40:23.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:40:23.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:40:23.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:40:23.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:40:23.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:40:23.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:40:23.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:40:23.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2020-11-16T21:40:23.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:40:23.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1727d8fc-962a-45a3-8232-dbe108f269cb 2020-11-16T21:40:23.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 1727d8fc-962a-45a3-8232-dbe108f269cb' 2020-11-16T21:40:23.840 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 1727d8fc-962a-45a3-8232-dbe108f269cb 2020-11-16T21:40:23.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:40:23.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDH8bJfzzHZMxAA9EGD04SivlOrMvFrDaXWMw== 2020-11-16T21:40:23.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDH8bJfzzHZMxAA9EGD04SivlOrMvFrDaXWMw=="}' 2020-11-16T21:40:23.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1727d8fc-962a-45a3-8232-dbe108f269cb -i td/osd-backfill-stats/5/new.json 2020-11-16T21:40:24.520 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T21:40:24.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2020-11-16T21:40:24.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDH8bJfzzHZMxAA9EGD04SivlOrMvFrDaXWMw== --osd-uuid 1727d8fc-962a-45a3-8232-dbe108f269cb 2020-11-16T21:40:24.565 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:24.569+0000 7fb9437bcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:24.568 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:24.573+0000 7fb9437bcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:24.569 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:24.573+0000 7fb9437bcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:24.570 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:24.577+0000 7fb9437bcd80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:40:24.570 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:24.577+0000 7fb9437bcd80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:40:24.570 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:24.577+0000 7fb9437bcd80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:40:24.625 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:24.629+0000 7fb9437bcd80 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2020-11-16T21:40:29.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2020-11-16T21:40:29.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:40:29.022 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T21:40:29.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T21:40:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:40:29.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T21:40:29.527 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T21:40:29.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:40:29.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:40:29.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:40:29.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:40:29.552 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:29.556+0000 7f5e356c3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:29.554 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:29.560+0000 7f5e356c3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:29.555 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:29.560+0000 7f5e356c3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:40:29.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T21:40:29.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:40:29.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T21:40:29.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:40:29.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:40:29.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:40:29.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:40:29.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:40:29.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:40:29.907 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:40:30.108 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:30.112+0000 7f5e356c3d80 -1 Falling back to public interface 2020-11-16T21:40:30.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:40:30.714 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:40:30.720+0000 7f5e356c3d80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T21:40:31.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:40:31.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:40:31.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:40:31.337 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:40:31.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:40:31.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:40:31.730 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/471692,v1:127.0.0.1:6843/471692] [v2:127.0.0.1:6844/471692,v1:127.0.0.1:6845/471692] exists,up 1727d8fc-962a-45a3-8232-dbe108f269cb 2020-11-16T21:40:31.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:40:31.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:40:31.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:40:31.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:289: TEST_backfill_down_out: create_pool test 1 1 2020-11-16T21:40:31.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T21:40:32.277 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T21:40:32.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:40:33.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:290: TEST_backfill_down_out: ceph osd pool set test size 2 2020-11-16T21:40:34.104 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T21:40:34.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:291: TEST_backfill_down_out: sleep 5 2020-11-16T21:40:39.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:293: TEST_backfill_down_out: wait_for_clean 2020-11-16T21:40:39.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:40:39.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:40:39.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:40:39.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:40:39.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:40:39.124 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:40:39.125 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:40:39.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:40:39.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:40:39.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:40:39.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:40:39.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:40:39.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:40:39.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:40:39.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:40:39.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:40:39.748 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:40:39.748 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:40:39.748 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:40:39.748 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:40:39.748 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:40:39.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:40:39.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:40:39.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:40:39.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836492 2020-11-16T21:40:39.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836492 2020-11-16T21:40:39.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492' 2020-11-16T21:40:39.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:40:39.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:40:40.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672970 2020-11-16T21:40:40.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672970 2020-11-16T21:40:40.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672970' 2020-11-16T21:40:40.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:40:40.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:40:40.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509450 2020-11-16T21:40:40.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509450 2020-11-16T21:40:40.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-64424509450' 2020-11-16T21:40:40.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:40:40.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:40:40.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345927 2020-11-16T21:40:40.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345927 2020-11-16T21:40:40.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-64424509450 3-85899345927' 2020-11-16T21:40:40.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:40:40.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:40:40.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182407 2020-11-16T21:40:40.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182407 2020-11-16T21:40:40.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-64424509450 3-85899345927 4-107374182407' 2020-11-16T21:40:40.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:40:40.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:40:40.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018884 2020-11-16T21:40:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018884 2020-11-16T21:40:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-64424509450 3-85899345927 4-107374182407 5-128849018884' 2020-11-16T21:40:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:40:40.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836492 2020-11-16T21:40:40.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:40:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:40:40.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836492 2020-11-16T21:40:40.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:40:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836492 2020-11-16T21:40:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-11-16T21:40:40.750 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836492 2020-11-16T21:40:40.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:40:41.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836492 2020-11-16T21:40:41.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:40:42.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:40:42.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:40:42.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836492 2020-11-16T21:40:42.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:40:42.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2020-11-16T21:40:42.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:40:42.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:40:42.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672970 2020-11-16T21:40:42.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:40:42.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672970 2020-11-16T21:40:42.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2020-11-16T21:40:42.544 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672970 2020-11-16T21:40:42.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:40:46.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672970 2020-11-16T21:40:46.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:40:46.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509450 2020-11-16T21:40:46.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:40:46.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:40:46.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509450 2020-11-16T21:40:46.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:40:46.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509450 2020-11-16T21:40:46.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2020-11-16T21:40:46.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:40:46.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509450 -lt 64424509450 2020-11-16T21:40:46.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:40:46.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345927 2020-11-16T21:40:46.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:40:46.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:40:46.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345927 2020-11-16T21:40:46.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:40:46.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345927 2020-11-16T21:40:46.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2020-11-16T21:40:46.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:40:46.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345928 -lt 85899345927 2020-11-16T21:40:46.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:40:46.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182407 2020-11-16T21:40:46.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:40:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:40:46.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:40:46.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182407 2020-11-16T21:40:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182407 2020-11-16T21:40:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182407' 2020-11-16T21:40:46.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:40:46.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182407 -lt 107374182407 2020-11-16T21:40:46.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:40:46.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018884 2020-11-16T21:40:46.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:40:46.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:40:46.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018884 2020-11-16T21:40:46.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:40:46.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018884 2020-11-16T21:40:46.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018884' 2020-11-16T21:40:46.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:40:46.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018885 -lt 128849018884 2020-11-16T21:40:46.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:40:46.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:40:46.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:40:46.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:40:46.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:40:46.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:40:46.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:40:46.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:40:46.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:40:46.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:40:46.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:40:46.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:40:46.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:40:46.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:40:46.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:40:46.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:40:46.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:40:46.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:40:46.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: seq 1 200 2020-11-16T21:40:46.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:46.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj1 /dev/null 2020-11-16T21:40:46.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:46.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj2 /dev/null 2020-11-16T21:40:46.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:46.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj3 /dev/null 2020-11-16T21:40:46.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:46.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj4 /dev/null 2020-11-16T21:40:46.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:46.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj5 /dev/null 2020-11-16T21:40:46.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:46.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj6 /dev/null 2020-11-16T21:40:46.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:46.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj7 /dev/null 2020-11-16T21:40:46.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:46.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj8 /dev/null 2020-11-16T21:40:46.659 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509450 2020-11-16T21:40:46.659 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 85899345927 2020-11-16T21:40:46.659 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 107374182407 2020-11-16T21:40:46.659 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 128849018884 2020-11-16T21:40:46.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:46.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj9 /dev/null 2020-11-16T21:40:46.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:46.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj10 /dev/null 2020-11-16T21:40:46.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:46.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj11 /dev/null 2020-11-16T21:40:47.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:47.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj12 /dev/null 2020-11-16T21:40:47.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:47.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj13 /dev/null 2020-11-16T21:40:47.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:47.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj14 /dev/null 2020-11-16T21:40:47.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:47.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj15 /dev/null 2020-11-16T21:40:47.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:47.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj16 /dev/null 2020-11-16T21:40:47.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:47.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj17 /dev/null 2020-11-16T21:40:47.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:47.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj18 /dev/null 2020-11-16T21:40:47.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:47.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj19 /dev/null 2020-11-16T21:40:47.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:47.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj20 /dev/null 2020-11-16T21:40:48.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:48.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj21 /dev/null 2020-11-16T21:40:48.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:48.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj22 /dev/null 2020-11-16T21:40:52.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj23 /dev/null 2020-11-16T21:40:52.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj24 /dev/null 2020-11-16T21:40:52.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj25 /dev/null 2020-11-16T21:40:52.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj26 /dev/null 2020-11-16T21:40:52.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj27 /dev/null 2020-11-16T21:40:52.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj28 /dev/null 2020-11-16T21:40:52.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj29 /dev/null 2020-11-16T21:40:52.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj30 /dev/null 2020-11-16T21:40:52.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj31 /dev/null 2020-11-16T21:40:52.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj32 /dev/null 2020-11-16T21:40:52.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj33 /dev/null 2020-11-16T21:40:52.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj34 /dev/null 2020-11-16T21:40:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj35 /dev/null 2020-11-16T21:40:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj36 /dev/null 2020-11-16T21:40:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj37 /dev/null 2020-11-16T21:40:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj38 /dev/null 2020-11-16T21:40:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj39 /dev/null 2020-11-16T21:40:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj40 /dev/null 2020-11-16T21:40:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj41 /dev/null 2020-11-16T21:40:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj42 /dev/null 2020-11-16T21:40:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj43 /dev/null 2020-11-16T21:40:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj44 /dev/null 2020-11-16T21:40:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj45 /dev/null 2020-11-16T21:40:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj46 /dev/null 2020-11-16T21:40:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj47 /dev/null 2020-11-16T21:40:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj48 /dev/null 2020-11-16T21:40:52.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj49 /dev/null 2020-11-16T21:40:52.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj50 /dev/null 2020-11-16T21:40:52.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj51 /dev/null 2020-11-16T21:40:52.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj52 /dev/null 2020-11-16T21:40:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj53 /dev/null 2020-11-16T21:40:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj54 /dev/null 2020-11-16T21:40:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj55 /dev/null 2020-11-16T21:40:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj56 /dev/null 2020-11-16T21:40:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj57 /dev/null 2020-11-16T21:40:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj58 /dev/null 2020-11-16T21:40:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj59 /dev/null 2020-11-16T21:40:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj60 /dev/null 2020-11-16T21:40:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj61 /dev/null 2020-11-16T21:40:52.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj62 /dev/null 2020-11-16T21:40:52.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj63 /dev/null 2020-11-16T21:40:52.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj64 /dev/null 2020-11-16T21:40:52.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj65 /dev/null 2020-11-16T21:40:52.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:52.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj66 /dev/null 2020-11-16T21:40:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj67 /dev/null 2020-11-16T21:40:53.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:53.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj68 /dev/null 2020-11-16T21:40:53.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:53.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj69 /dev/null 2020-11-16T21:40:53.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:53.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj70 /dev/null 2020-11-16T21:40:53.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:53.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj71 /dev/null 2020-11-16T21:40:53.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:53.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj72 /dev/null 2020-11-16T21:40:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:53.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj73 /dev/null 2020-11-16T21:40:53.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:53.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj74 /dev/null 2020-11-16T21:40:53.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:53.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj75 /dev/null 2020-11-16T21:40:54.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:54.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj76 /dev/null 2020-11-16T21:40:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj77 /dev/null 2020-11-16T21:40:54.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:54.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj78 /dev/null 2020-11-16T21:40:54.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:54.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj79 /dev/null 2020-11-16T21:40:54.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj80 /dev/null 2020-11-16T21:40:54.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:54.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj81 /dev/null 2020-11-16T21:40:54.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:54.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj82 /dev/null 2020-11-16T21:40:54.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:54.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj83 /dev/null 2020-11-16T21:40:54.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:54.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj84 /dev/null 2020-11-16T21:40:54.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:54.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj85 /dev/null 2020-11-16T21:40:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj86 /dev/null 2020-11-16T21:40:55.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:55.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj87 /dev/null 2020-11-16T21:40:55.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:55.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj88 /dev/null 2020-11-16T21:40:55.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:55.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj89 /dev/null 2020-11-16T21:40:55.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:55.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj90 /dev/null 2020-11-16T21:40:55.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:55.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj91 /dev/null 2020-11-16T21:40:55.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:55.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj92 /dev/null 2020-11-16T21:40:55.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:55.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj93 /dev/null 2020-11-16T21:40:55.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:55.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj94 /dev/null 2020-11-16T21:40:56.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:56.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj95 /dev/null 2020-11-16T21:40:56.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:56.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj96 /dev/null 2020-11-16T21:40:56.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:56.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj97 /dev/null 2020-11-16T21:40:56.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:56.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj98 /dev/null 2020-11-16T21:40:56.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:56.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj99 /dev/null 2020-11-16T21:40:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj100 /dev/null 2020-11-16T21:40:56.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:56.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj101 /dev/null 2020-11-16T21:40:56.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:56.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj102 /dev/null 2020-11-16T21:40:56.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:56.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj103 /dev/null 2020-11-16T21:40:57.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:57.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj104 /dev/null 2020-11-16T21:40:57.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:57.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj105 /dev/null 2020-11-16T21:40:57.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:57.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj106 /dev/null 2020-11-16T21:40:57.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:57.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj107 /dev/null 2020-11-16T21:40:57.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:57.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj108 /dev/null 2020-11-16T21:40:57.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj109 /dev/null 2020-11-16T21:40:57.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:57.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj110 /dev/null 2020-11-16T21:40:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:57.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj111 /dev/null 2020-11-16T21:40:57.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:57.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj112 /dev/null 2020-11-16T21:40:58.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:58.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj113 /dev/null 2020-11-16T21:40:58.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:58.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj114 /dev/null 2020-11-16T21:40:58.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:58.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj115 /dev/null 2020-11-16T21:40:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:58.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj116 /dev/null 2020-11-16T21:40:58.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:58.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj117 /dev/null 2020-11-16T21:40:58.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:58.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj118 /dev/null 2020-11-16T21:40:58.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:58.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj119 /dev/null 2020-11-16T21:40:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj120 /dev/null 2020-11-16T21:40:58.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:58.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj121 /dev/null 2020-11-16T21:40:59.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:59.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj122 /dev/null 2020-11-16T21:40:59.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:59.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj123 /dev/null 2020-11-16T21:40:59.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:59.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj124 /dev/null 2020-11-16T21:40:59.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:59.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj125 /dev/null 2020-11-16T21:40:59.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:59.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj126 /dev/null 2020-11-16T21:40:59.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:59.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj127 /dev/null 2020-11-16T21:40:59.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:59.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj128 /dev/null 2020-11-16T21:40:59.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:59.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj129 /dev/null 2020-11-16T21:40:59.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:40:59.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj130 /dev/null 2020-11-16T21:41:00.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:00.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj131 /dev/null 2020-11-16T21:41:00.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:00.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj132 /dev/null 2020-11-16T21:41:00.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:00.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj133 /dev/null 2020-11-16T21:41:00.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:00.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj134 /dev/null 2020-11-16T21:41:00.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj135 /dev/null 2020-11-16T21:41:00.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:00.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj136 /dev/null 2020-11-16T21:41:00.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:00.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj137 /dev/null 2020-11-16T21:41:00.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:00.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj138 /dev/null 2020-11-16T21:41:00.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:00.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj139 /dev/null 2020-11-16T21:41:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:00.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj140 /dev/null 2020-11-16T21:41:01.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:01.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj141 /dev/null 2020-11-16T21:41:01.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:01.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj142 /dev/null 2020-11-16T21:41:01.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:01.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj143 /dev/null 2020-11-16T21:41:01.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:01.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj144 /dev/null 2020-11-16T21:41:01.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:01.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj145 /dev/null 2020-11-16T21:41:01.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:01.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj146 /dev/null 2020-11-16T21:41:01.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:01.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj147 /dev/null 2020-11-16T21:41:01.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:01.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj148 /dev/null 2020-11-16T21:41:01.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:01.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj149 /dev/null 2020-11-16T21:41:02.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:02.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj150 /dev/null 2020-11-16T21:41:02.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:02.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj151 /dev/null 2020-11-16T21:41:02.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:02.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj152 /dev/null 2020-11-16T21:41:02.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:02.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj153 /dev/null 2020-11-16T21:41:02.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:02.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj154 /dev/null 2020-11-16T21:41:02.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:02.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj155 /dev/null 2020-11-16T21:41:02.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:02.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj156 /dev/null 2020-11-16T21:41:02.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:02.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj157 /dev/null 2020-11-16T21:41:02.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:02.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj158 /dev/null 2020-11-16T21:41:03.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:03.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj159 /dev/null 2020-11-16T21:41:03.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:03.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj160 /dev/null 2020-11-16T21:41:03.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:03.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj161 /dev/null 2020-11-16T21:41:03.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:03.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj162 /dev/null 2020-11-16T21:41:03.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:03.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj163 /dev/null 2020-11-16T21:41:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj164 /dev/null 2020-11-16T21:41:03.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:03.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj165 /dev/null 2020-11-16T21:41:03.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:03.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj166 /dev/null 2020-11-16T21:41:03.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:03.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj167 /dev/null 2020-11-16T21:41:03.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:03.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj168 /dev/null 2020-11-16T21:41:04.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:04.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj169 /dev/null 2020-11-16T21:41:04.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:04.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj170 /dev/null 2020-11-16T21:41:04.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:04.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj171 /dev/null 2020-11-16T21:41:04.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:04.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj172 /dev/null 2020-11-16T21:41:04.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj173 /dev/null 2020-11-16T21:41:04.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:04.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj174 /dev/null 2020-11-16T21:41:04.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:04.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj175 /dev/null 2020-11-16T21:41:04.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:04.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj176 /dev/null 2020-11-16T21:41:04.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:04.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj177 /dev/null 2020-11-16T21:41:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj178 /dev/null 2020-11-16T21:41:05.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:05.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj179 /dev/null 2020-11-16T21:41:05.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:05.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj180 /dev/null 2020-11-16T21:41:05.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:05.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj181 /dev/null 2020-11-16T21:41:05.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:05.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj182 /dev/null 2020-11-16T21:41:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj183 /dev/null 2020-11-16T21:41:05.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:05.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj184 /dev/null 2020-11-16T21:41:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:05.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj185 /dev/null 2020-11-16T21:41:05.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:05.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj186 /dev/null 2020-11-16T21:41:06.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:06.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj187 /dev/null 2020-11-16T21:41:06.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:06.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj188 /dev/null 2020-11-16T21:41:06.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:06.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj189 /dev/null 2020-11-16T21:41:06.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:06.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj190 /dev/null 2020-11-16T21:41:06.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:06.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj191 /dev/null 2020-11-16T21:41:06.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:06.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj192 /dev/null 2020-11-16T21:41:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:06.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj193 /dev/null 2020-11-16T21:41:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj194 /dev/null 2020-11-16T21:41:06.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:06.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj195 /dev/null 2020-11-16T21:41:07.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:07.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj196 /dev/null 2020-11-16T21:41:07.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:07.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj197 /dev/null 2020-11-16T21:41:07.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:07.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj198 /dev/null 2020-11-16T21:41:07.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:07.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj199 /dev/null 2020-11-16T21:41:07.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2020-11-16T21:41:07.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj200 /dev/null 2020-11-16T21:41:07.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:300: TEST_backfill_down_out: get_pg test obj1 2020-11-16T21:41:07.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T21:41:07.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T21:41:07.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T21:41:07.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T21:41:07.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:300: TEST_backfill_down_out: local PG=1.0 2020-11-16T21:41:07.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:302: TEST_backfill_down_out: get_primary test obj1 2020-11-16T21:41:07.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:41:07.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:41:07.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:41:07.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:41:08.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:302: TEST_backfill_down_out: local primary=1 2020-11-16T21:41:08.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:303: TEST_backfill_down_out: get_not_primary test obj1 2020-11-16T21:41:08.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2020-11-16T21:41:08.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2020-11-16T21:41:08.360 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2020-11-16T21:41:08.360 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:41:08.360 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:41:08.361 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:41:08.361 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:41:08.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T21:41:08.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2020-11-16T21:41:08.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T21:41:09.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:303: TEST_backfill_down_out: local otherosd=0 2020-11-16T21:41:09.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:305: TEST_backfill_down_out: ceph osd set nobackfill 2020-11-16T21:41:09.816 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T21:41:09.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:306: TEST_backfill_down_out: cat td/osd-backfill-stats/osd.0.pid 2020-11-16T21:41:09.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:306: TEST_backfill_down_out: kill 466721 2020-11-16T21:41:09.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:307: TEST_backfill_down_out: ceph osd down osd.0 2020-11-16T21:41:10.231 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already down. 2020-11-16T21:41:10.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:308: TEST_backfill_down_out: ceph osd out osd.0 2020-11-16T21:41:10.872 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T21:41:10.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:309: TEST_backfill_down_out: sleep 2 2020-11-16T21:41:12.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:310: TEST_backfill_down_out: ceph osd unset nobackfill 2020-11-16T21:41:13.613 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T21:41:13.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:312: TEST_backfill_down_out: wait_for_clean 2020-11-16T21:41:13.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:41:13.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:41:13.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:41:13.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:41:13.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:41:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:41:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:41:13.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:41:13.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:41:13.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:41:13.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:41:13.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:41:13.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:41:13.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:41:13.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:41:14.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:41:14.240 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:41:14.240 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:41:14.241 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:41:14.241 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:41:14.241 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:41:14.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:41:14.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:41:14.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:41:14.368 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.0 2020-11-16T21:41:14.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T21:41:14.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T21:41:14.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T21:41:14.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:41:14.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:41:14.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672979 2020-11-16T21:41:14.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672979 2020-11-16T21:41:14.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672979' 2020-11-16T21:41:14.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:41:18.720 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:41:18.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509459 2020-11-16T21:41:18.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509459 2020-11-16T21:41:18.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672979 2-64424509459' 2020-11-16T21:41:18.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:41:18.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:41:18.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345936 2020-11-16T21:41:18.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345936 2020-11-16T21:41:18.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672979 2-64424509459 3-85899345936' 2020-11-16T21:41:18.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:41:18.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:41:18.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182416 2020-11-16T21:41:18.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182416 2020-11-16T21:41:18.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672979 2-64424509459 3-85899345936 4-107374182416' 2020-11-16T21:41:18.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:41:18.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:41:18.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018893 2020-11-16T21:41:18.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018893 2020-11-16T21:41:18.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672979 2-64424509459 3-85899345936 4-107374182416 5-128849018893' 2020-11-16T21:41:18.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:41:18.724 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672979 2020-11-16T21:41:18.724 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:41:18.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:41:18.724 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672979 2020-11-16T21:41:18.724 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:41:18.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672979 2020-11-16T21:41:18.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2020-11-16T21:41:18.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:41:18.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672977 -lt 42949672979 2020-11-16T21:41:18.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:41:18.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:41:18.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:41:18.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672980 -lt 42949672979 2020-11-16T21:41:18.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:41:18.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509459 2020-11-16T21:41:18.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:41:18.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:41:18.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509459 2020-11-16T21:41:18.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:41:18.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509459 2020-11-16T21:41:18.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2020-11-16T21:41:18.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:41:18.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509459 -lt 64424509459 2020-11-16T21:41:18.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:41:18.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345936 2020-11-16T21:41:18.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:41:18.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:41:18.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345936 2020-11-16T21:41:18.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:41:18.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345936 2020-11-16T21:41:18.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345936' 2020-11-16T21:41:18.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:41:18.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345936 -lt 85899345936 2020-11-16T21:41:18.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:41:18.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182416 2020-11-16T21:41:18.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:41:18.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:41:18.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182416 2020-11-16T21:41:18.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:41:18.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182416 2020-11-16T21:41:18.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182416' 2020-11-16T21:41:18.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:41:18.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182416 -lt 107374182416 2020-11-16T21:41:18.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:41:18.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018893 2020-11-16T21:41:18.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:41:18.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:41:18.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018893 2020-11-16T21:41:18.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:41:18.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018893 2020-11-16T21:41:18.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018893' 2020-11-16T21:41:18.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:41:18.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018894 -lt 128849018893 2020-11-16T21:41:18.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:41:18.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:18.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:18.731 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672979 2020-11-16T21:41:18.732 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509459 2020-11-16T21:41:18.732 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 85899345936 2020-11-16T21:41:18.732 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 107374182416 2020-11-16T21:41:18.732 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 128849018893 2020-11-16T21:41:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:41:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:19.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:19.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:19.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:19.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:19.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:19.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:19.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:19.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:23.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:23.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:23.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:23.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T21:41:23.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T21:41:23.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T21:41:23.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:23.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:23.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:23.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:23.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:23.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:23.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:23.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:23.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:23.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:23.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:23.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:23.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:23.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:23.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:23.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:23.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:23.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:23.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:23.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:23.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:23.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:23.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:23.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=1 2020-11-16T21:41:23.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 1 '!=' null 2020-11-16T21:41:23.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:23.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:23.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:23.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:23.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:23.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:23.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:23.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:23.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:23.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:23.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:23.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:23.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:23.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:23.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:23.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:23.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:23.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:23.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:23.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:23.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:23.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:41:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:41:23.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:23.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:23.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:23.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:23.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:23.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:23.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:23.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:23.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:23.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:23.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:23.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:23.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:23.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:23.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:23.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:23.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:23.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:23.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:23.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:23.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:23.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:23.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:23.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:24.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:41:24.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:41:24.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:24.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:24.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:24.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:24.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:24.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:24.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:24.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:24.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:24.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:24.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:24.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:24.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:25.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:25.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:25.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:25.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:25.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:25.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:25.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:25.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:25.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:25.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:41:25.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:41:25.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:25.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:25.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:25.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:25.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:25.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:25.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:25.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:25.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:25.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:25.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:26.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:26.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:26.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:26.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:26.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:26.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:26.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:27.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:41:27.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:41:27.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:27.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:27.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:27.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:27.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:27.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:27.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:27.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:27.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:27.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:27.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:27.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:27.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:27.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:27.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:28.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:28.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:28.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:28.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:28.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:28.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:28.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:28.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:28.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:28.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:41:28.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:41:28.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:28.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:28.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:28.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:28.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:28.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:28.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:28.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:28.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:29.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:29.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:29.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:29.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:29.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:29.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:29.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:29.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:29.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:29.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:29.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:29.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:29.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:30.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:41:30.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:41:30.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:30.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:30.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:30.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:30.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:30.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:30.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:30.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:30.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:30.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:30.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:30.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:30.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:31.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:31.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:31.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:31.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:31.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:31.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:31.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:35.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:35.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:41:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:41:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:35.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:35.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:35.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:35.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:35.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:35.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:35.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:35.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:35.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:35.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:35.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:35.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:35.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:35.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:35.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:35.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:35.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:35.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:35.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:35.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2020-11-16T21:41:35.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2020-11-16T21:41:35.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:35.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:35.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:35.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:35.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:35.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:35.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:35.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:35.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:35.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:35.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:35.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:35.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:35.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:35.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:35.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:35.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:35.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:35.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:35.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:35.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:35.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:35.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:35.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:35.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:41:35.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:41:35.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:35.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:35.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:35.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:35.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:35.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:35.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:35.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:35.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:35.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:35.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:35.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:35.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:35.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:35.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:35.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:35.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:35.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:35.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:35.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:35.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:35.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:35.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:35.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:35.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2020-11-16T21:41:35.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2020-11-16T21:41:35.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:35.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:35.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:36.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:36.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:36.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:36.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:36.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:36.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:36.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:36.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:36.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:36.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:36.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:36.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:36.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:36.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:36.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:36.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:36.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:36.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:36.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:36.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:36.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:37.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2020-11-16T21:41:37.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2020-11-16T21:41:37.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:37.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:37.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:37.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:37.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:37.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:37.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:37.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:37.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:37.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:37.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:37.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:37.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:37.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:37.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:38.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:38.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:38.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:41:38.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:41:38.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:38.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:38.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:39.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:39.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:39.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:39.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:39.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:39.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:39.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:39.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:39.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:39.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:39.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:39.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:40.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:41:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:41:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:40.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:40.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:40.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:40.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:40.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:40.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:40.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:40.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:40.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:40.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:40.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:41.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:41.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:41.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:41.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:41.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:41.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:41.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:41.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:41.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:41.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:41:41.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:41:41.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:41.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:41.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:41.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:41.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:41.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:41.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:41.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:42.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:42.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:42.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:42.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:42.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:42.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:42.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:42.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:43.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2020-11-16T21:41:43.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2020-11-16T21:41:43.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:43.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:43.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:43.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:43.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:43.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:43.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:43.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:43.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:43.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:43.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:43.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:43.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:44.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:44.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:44.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:44.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:44.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:44.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:44.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:44.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:44.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:41:44.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:41:44.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:44.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:44.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:44.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:44.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:44.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:44.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:44.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:44.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:44.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:44.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:45.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:45.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:45.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:45.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:45.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:45.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:45.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:45.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:45.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:45.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:45.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:45.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:45.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7 2020-11-16T21:41:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7 '!=' null 2020-11-16T21:41:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:46.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:46.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:46.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:46.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:46.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:46.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:46.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:46.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:46.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:46.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:46.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:47.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:47.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:47.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:47.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:47.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:47.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:47.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:47.301 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:47.301 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:47.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7 2020-11-16T21:41:47.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7 '!=' null 2020-11-16T21:41:47.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:47.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:47.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:47.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:47.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:47.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:47.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:47.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:47.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:47.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:47.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:48.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:48.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:48.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:48.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:48.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:48.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:48.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:48.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:48.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:48.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:48.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:48.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:48.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:49.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7 2020-11-16T21:41:49.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7 '!=' null 2020-11-16T21:41:49.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:49.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:49.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:49.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:49.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:49.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:49.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:49.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:49.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:49.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:49.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:49.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:49.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:49.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:49.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:50.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:50.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:50.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:50.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:50.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:50.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:50.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:50.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:50.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:50.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:41:50.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:41:50.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:50.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:50.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:50.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:50.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:50.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:50.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:50.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:50.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:50.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:50.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:51.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:41:51.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:51.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:51.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:51.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:41:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:41:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:41:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:41:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:41:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:41:51.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:41:51.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:41:51.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:41:52.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:41:52.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:41:52.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:41:52.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:41:52.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:41:52.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:41:52.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:41:52.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:41:52.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:41:52.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:41:52.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:41:52.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:41:52.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:41:52.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:41:52.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:41:52.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:41:52.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:41:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:41:52.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:41:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:41:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:314: TEST_backfill_down_out: check td/osd-backfill-stats 1.0 1 replicated 200 0 0 0 2020-11-16T21:41:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2020-11-16T21:41:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2020-11-16T21:41:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2020-11-16T21:41:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2020-11-16T21:41:52.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=200 2020-11-16T21:41:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2020-11-16T21:41:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=0 2020-11-16T21:41:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2020-11-16T21:41:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2020-11-16T21:41:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2020-11-16T21:41:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2020-11-16T21:41:52.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2020-11-16T21:41:52.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2020-11-16T21:41:52.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2020-11-16T21:41:53.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2020-11-16T21:41:53.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2020-11-16T21:41:53.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2020-11-16T21:41:53.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2020-11-16T21:41:53.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2020-11-16T21:41:53.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2020-11-16T21:41:53.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2020-11-16T21:41:53.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2020-11-16T21:41:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:41:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2020-11-16T21:41:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2020-11-16T21:41:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[1,2]/[1]' 2020-11-16T21:41:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[1,2]/[1]' ']' 2020-11-16T21:41:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2020-11-16T21:41:53.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:41:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:41:53.040 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2020-11-16T21:41:53.040 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [1,2]/[1] ' 2020-11-16T21:41:53.040 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:41:53.041 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2020-11-16T21:41:53.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=200 2020-11-16T21:41:53.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 200 ']' 2020-11-16T21:41:53.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 200 200 2020-11-16T21:41:53.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2020-11-16T21:41:53.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:41:53.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2020-11-16T21:41:53.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:41:53.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:41:53.046 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:41:53.047 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2020-11-16T21:41:53.047 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:41:53.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=1 2020-11-16T21:41:53.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 1 ']' 2020-11-16T21:41:53.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 1 0 2020-11-16T21:41:53.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2020-11-16T21:41:53.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:41:53.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:41:53.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:41:53.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2020-11-16T21:41:53.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:41:53.056 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:41:53.056 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [1,2]/[1] ' 2020-11-16T21:41:53.057 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2020-11-16T21:41:53.057 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2020-11-16T21:41:53.057 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:41:53.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=0 2020-11-16T21:41:53.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 0 ']' 2020-11-16T21:41:53.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 0 0 2020-11-16T21:41:53.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2020-11-16T21:41:53.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:41:53.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2020-11-16T21:41:53.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:41:53.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:41:53.066 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:41:53.066 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:41:53.067 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2020-11-16T21:41:53.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=0 2020-11-16T21:41:53.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 0 ']' 2020-11-16T21:41:53.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 0 0 2020-11-16T21:41:53.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2020-11-16T21:41:53.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:41:53.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:41:53.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:41:53.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2020-11-16T21:41:53.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:316: TEST_backfill_down_out: delete_pool test 2020-11-16T21:41:53.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T21:41:53.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T21:41:53.587 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T21:41:53.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:317: TEST_backfill_down_out: kill_daemons td/osd-backfill-stats 2020-11-16T21:41:53.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:41:53.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:41:53.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:41:53.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:41:53.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:41:53.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:41:53.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2020-11-16T21:41:53.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:41:53.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:41:53.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:41:53.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:41:53.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:41:53.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:41:53.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:41:53.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:41:53.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:41:53.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:41:53.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:41:53.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:41:53.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:41:53.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:41:53.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:41:53.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:41:53.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:41:53.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:41:53.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:41:53.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:41:53.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:41:53.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:41:53.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:41:53.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:41:53.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:41:53.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:41:53.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:41:53.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:41:53.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2020-11-16T21:41:53.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2020-11-16T21:41:53.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2020-11-16T21:41:53.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2020-11-16T21:41:53.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:41:53.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:41:53.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:41:53.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:41:53.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:41:53.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:41:53.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:41:53.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:41:53.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:41:53.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:41:53.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:41:53.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:41:53.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:41:53.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:41:53.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:41:53.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:41:53.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:41:53.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:41:53.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:41:53.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:41:53.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:41:53.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:41:53.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:41:53.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:41:53.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:41:53.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:41:53.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:41:53.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:41:53.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2020-11-16T21:41:53.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:41:53.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:41:53.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:41:53.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.465868 2020-11-16T21:41:53.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:41:53.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:41:53.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:41:53.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2020-11-16T21:41:53.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_ec_all_out td/osd-backfill-stats 2020-11-16T21:41:53.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:515: TEST_backfill_ec_all_out: local dir=td/osd-backfill-stats 2020-11-16T21:41:53.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:517: TEST_backfill_ec_all_out: run_mon td/osd-backfill-stats a 2020-11-16T21:41:53.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2020-11-16T21:41:53.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:41:53.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:41:53.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:41:53.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2020-11-16T21:41:53.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2020-11-16T21:41:54.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:41:54.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:41:54.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:41:54.080 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:41:54.081 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:41:54.081 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:41:54.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:41:54.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:41:54.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:41:54.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:41:54.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:41:54.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:41:54.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:41:54.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:41:54.286 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:41:54.286 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:41:54.286 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:41:54.286 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:41:54.287 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:41:54.287 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:41:54.287 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:41:54.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:41:54.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get fsid 2020-11-16T21:41:54.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:41:54.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:41:54.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:41:54.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:41:54.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:41:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:41:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:41:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:41:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:41:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:41:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:41:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:41:54.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:41:54.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get mon_host 2020-11-16T21:41:54.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:518: TEST_backfill_ec_all_out: run_mgr td/osd-backfill-stats x 2020-11-16T21:41:54.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2020-11-16T21:41:54.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:41:54.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:41:54.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:41:54.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2020-11-16T21:41:54.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:41:55.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:41:55.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:41:55.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:41:55.023 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:41:55.023 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:41:55.023 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:41:55.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:41:55.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:41:55.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:41:55.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:519: TEST_backfill_ec_all_out: run_osd td/osd-backfill-stats 0 2020-11-16T21:41:55.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:41:55.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:41:55.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:41:55.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:41:55.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2020-11-16T21:41:55.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:41:55.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:41:55.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:41:55.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:41:55.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2020-11-16T21:41:55.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2020-11-16T21:41:55.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:41:55.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:41:55.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:41:55.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:41:55.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:41:55.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:41:55.053 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:41:55.053 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:41:55.053 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:41:55.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:41:55.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:41:55.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:41:55.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:41:55.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:41:55.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:41:55.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:41:55.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:41:55.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:41:55.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:41:55.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:41:55.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:41:55.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2020-11-16T21:41:55.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:41:55.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ade2b957-36a8-4e24-ad76-a7a34165ef90 2020-11-16T21:41:55.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 ade2b957-36a8-4e24-ad76-a7a34165ef90' 2020-11-16T21:41:55.056 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 ade2b957-36a8-4e24-ad76-a7a34165ef90 2020-11-16T21:41:55.057 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:41:55.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAj8rJfrnUgBRAApHTObvcP4JOVcNipBAdPdQ== 2020-11-16T21:41:55.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAj8rJfrnUgBRAApHTObvcP4JOVcNipBAdPdQ=="}' 2020-11-16T21:41:55.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ade2b957-36a8-4e24-ad76-a7a34165ef90 -i td/osd-backfill-stats/0/new.json 2020-11-16T21:41:55.438 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:41:55.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2020-11-16T21:41:55.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAj8rJfrnUgBRAApHTObvcP4JOVcNipBAdPdQ== --osd-uuid ade2b957-36a8-4e24-ad76-a7a34165ef90 2020-11-16T21:41:55.481 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:41:55.487+0000 7f154f947d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:41:55.485 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:41:55.491+0000 7f154f947d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:41:55.486 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:41:55.495+0000 7f154f947d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:41:55.486 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:41:55.495+0000 7f154f947d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:41:55.486 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:41:55.495+0000 7f154f947d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:41:55.487 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:41:55.495+0000 7f154f947d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:41:55.534 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:41:55.543+0000 7f154f947d80 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2020-11-16T21:41:59.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2020-11-16T21:41:59.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:41:59.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:41:59.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:41:59.899 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:42:00.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:42:00.403 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:42:00.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:42:00.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:42:00.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:42:00.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:42:00.433 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:00.439+0000 7fc87df72d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:00.435 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:00.443+0000 7fc87df72d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:00.436 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:00.443+0000 7fc87df72d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:00.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:42:00.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:42:00.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:42:00.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:42:00.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:42:00.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:42:00.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:42:00.779 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:42:00.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:42:00.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:42:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:00.991+0000 7fc87df72d80 -1 Falling back to public interface 2020-11-16T21:42:01.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:42:01.629 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:01.631+0000 7fc87df72d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:42:02.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:42:02.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:42:02.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:42:02.200 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:42:02.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:42:02.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:42:02.602 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/480177,v1:127.0.0.1:6803/480177] [v2:127.0.0.1:6804/480177,v1:127.0.0.1:6805/480177] exists,up ade2b957-36a8-4e24-ad76-a7a34165ef90 2020-11-16T21:42:02.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:42:02.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:42:02.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:42:02.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:520: TEST_backfill_ec_all_out: run_osd td/osd-backfill-stats 1 2020-11-16T21:42:02.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:42:02.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:42:02.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:42:02.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:42:02.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2020-11-16T21:42:02.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:42:02.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:42:02.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:42:02.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:42:02.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2020-11-16T21:42:02.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2020-11-16T21:42:02.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:42:02.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:42:02.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:42:02.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:42:02.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:42:02.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:42:02.607 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:42:02.607 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:42:02.607 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:42:02.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:42:02.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:42:02.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:42:02.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:42:02.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:42:02.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:42:02.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:42:02.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:42:02.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:42:02.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:42:02.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:42:02.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:42:02.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2020-11-16T21:42:02.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:42:02.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1fd5c34c-e6f6-4aa2-8747-8892190cd18d 2020-11-16T21:42:02.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 1fd5c34c-e6f6-4aa2-8747-8892190cd18d' 2020-11-16T21:42:02.610 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 1fd5c34c-e6f6-4aa2-8747-8892190cd18d 2020-11-16T21:42:02.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:42:02.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAq8rJfp/UYJhAANcEteTXOObVkSVI0bpnDwA== 2020-11-16T21:42:02.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAq8rJfp/UYJhAANcEteTXOObVkSVI0bpnDwA=="}' 2020-11-16T21:42:02.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1fd5c34c-e6f6-4aa2-8747-8892190cd18d -i td/osd-backfill-stats/1/new.json 2020-11-16T21:42:03.138 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:42:03.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2020-11-16T21:42:03.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAq8rJfp/UYJhAANcEteTXOObVkSVI0bpnDwA== --osd-uuid 1fd5c34c-e6f6-4aa2-8747-8892190cd18d 2020-11-16T21:42:03.185 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:03.191+0000 7efe7e1d9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:03.188 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:03.195+0000 7efe7e1d9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:03.189 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:03.195+0000 7efe7e1d9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:03.190 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:03.199+0000 7efe7e1d9d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:42:03.190 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:03.199+0000 7efe7e1d9d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:42:03.191 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:03.199+0000 7efe7e1d9d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:42:03.238 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:03.247+0000 7efe7e1d9d80 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2020-11-16T21:42:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2020-11-16T21:42:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:42:07.549 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:42:07.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:42:07.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:42:08.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:42:12.139 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:42:12.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:42:12.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:42:12.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:42:12.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:42:12.145 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:08.058+0000 7f42d406dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:12.145 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:08.058+0000 7f42d406dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:12.145 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:08.062+0000 7f42d406dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:12.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:42:12.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:42:12.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:42:12.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:42:12.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:42:12.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:42:12.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:42:12.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:42:12.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:42:12.147 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:08.606+0000 7f42d406dd80 -1 Falling back to public interface 2020-11-16T21:42:12.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:42:12.147 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:09.522+0000 7f42d406dd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:42:12.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:42:12.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:42:12.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:42:12.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:42:12.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:42:12.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:42:12.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:42:12.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:42:12.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:521: TEST_backfill_ec_all_out: run_osd td/osd-backfill-stats 2 2020-11-16T21:42:12.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:42:12.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:42:12.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:42:12.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:42:12.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2020-11-16T21:42:12.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:42:12.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:42:12.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:42:12.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:42:12.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2020-11-16T21:42:12.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2020-11-16T21:42:12.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:42:12.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:42:12.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:42:12.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:42:12.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:42:12.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:42:12.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:42:12.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:42:12.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:42:12.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:42:12.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:42:12.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:42:12.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:42:12.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:42:12.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:42:12.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:42:12.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:42:12.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:42:12.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:42:12.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:42:12.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:42:12.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2020-11-16T21:42:12.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:42:12.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=58fe3163-4aba-4203-a7fd-53f8d05f0266 2020-11-16T21:42:12.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 58fe3163-4aba-4203-a7fd-53f8d05f0266' 2020-11-16T21:42:12.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:42:12.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAy8rJfZh8uDxAA1MVaPm8snnvf6epHxzpzaQ== 2020-11-16T21:42:12.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAy8rJfZh8uDxAA1MVaPm8snnvf6epHxzpzaQ=="}' 2020-11-16T21:42:12.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 58fe3163-4aba-4203-a7fd-53f8d05f0266 -i td/osd-backfill-stats/2/new.json 2020-11-16T21:42:12.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2020-11-16T21:42:12.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAy8rJfZh8uDxAA1MVaPm8snnvf6epHxzpzaQ== --osd-uuid 58fe3163-4aba-4203-a7fd-53f8d05f0266 2020-11-16T21:42:12.155 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:10.866+0000 7fb11a472d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:12.155 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:10.870+0000 7fb11a472d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:12.155 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:10.870+0000 7fb11a472d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:12.155 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:10.870+0000 7fb11a472d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:42:12.156 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:10.870+0000 7fb11a472d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:42:12.156 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:10.870+0000 7fb11a472d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:42:12.156 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:10.954+0000 7fb11a472d80 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2020-11-16T21:42:12.156 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:42:12.156 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:42:12.156 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/481171,v1:127.0.0.1:6811/481171] [v2:127.0.0.1:6812/481171,v1:127.0.0.1:6813/481171] exists,up 1fd5c34c-e6f6-4aa2-8747-8892190cd18d 2020-11-16T21:42:12.156 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 58fe3163-4aba-4203-a7fd-53f8d05f0266 2020-11-16T21:42:12.157 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:42:15.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2020-11-16T21:42:15.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:42:15.285 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:42:15.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:42:15.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:42:15.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:42:15.780 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:42:15.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:42:15.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:42:15.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:42:15.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:42:15.809 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:15.814+0000 7f00ea59ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:15.812 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:15.818+0000 7f00ea59ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:15.814 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:15.818+0000 7f00ea59ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:16.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:42:16.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:42:16.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:42:16.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:42:16.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:42:16.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:42:16.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:42:16.174 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:42:16.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:42:16.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:42:16.364 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:16.370+0000 7f00ea59ed80 -1 Falling back to public interface 2020-11-16T21:42:16.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:42:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:17.214+0000 7f00ea59ed80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:42:17.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:42:17.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:42:17.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:42:17.568 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:42:17.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:42:17.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:42:17.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:42:18.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:42:18.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:42:18.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:42:18.968 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:42:18.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:42:18.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:42:19.359 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/482165,v1:127.0.0.1:6819/482165] [v2:127.0.0.1:6820/482165,v1:127.0.0.1:6821/482165] exists,up 58fe3163-4aba-4203-a7fd-53f8d05f0266 2020-11-16T21:42:19.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:42:19.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:42:19.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:42:19.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:522: TEST_backfill_ec_all_out: run_osd td/osd-backfill-stats 3 2020-11-16T21:42:19.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:42:19.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:42:19.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:42:19.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:42:19.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2020-11-16T21:42:19.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:42:19.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:42:19.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:42:19.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:42:19.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2020-11-16T21:42:19.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2020-11-16T21:42:19.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:42:19.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:42:19.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:42:19.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:42:19.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:42:19.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:42:19.364 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:42:19.364 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:42:19.364 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:42:19.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:42:19.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:42:19.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:42:19.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:42:19.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:42:19.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:42:19.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:42:19.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:42:19.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:42:19.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:42:19.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:42:19.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:42:19.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2020-11-16T21:42:19.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:42:19.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=58a664a3-1a2c-48ed-bc56-660ac99ac87b 2020-11-16T21:42:19.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 58a664a3-1a2c-48ed-bc56-660ac99ac87b' 2020-11-16T21:42:19.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:42:19.368 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 58a664a3-1a2c-48ed-bc56-660ac99ac87b 2020-11-16T21:42:19.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA78rJfJDuNFxAABT7AR3VTj4p4BQecA4oNRg== 2020-11-16T21:42:19.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA78rJfJDuNFxAABT7AR3VTj4p4BQecA4oNRg=="}' 2020-11-16T21:42:19.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 58a664a3-1a2c-48ed-bc56-660ac99ac87b -i td/osd-backfill-stats/3/new.json 2020-11-16T21:42:19.847 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:42:19.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2020-11-16T21:42:19.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA78rJfJDuNFxAABT7AR3VTj4p4BQecA4oNRg== --osd-uuid 58a664a3-1a2c-48ed-bc56-660ac99ac87b 2020-11-16T21:42:19.893 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:19.898+0000 7f7927b53d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:19.895 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:19.902+0000 7f7927b53d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:19.896 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:19.902+0000 7f7927b53d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:19.897 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:19.902+0000 7f7927b53d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:42:19.897 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:19.902+0000 7f7927b53d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:42:19.897 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:19.902+0000 7f7927b53d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:42:19.941 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:19.946+0000 7f7927b53d80 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2020-11-16T21:42:24.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2020-11-16T21:42:24.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:42:24.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:42:24.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:42:24.325 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:42:24.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:42:24.855 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:42:24.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:42:24.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:42:24.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:42:24.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:42:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:24.890+0000 7f3003013d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:24.888 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:24.894+0000 7f3003013d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:24.889 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:24.894+0000 7f3003013d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:42:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:42:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:42:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:42:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:42:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:42:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:42:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:42:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:42:25.251 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:42:25.432 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:25.438+0000 7f3003013d80 -1 Falling back to public interface 2020-11-16T21:42:25.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:42:26.258 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:26.262+0000 7f3003013d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:42:26.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:42:26.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:42:26.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:42:26.643 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:42:26.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:42:26.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:42:27.061 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/483185,v1:127.0.0.1:6827/483185] [v2:127.0.0.1:6828/483185,v1:127.0.0.1:6829/483185] exists,up 58a664a3-1a2c-48ed-bc56-660ac99ac87b 2020-11-16T21:42:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:42:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:42:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:42:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:523: TEST_backfill_ec_all_out: run_osd td/osd-backfill-stats 4 2020-11-16T21:42:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:42:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:42:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:42:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:42:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2020-11-16T21:42:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:42:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:42:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:42:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:42:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2020-11-16T21:42:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2020-11-16T21:42:27.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:42:27.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:42:27.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:42:27.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:42:27.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:42:27.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:42:27.066 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:42:27.067 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:42:27.067 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:42:27.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:42:27.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:42:27.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:42:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:42:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:42:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:42:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:42:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:42:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:42:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:42:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:42:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:42:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2020-11-16T21:42:27.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:42:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=bbf5e661-958f-4b25-8910-08704687802a 2020-11-16T21:42:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 bbf5e661-958f-4b25-8910-08704687802a' 2020-11-16T21:42:27.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:42:27.071 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 bbf5e661-958f-4b25-8910-08704687802a 2020-11-16T21:42:27.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBD8rJf1szKBRAAG2CM7TxMwHp8abY/Hl6mrw== 2020-11-16T21:42:27.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBD8rJf1szKBRAAG2CM7TxMwHp8abY/Hl6mrw=="}' 2020-11-16T21:42:27.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new bbf5e661-958f-4b25-8910-08704687802a -i td/osd-backfill-stats/4/new.json 2020-11-16T21:42:27.616 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:42:27.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2020-11-16T21:42:27.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBD8rJf1szKBRAAG2CM7TxMwHp8abY/Hl6mrw== --osd-uuid bbf5e661-958f-4b25-8910-08704687802a 2020-11-16T21:42:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:27.666+0000 7f1f1773cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:27.673 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:27.678+0000 7f1f1773cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:27.675 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:27.682+0000 7f1f1773cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:27.676 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:27.682+0000 7f1f1773cd80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:42:27.676 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:27.682+0000 7f1f1773cd80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:42:27.676 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:27.682+0000 7f1f1773cd80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:42:27.740 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:27.746+0000 7f1f1773cd80 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2020-11-16T21:42:32.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2020-11-16T21:42:32.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:42:32.365 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:42:32.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:42:32.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:42:32.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:42:32.877 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:42:32.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:42:32.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:42:32.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:42:32.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:42:32.906 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:32.914+0000 7fb310dd7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:32.908 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:32.914+0000 7fb310dd7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:32.909 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:32.914+0000 7fb310dd7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:42:33.192 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:33.198+0000 7fb310dd7d80 -1 Falling back to public interface 2020-11-16T21:42:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:42:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:42:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:42:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:42:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:42:33.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:42:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:42:33.275 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:42:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:42:33.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:42:33.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:42:34.082 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:42:34.090+0000 7fb310dd7d80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:42:34.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:42:34.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:42:34.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:42:34.692 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:42:34.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:42:34.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:42:35.192 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/484179,v1:127.0.0.1:6835/484179] [v2:127.0.0.1:6836/484179,v1:127.0.0.1:6837/484179] exists,up bbf5e661-958f-4b25-8910-08704687802a 2020-11-16T21:42:35.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:42:35.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:42:35.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:42:35.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:525: TEST_backfill_ec_all_out: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2020-11-16T21:42:35.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:526: TEST_backfill_ec_all_out: create_pool test 1 1 erasure myprofile 2020-11-16T21:42:35.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-11-16T21:42:36.573 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T21:42:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:42:37.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:528: TEST_backfill_ec_all_out: wait_for_clean 2020-11-16T21:42:37.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:42:37.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:42:37.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:42:37.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:42:37.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:42:37.587 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:42:37.587 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:42:37.588 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:42:37.588 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:42:37.588 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:42:37.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:42:37.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:42:37.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:42:37.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:42:37.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:42:38.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:42:38.211 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:42:38.211 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:42:38.211 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:42:38.211 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T21:42:38.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:42:38.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:42:38.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:42:38.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2020-11-16T21:42:38.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2020-11-16T21:42:38.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2020-11-16T21:42:38.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:42:38.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:42:38.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672969 2020-11-16T21:42:38.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672969 2020-11-16T21:42:38.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2020-11-16T21:42:38.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:42:38.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:42:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542151 2020-11-16T21:42:38.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542151 2020-11-16T21:42:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672969 2-60129542151' 2020-11-16T21:42:38.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:42:38.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:42:38.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378629 2020-11-16T21:42:38.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378629 2020-11-16T21:42:38.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672969 2-60129542151 3-81604378629' 2020-11-16T21:42:38.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:42:38.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:42:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215107 2020-11-16T21:42:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215107 2020-11-16T21:42:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672969 2-60129542151 3-81604378629 4-103079215107' 2020-11-16T21:42:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:42:39.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2020-11-16T21:42:39.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:42:39.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:42:39.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2020-11-16T21:42:39.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:42:39.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2020-11-16T21:42:39.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-11-16T21:42:39.063 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836490 2020-11-16T21:42:39.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:42:39.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2020-11-16T21:42:39.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:42:39.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672969 2020-11-16T21:42:39.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:42:39.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:42:39.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2020-11-16T21:42:39.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:42:39.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672969 2020-11-16T21:42:39.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-11-16T21:42:39.448 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672969 2020-11-16T21:42:39.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:42:39.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672969 -lt 42949672969 2020-11-16T21:42:39.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:42:39.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542151 2020-11-16T21:42:39.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:42:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:42:39.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542151 2020-11-16T21:42:39.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:42:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542151 2020-11-16T21:42:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2020-11-16T21:42:39.818 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542151 2020-11-16T21:42:39.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:42:40.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542151 -lt 60129542151 2020-11-16T21:42:40.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:42:40.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378629 2020-11-16T21:42:40.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:42:40.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:42:40.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378629 2020-11-16T21:42:40.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:42:40.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378629 2020-11-16T21:42:40.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378629' 2020-11-16T21:42:40.213 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378629 2020-11-16T21:42:40.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:42:40.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378627 -lt 81604378629 2020-11-16T21:42:40.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:42:41.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:42:41.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:42:41.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378629 -lt 81604378629 2020-11-16T21:42:41.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:42:41.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215107 2020-11-16T21:42:41.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:42:41.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:42:41.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215107 2020-11-16T21:42:41.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:42:41.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215107 2020-11-16T21:42:41.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215107' 2020-11-16T21:42:41.970 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215107 2020-11-16T21:42:41.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:42:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215108 -lt 103079215107 2020-11-16T21:42:42.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:42:42.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:42:42.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:42:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:42:42.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:42:42.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:42:42.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:42:42.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:42:42.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:42:42.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:42:42.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:42:43.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:42:43.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:42:43.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:42:43.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:42:43.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:42:43.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:42:43.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:42:43.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: seq 1 200 2020-11-16T21:42:43.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:43.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj1 /dev/null 2020-11-16T21:42:43.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:43.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj2 /dev/null 2020-11-16T21:42:44.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:44.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj3 /dev/null 2020-11-16T21:42:44.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:44.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj4 /dev/null 2020-11-16T21:42:44.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:44.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj5 /dev/null 2020-11-16T21:42:44.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:44.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj6 /dev/null 2020-11-16T21:42:44.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:44.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj7 /dev/null 2020-11-16T21:42:45.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:45.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj8 /dev/null 2020-11-16T21:42:45.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:45.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj9 /dev/null 2020-11-16T21:42:45.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:45.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj10 /dev/null 2020-11-16T21:42:45.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:45.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj11 /dev/null 2020-11-16T21:42:45.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:45.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj12 /dev/null 2020-11-16T21:42:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:46.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj13 /dev/null 2020-11-16T21:42:46.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:46.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj14 /dev/null 2020-11-16T21:42:46.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:46.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj15 /dev/null 2020-11-16T21:42:46.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:46.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj16 /dev/null 2020-11-16T21:42:46.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:46.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj17 /dev/null 2020-11-16T21:42:47.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:47.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj18 /dev/null 2020-11-16T21:42:47.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:47.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj19 /dev/null 2020-11-16T21:42:47.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:47.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj20 /dev/null 2020-11-16T21:42:47.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:47.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj21 /dev/null 2020-11-16T21:42:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj22 /dev/null 2020-11-16T21:42:48.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:48.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj23 /dev/null 2020-11-16T21:42:48.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:48.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj24 /dev/null 2020-11-16T21:42:48.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:48.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj25 /dev/null 2020-11-16T21:42:48.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:48.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj26 /dev/null 2020-11-16T21:42:49.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:49.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj27 /dev/null 2020-11-16T21:42:49.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:49.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj28 /dev/null 2020-11-16T21:42:49.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:49.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj29 /dev/null 2020-11-16T21:42:49.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:49.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj30 /dev/null 2020-11-16T21:42:49.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:49.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj31 /dev/null 2020-11-16T21:42:50.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:50.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj32 /dev/null 2020-11-16T21:42:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:50.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj33 /dev/null 2020-11-16T21:42:50.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:50.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj34 /dev/null 2020-11-16T21:42:50.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:50.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj35 /dev/null 2020-11-16T21:42:50.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:50.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj36 /dev/null 2020-11-16T21:42:51.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:51.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj37 /dev/null 2020-11-16T21:42:51.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:51.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj38 /dev/null 2020-11-16T21:42:51.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:51.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj39 /dev/null 2020-11-16T21:42:51.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:51.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj40 /dev/null 2020-11-16T21:42:51.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:51.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj41 /dev/null 2020-11-16T21:42:52.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:52.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj42 /dev/null 2020-11-16T21:42:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:52.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj43 /dev/null 2020-11-16T21:42:52.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:52.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj44 /dev/null 2020-11-16T21:42:52.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:52.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj45 /dev/null 2020-11-16T21:42:52.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:52.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj46 /dev/null 2020-11-16T21:42:53.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:53.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj47 /dev/null 2020-11-16T21:42:53.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:53.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj48 /dev/null 2020-11-16T21:42:53.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:53.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj49 /dev/null 2020-11-16T21:42:53.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:53.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj50 /dev/null 2020-11-16T21:42:53.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:53.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj51 /dev/null 2020-11-16T21:42:54.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:54.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj52 /dev/null 2020-11-16T21:42:54.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:54.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj53 /dev/null 2020-11-16T21:42:54.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:54.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj54 /dev/null 2020-11-16T21:42:54.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:54.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj55 /dev/null 2020-11-16T21:42:54.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:54.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj56 /dev/null 2020-11-16T21:42:55.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:55.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj57 /dev/null 2020-11-16T21:42:55.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:55.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj58 /dev/null 2020-11-16T21:42:55.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:55.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj59 /dev/null 2020-11-16T21:42:55.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:55.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj60 /dev/null 2020-11-16T21:42:55.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:55.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj61 /dev/null 2020-11-16T21:42:55.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:55.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj62 /dev/null 2020-11-16T21:42:56.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:56.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj63 /dev/null 2020-11-16T21:42:56.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:56.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj64 /dev/null 2020-11-16T21:42:56.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:56.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj65 /dev/null 2020-11-16T21:42:56.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:56.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj66 /dev/null 2020-11-16T21:42:56.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:56.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj67 /dev/null 2020-11-16T21:42:57.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:57.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj68 /dev/null 2020-11-16T21:42:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:57.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj69 /dev/null 2020-11-16T21:42:57.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:57.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj70 /dev/null 2020-11-16T21:42:57.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:57.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj71 /dev/null 2020-11-16T21:42:57.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:57.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj72 /dev/null 2020-11-16T21:42:58.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:58.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj73 /dev/null 2020-11-16T21:42:58.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:58.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj74 /dev/null 2020-11-16T21:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj75 /dev/null 2020-11-16T21:42:58.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:58.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj76 /dev/null 2020-11-16T21:42:59.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:59.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj77 /dev/null 2020-11-16T21:42:59.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:59.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj78 /dev/null 2020-11-16T21:42:59.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:59.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj79 /dev/null 2020-11-16T21:42:59.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:59.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj80 /dev/null 2020-11-16T21:42:59.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:59.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj81 /dev/null 2020-11-16T21:42:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:42:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj82 /dev/null 2020-11-16T21:43:00.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:00.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj83 /dev/null 2020-11-16T21:43:00.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:00.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj84 /dev/null 2020-11-16T21:43:00.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:00.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj85 /dev/null 2020-11-16T21:43:00.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:00.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj86 /dev/null 2020-11-16T21:43:00.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:00.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj87 /dev/null 2020-11-16T21:43:01.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:01.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj88 /dev/null 2020-11-16T21:43:01.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:01.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj89 /dev/null 2020-11-16T21:43:01.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:01.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj90 /dev/null 2020-11-16T21:43:01.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:01.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj91 /dev/null 2020-11-16T21:43:01.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:01.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj92 /dev/null 2020-11-16T21:43:02.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:02.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj93 /dev/null 2020-11-16T21:43:02.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:02.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj94 /dev/null 2020-11-16T21:43:02.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:02.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj95 /dev/null 2020-11-16T21:43:02.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:02.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj96 /dev/null 2020-11-16T21:43:02.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:02.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj97 /dev/null 2020-11-16T21:43:03.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:03.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj98 /dev/null 2020-11-16T21:43:03.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:03.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj99 /dev/null 2020-11-16T21:43:03.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:03.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj100 /dev/null 2020-11-16T21:43:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:03.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj101 /dev/null 2020-11-16T21:43:03.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:03.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj102 /dev/null 2020-11-16T21:43:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj103 /dev/null 2020-11-16T21:43:04.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:04.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj104 /dev/null 2020-11-16T21:43:04.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:04.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj105 /dev/null 2020-11-16T21:43:04.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:04.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj106 /dev/null 2020-11-16T21:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:04.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj107 /dev/null 2020-11-16T21:43:04.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:04.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj108 /dev/null 2020-11-16T21:43:05.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:05.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj109 /dev/null 2020-11-16T21:43:05.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:05.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj110 /dev/null 2020-11-16T21:43:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj111 /dev/null 2020-11-16T21:43:05.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:05.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj112 /dev/null 2020-11-16T21:43:05.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:05.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj113 /dev/null 2020-11-16T21:43:06.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:06.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj114 /dev/null 2020-11-16T21:43:06.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:06.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj115 /dev/null 2020-11-16T21:43:06.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:06.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj116 /dev/null 2020-11-16T21:43:06.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:06.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj117 /dev/null 2020-11-16T21:43:06.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:06.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj118 /dev/null 2020-11-16T21:43:07.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:07.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj119 /dev/null 2020-11-16T21:43:07.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:07.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj120 /dev/null 2020-11-16T21:43:07.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:07.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj121 /dev/null 2020-11-16T21:43:07.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:07.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj122 /dev/null 2020-11-16T21:43:07.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:07.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj123 /dev/null 2020-11-16T21:43:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj124 /dev/null 2020-11-16T21:43:08.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:08.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj125 /dev/null 2020-11-16T21:43:08.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:08.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj126 /dev/null 2020-11-16T21:43:08.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj127 /dev/null 2020-11-16T21:43:08.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:08.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj128 /dev/null 2020-11-16T21:43:08.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:08.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj129 /dev/null 2020-11-16T21:43:09.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:09.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj130 /dev/null 2020-11-16T21:43:09.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:09.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj131 /dev/null 2020-11-16T21:43:09.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:09.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj132 /dev/null 2020-11-16T21:43:09.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:09.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj133 /dev/null 2020-11-16T21:43:09.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:09.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj134 /dev/null 2020-11-16T21:43:10.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:10.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj135 /dev/null 2020-11-16T21:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj136 /dev/null 2020-11-16T21:43:10.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:10.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj137 /dev/null 2020-11-16T21:43:10.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:10.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj138 /dev/null 2020-11-16T21:43:10.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:10.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj139 /dev/null 2020-11-16T21:43:11.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:11.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj140 /dev/null 2020-11-16T21:43:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:11.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj141 /dev/null 2020-11-16T21:43:11.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:11.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj142 /dev/null 2020-11-16T21:43:11.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:11.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj143 /dev/null 2020-11-16T21:43:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj144 /dev/null 2020-11-16T21:43:11.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:11.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj145 /dev/null 2020-11-16T21:43:12.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj146 /dev/null 2020-11-16T21:43:12.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:12.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj147 /dev/null 2020-11-16T21:43:12.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj148 /dev/null 2020-11-16T21:43:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj149 /dev/null 2020-11-16T21:43:12.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:12.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj150 /dev/null 2020-11-16T21:43:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:13.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj151 /dev/null 2020-11-16T21:43:13.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:13.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj152 /dev/null 2020-11-16T21:43:13.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:13.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj153 /dev/null 2020-11-16T21:43:13.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:13.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj154 /dev/null 2020-11-16T21:43:13.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:13.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj155 /dev/null 2020-11-16T21:43:14.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:14.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj156 /dev/null 2020-11-16T21:43:14.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:14.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj157 /dev/null 2020-11-16T21:43:14.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:14.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj158 /dev/null 2020-11-16T21:43:14.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:14.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj159 /dev/null 2020-11-16T21:43:14.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:14.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj160 /dev/null 2020-11-16T21:43:14.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:14.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj161 /dev/null 2020-11-16T21:43:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj162 /dev/null 2020-11-16T21:43:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:15.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj163 /dev/null 2020-11-16T21:43:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj164 /dev/null 2020-11-16T21:43:15.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:15.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj165 /dev/null 2020-11-16T21:43:15.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:15.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj166 /dev/null 2020-11-16T21:43:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj167 /dev/null 2020-11-16T21:43:16.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:16.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj168 /dev/null 2020-11-16T21:43:16.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:16.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj169 /dev/null 2020-11-16T21:43:16.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:16.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj170 /dev/null 2020-11-16T21:43:16.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:16.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj171 /dev/null 2020-11-16T21:43:17.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:17.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj172 /dev/null 2020-11-16T21:43:17.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:17.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj173 /dev/null 2020-11-16T21:43:17.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:17.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj174 /dev/null 2020-11-16T21:43:17.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:17.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj175 /dev/null 2020-11-16T21:43:17.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:17.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj176 /dev/null 2020-11-16T21:43:18.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:18.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj177 /dev/null 2020-11-16T21:43:18.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:18.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj178 /dev/null 2020-11-16T21:43:18.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:18.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj179 /dev/null 2020-11-16T21:43:18.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:18.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj180 /dev/null 2020-11-16T21:43:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj181 /dev/null 2020-11-16T21:43:18.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:18.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj182 /dev/null 2020-11-16T21:43:19.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:19.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj183 /dev/null 2020-11-16T21:43:19.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:19.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj184 /dev/null 2020-11-16T21:43:19.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:19.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj185 /dev/null 2020-11-16T21:43:19.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:19.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj186 /dev/null 2020-11-16T21:43:19.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:19.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj187 /dev/null 2020-11-16T21:43:20.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:20.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj188 /dev/null 2020-11-16T21:43:20.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:20.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj189 /dev/null 2020-11-16T21:43:20.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj190 /dev/null 2020-11-16T21:43:20.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:20.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj191 /dev/null 2020-11-16T21:43:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj192 /dev/null 2020-11-16T21:43:21.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:21.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj193 /dev/null 2020-11-16T21:43:21.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:21.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj194 /dev/null 2020-11-16T21:43:21.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:21.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj195 /dev/null 2020-11-16T21:43:21.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:21.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj196 /dev/null 2020-11-16T21:43:21.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:21.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj197 /dev/null 2020-11-16T21:43:22.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:22.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj198 /dev/null 2020-11-16T21:43:22.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:22.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj199 /dev/null 2020-11-16T21:43:22.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2020-11-16T21:43:22.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj200 /dev/null 2020-11-16T21:43:22.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:535: TEST_backfill_ec_all_out: get_pg test obj1 2020-11-16T21:43:22.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T21:43:22.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T21:43:22.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T21:43:22.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T21:43:23.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:535: TEST_backfill_ec_all_out: local PG=1.0 2020-11-16T21:43:23.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:537: TEST_backfill_ec_all_out: get_primary test obj1 2020-11-16T21:43:23.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:43:23.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:43:23.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:43:23.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:43:23.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:537: TEST_backfill_ec_all_out: local primary=1 2020-11-16T21:43:23.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:539: TEST_backfill_ec_all_out: ceph osd set nobackfill 2020-11-16T21:43:24.083 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T21:43:24.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:540: TEST_backfill_ec_all_out: get_osds test obj1 2020-11-16T21:43:24.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test 2020-11-16T21:43:24.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj1 2020-11-16T21:43:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test obj1 2020-11-16T21:43:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-11-16T21:43:24.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=1 2020-11-16T21:43:24.519 INFO:tasks.workunit.client.0.smithi189.stderr:0 2020-11-16T21:43:24.519 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T21:43:24.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 1 0 2 2020-11-16T21:43:24.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:540: TEST_backfill_ec_all_out: for o in $(get_osds $poolname obj1) 2020-11-16T21:43:24.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:542: TEST_backfill_ec_all_out: ceph osd out osd.1 2020-11-16T21:43:25.100 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. 2020-11-16T21:43:25.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:540: TEST_backfill_ec_all_out: for o in $(get_osds $poolname obj1) 2020-11-16T21:43:25.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:542: TEST_backfill_ec_all_out: ceph osd out osd.0 2020-11-16T21:43:25.740 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T21:43:25.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:540: TEST_backfill_ec_all_out: for o in $(get_osds $poolname obj1) 2020-11-16T21:43:25.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:542: TEST_backfill_ec_all_out: ceph osd out osd.2 2020-11-16T21:43:26.411 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already out. 2020-11-16T21:43:26.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:545: TEST_backfill_ec_all_out: sleep 2 2020-11-16T21:43:28.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:546: TEST_backfill_ec_all_out: get_primary test obj1 2020-11-16T21:43:28.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:43:28.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:43:28.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:43:28.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:43:28.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:546: TEST_backfill_ec_all_out: primary=1 2020-11-16T21:43:28.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:547: TEST_backfill_ec_all_out: ceph osd unset nobackfill 2020-11-16T21:43:29.650 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T21:43:29.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:548: TEST_backfill_ec_all_out: ceph tell osd.1 get_latest_osdmap 2020-11-16T21:43:29.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:549: TEST_backfill_ec_all_out: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T21:43:30.005 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T21:43:30.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:550: TEST_backfill_ec_all_out: sleep 2 2020-11-16T21:43:32.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:552: TEST_backfill_ec_all_out: wait_for_clean 2020-11-16T21:43:32.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:43:32.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:43:32.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:43:32.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:43:32.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:43:32.018 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:43:32.018 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:43:32.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:43:32.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:43:32.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:43:32.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:43:32.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:43:32.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:43:32.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:43:32.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:43:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:43:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:43:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:43:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:43:32.632 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T21:43:32.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:43:32.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:43:32.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:43:32.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836502 2020-11-16T21:43:32.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836502 2020-11-16T21:43:32.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502' 2020-11-16T21:43:32.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:43:32.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:43:32.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672982 2020-11-16T21:43:32.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672982 2020-11-16T21:43:32.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672982' 2020-11-16T21:43:32.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:43:32.972 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:43:33.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542163 2020-11-16T21:43:33.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542163 2020-11-16T21:43:33.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672982 2-60129542163' 2020-11-16T21:43:33.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:43:33.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:43:33.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378642 2020-11-16T21:43:33.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378642 2020-11-16T21:43:33.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672982 2-60129542163 3-81604378642' 2020-11-16T21:43:33.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:43:33.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:43:33.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215120 2020-11-16T21:43:33.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215120 2020-11-16T21:43:33.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672982 2-60129542163 3-81604378642 4-103079215120' 2020-11-16T21:43:33.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:43:33.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836502 2020-11-16T21:43:33.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:43:33.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:43:33.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836502 2020-11-16T21:43:33.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:43:33.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836502 2020-11-16T21:43:33.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2020-11-16T21:43:33.468 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836502 2020-11-16T21:43:33.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:43:33.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836502 2020-11-16T21:43:33.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:43:34.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:43:34.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:43:35.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836503 -lt 21474836502 2020-11-16T21:43:35.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:43:35.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672982 2020-11-16T21:43:35.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:43:35.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:43:35.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672982 2020-11-16T21:43:35.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:43:35.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672982 2020-11-16T21:43:35.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2020-11-16T21:43:35.290 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672982 2020-11-16T21:43:35.290 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:43:35.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672982 -lt 42949672982 2020-11-16T21:43:35.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:43:35.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542163 2020-11-16T21:43:35.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:43:35.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:43:35.679 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542163 2020-11-16T21:43:35.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:43:35.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542163 2020-11-16T21:43:35.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542163' 2020-11-16T21:43:35.682 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542163 2020-11-16T21:43:35.682 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:43:36.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542164 -lt 60129542163 2020-11-16T21:43:36.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:43:36.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378642 2020-11-16T21:43:36.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:43:36.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:43:36.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378642 2020-11-16T21:43:36.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:43:36.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378642 2020-11-16T21:43:36.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378642' 2020-11-16T21:43:36.072 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378642 2020-11-16T21:43:36.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:43:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378642 -lt 81604378642 2020-11-16T21:43:36.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:43:36.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215120 2020-11-16T21:43:36.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:43:36.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:43:36.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215120 2020-11-16T21:43:36.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:43:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215120 2020-11-16T21:43:36.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215120' 2020-11-16T21:43:36.492 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215120 2020-11-16T21:43:36.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:43:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215121 -lt 103079215120 2020-11-16T21:43:36.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:43:36.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:36.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:37.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:43:37.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:37.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:37.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:37.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:37.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:37.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:37.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:37.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:37.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:37.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:37.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T21:43:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T21:43:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T21:43:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:38.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:38.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:38.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:38.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:38.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:38.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:38.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:38.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:38.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:38.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:38.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:38.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:38.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:39.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:39.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:39.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:39.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:39.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T21:43:39.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T21:43:39.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:39.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:39.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:39.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:39.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:39.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:39.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:39.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:39.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:39.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:39.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:40.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:40.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:40.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:40.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:40.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:40.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:40.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:40.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:41.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T21:43:41.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T21:43:41.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:41.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:41.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:41.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:41.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:41.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:41.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:41.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:41.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:41.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:41.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:41.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:41.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:41.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:41.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:42.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:42.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T21:43:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T21:43:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:42.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:42.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:42.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:42.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:42.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:42.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:42.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:42.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:42.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:42.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:43.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:43.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:43.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:43.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:43.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:43.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:43.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:43.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:43:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:43:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:44.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:44.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:44.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:44.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:44.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:44.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:44.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:44.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:44.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:44.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:45.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:45.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:45.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:45.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:45.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:45.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:45.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:45.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:45.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:45.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:43:45.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:43:45.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:45.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:45.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:45.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:45.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:45.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:45.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:45.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:45.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:45.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:45.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:46.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:46.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:46.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:46.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:46.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:46.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:46.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:46.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:46.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:46.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:46.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:46.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:46.679 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:47.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:43:47.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:43:47.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:47.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:47.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:47.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:47.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:47.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:47.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:47.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:47.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:47.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:47.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:47.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:47.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:47.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:51.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:51.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:51.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:51.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:43:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:43:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:51.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:51.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:51.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:51.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:51.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:51.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:51.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:51.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:51.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:51.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:51.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:43:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:43:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:51.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:51.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:51.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:51.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:51.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:51.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:51.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:51.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:51.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:51.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:51.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:51.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:51.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:51.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:51.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:51.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:51.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:51.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:51.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:51.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:43:51.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:43:51.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:51.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:51.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:51.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:51.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:51.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:51.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:51.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:51.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:51.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:51.805 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:52.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:52.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:52.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:52.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:52.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:52.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:52.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:52.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:52.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:52.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:52.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:52.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:52.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:43:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:43:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:53.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:53.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:53.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:53.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:53.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:53.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:53.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:53.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:53.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:53.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:53.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:53.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:53.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:53.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:53.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:53.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:53.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:53.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:53.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:53.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:53.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:54.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:43:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:43:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:54.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:54.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:54.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:54.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:54.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:54.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:54.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:54.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:54.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:54.926 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:54.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:54.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:55.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:55.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:55.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:55.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:55.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:55.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:55.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:55.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:55.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:55.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:43:55.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:43:55.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:55.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:55.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:56.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:56.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:56.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:56.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:56.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:56.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:56.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:56.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:56.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:56.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:56.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:56.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:56.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:56.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:56.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:56.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:56.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:56.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:56.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:56.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:56.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:57.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:43:57.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:43:57.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:57.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:57.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:57.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:57.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:57.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:57.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:57.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:57.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:57.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:57.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:57.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:57.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:57.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:57.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:58.361 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:58.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:43:58.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:43:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:43:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:43:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:43:58.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:43:58.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:43:58.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:43:58.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:43:58.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:43:58.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:43:58.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:43:58.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:43:58.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:43:59.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:43:59.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:43:59.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:43:59.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:43:59.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:43:59.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:43:59.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:43:59.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:43:59.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:43:59.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:43:59.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:43:59.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:43:59.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:00.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:44:00.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:44:00.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:00.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:00.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:00.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:00.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:00.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:00.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:00.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:00.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:00.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:00.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:00.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:00.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:00.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:00.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:01.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:01.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:01.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:01.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:01.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:01.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:01.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:01.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:01.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:44:01.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:44:01.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:01.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:01.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:01.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:01.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:01.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:01.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:01.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:01.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:02.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:02.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:02.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:02.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:02.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:02.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:02.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:02.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:02.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:02.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:02.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:02.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:02.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:03.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:44:03.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:44:03.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:03.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:03.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:03.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:03.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:03.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:03.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:03.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:03.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:03.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:03.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:03.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:03.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:03.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:03.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:04.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:04.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:04.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:04.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:04.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:04.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:04.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:04.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:04.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:04.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:44:04.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:44:04.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:04.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:04.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:04.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:04.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:04.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:04.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:04.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:04.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:04.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:04.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:05.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:05.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:05.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:05.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:05.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:05.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:05.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:05.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:05.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:05.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:05.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:05.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:05.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:06.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:44:06.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:44:06.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:06.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:06.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:06.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:06.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:06.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:06.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:06.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:06.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:06.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:06.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:06.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:06.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:07.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:07.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:07.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:07.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:07.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:07.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:07.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:07.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:07.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:07.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:44:07.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:44:07.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:07.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:07.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:07.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:07.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:07.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:07.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:07.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:07.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:07.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:07.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:08.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:08.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:08.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:08.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:08.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:08.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:08.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:08.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:08.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:08.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:08.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:08.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:08.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:09.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:44:09.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:44:09.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:09.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:09.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:09.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:09.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:09.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:09.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:09.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:09.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:09.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:09.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:09.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:09.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:09.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:09.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:10.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:10.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:10.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:10.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:10.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:10.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:10.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:10.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:10.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:10.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:44:10.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:44:10.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:10.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:10.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:10.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:10.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:10.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:10.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:10.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:10.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:10.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:10.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:11.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:11.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:11.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:11.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:11.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:11.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:11.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:11.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:11.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:12.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:44:12.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:44:12.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:12.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:12.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:12.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:12.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:12.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:12.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:12.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:12.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:12.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:12.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:12.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:12.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:12.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:12.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:13.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:13.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:13.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:13.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:13.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:13.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:13.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:13.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:13.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:44:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:44:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:13.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:13.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:13.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:13.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:13.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:13.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:13.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:13.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:13.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:14.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:14.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:14.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:14.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:14.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:14.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:14.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:14.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:14.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:14.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:14.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:14.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:14.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:44:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:44:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:15.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:15.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:15.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:15.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:15.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:15.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:15.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:15.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:15.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:15.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:15.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:15.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:16.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:16.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:16.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:16.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:16.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:16.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:16.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:16.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:16.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:16.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:44:16.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:44:16.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:16.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:16.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:16.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:16.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:16.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:16.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:16.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:16.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:16.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:16.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:16.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:16.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:16.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:16.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:17.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:17.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:17.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:17.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:17.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:17.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:17.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:17.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:17.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:17.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:44:17.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:44:17.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:17.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:17.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:18.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:18.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:18.004 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:18.004 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:18.004 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:18.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:18.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:18.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:18.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:18.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:18.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:18.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:18.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:18.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:18.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:18.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:18.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:18.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:18.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:18.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:18.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:19.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:44:19.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:44:19.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:19.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:19.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:19.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:19.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:19.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:19.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:19.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:19.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:19.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:19.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:19.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:19.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:19.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:19.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:20.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:20.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:20.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:20.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:44:20.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:44:20.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:20.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:20.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:20.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:20.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:20.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:20.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:20.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:20.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:20.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:20.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:21.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:21.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:21.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:21.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:21.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:21.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:21.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:21.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:21.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:21.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:21.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:21.805 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:21.805 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:22.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:44:22.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:44:22.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:22.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:22.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:22.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:22.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:22.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:22.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:22.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:22.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:22.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:22.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:22.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:22.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:22.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:22.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:23.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:23.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:23.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:44:23.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:44:23.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:23.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:23.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:23.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:23.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:23.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:23.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:23.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:23.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:23.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:23.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:24.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:24.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:24.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:24.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:24.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:24.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:24.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:24.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:24.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:24.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:24.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:24.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:24.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T21:44:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T21:44:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:25.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:25.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:25.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:25.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:25.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:25.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:25.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:25.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:44:25.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:25.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:25.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:26.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:44:26.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:44:26.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:44:26.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:44:26.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:44:26.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:44:26.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:44:26.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:44:26.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:44:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T21:44:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T21:44:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:44:26.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:44:26.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:44:26.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:44:26.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:44:26.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:44:26.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:44:26.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:44:26.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:44:26.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:44:26.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:44:27.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:44:27.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:44:27.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:44:27.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:44:27.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:44:27.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:44:27.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:44:27.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:554: TEST_backfill_ec_all_out: expr 200 '*' 3 2020-11-16T21:44:27.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:554: TEST_backfill_ec_all_out: local misplaced=600 2020-11-16T21:44:27.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:555: TEST_backfill_ec_all_out: check td/osd-backfill-stats 1.0 1 erasure 0 0 600 200 2020-11-16T21:44:27.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2020-11-16T21:44:27.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2020-11-16T21:44:27.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2020-11-16T21:44:27.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=erasure 2020-11-16T21:44:27.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=0 2020-11-16T21:44:27.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2020-11-16T21:44:27.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=600 2020-11-16T21:44:27.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=200 2020-11-16T21:44:27.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2020-11-16T21:44:27.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2020-11-16T21:44:27.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2020-11-16T21:44:27.632 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2020-11-16T21:44:27.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2020-11-16T21:44:27.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2020-11-16T21:44:27.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log 2020-11-16T21:44:27.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2020-11-16T21:44:27.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2020-11-16T21:44:27.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2020-11-16T21:44:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2020-11-16T21:44:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2020-11-16T21:44:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' erasure = erasure ']' 2020-11-16T21:44:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:89: check: addp=p 2020-11-16T21:44:27.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2020-11-16T21:44:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:44:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2020-11-16T21:44:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2020-11-16T21:44:27.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[4,3,NONE]/[1,0,2]' 2020-11-16T21:44:27.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[4,3,NONE]/[1,0,2]' ']' 2020-11-16T21:44:27.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2020-11-16T21:44:27.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:44:27.691 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:44:27.692 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2020-11-16T21:44:27.692 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [4,3,NONE]/[1,0,2]p' 2020-11-16T21:44:27.692 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2020-11-16T21:44:27.693 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:44:27.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=0 2020-11-16T21:44:27.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 0 ']' 2020-11-16T21:44:27.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 0 0 2020-11-16T21:44:27.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2020-11-16T21:44:27.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:44:27.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2020-11-16T21:44:27.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:44:27.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:44:27.700 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:44:27.700 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2020-11-16T21:44:27.700 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:44:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=0 2020-11-16T21:44:27.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 0 ']' 2020-11-16T21:44:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 0 0 2020-11-16T21:44:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2020-11-16T21:44:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:44:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:44:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:44:27.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2020-11-16T21:44:27.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:44:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:44:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [4,3,NONE]/[1,0,2]p' 2020-11-16T21:44:27.721 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2020-11-16T21:44:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2020-11-16T21:44:27.722 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:44:27.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=600 2020-11-16T21:44:27.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 600 ']' 2020-11-16T21:44:27.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 600 600 2020-11-16T21:44:27.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=600 2020-11-16T21:44:27.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:44:27.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=600 2020-11-16T21:44:27.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:44:27.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:44:27.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:44:27.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2020-11-16T21:44:27.732 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:44:27.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=202 2020-11-16T21:44:27.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 202 ']' 2020-11-16T21:44:27.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 202 200 2020-11-16T21:44:27.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=202 2020-11-16T21:44:27.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:44:27.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=200 2020-11-16T21:44:27.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:44:27.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2020-11-16T21:44:27.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:557: TEST_backfill_ec_all_out: delete_pool test 2020-11-16T21:44:27.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T21:44:27.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T21:44:28.231 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T21:44:28.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:558: TEST_backfill_ec_all_out: kill_daemons td/osd-backfill-stats 2020-11-16T21:44:28.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:44:28.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:44:28.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:44:28.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:44:28.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:44:28.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:44:28.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2020-11-16T21:44:28.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:44:28.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:44:28.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:44:28.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:44:28.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:44:28.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:44:28.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:44:28.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:44:28.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:44:28.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:44:28.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:44:28.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:44:28.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:44:28.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:44:28.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:44:28.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:44:28.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:44:28.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:44:28.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:44:28.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:44:28.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:44:28.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:44:28.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:44:28.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:44:28.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:44:28.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:44:28.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:44:28.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:44:28.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2020-11-16T21:44:28.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2020-11-16T21:44:28.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2020-11-16T21:44:28.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2020-11-16T21:44:28.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:44:28.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:44:28.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:44:28.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:44:28.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:44:28.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:44:28.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:44:28.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:44:28.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:44:28.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:44:28.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:44:28.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:44:28.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:44:28.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:44:28.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:44:28.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:44:28.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:44:28.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:44:28.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:44:28.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:44:28.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:44:28.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:44:28.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:44:28.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:44:28.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:44:28.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:44:28.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:44:28.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:44:28.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2020-11-16T21:44:28.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:44:28.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:44:28.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:44:28.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.465868 2020-11-16T21:44:28.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:44:28.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:44:28.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:44:28.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2020-11-16T21:44:28.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_ec_down_all_out td/osd-backfill-stats 2020-11-16T21:44:28.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:619: TEST_backfill_ec_down_all_out: local dir=td/osd-backfill-stats 2020-11-16T21:44:28.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:621: TEST_backfill_ec_down_all_out: run_mon td/osd-backfill-stats a 2020-11-16T21:44:28.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2020-11-16T21:44:28.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:44:28.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:44:28.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:44:28.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2020-11-16T21:44:28.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2020-11-16T21:44:28.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:44:28.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:44:28.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:44:28.951 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:44:28.951 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:44:28.951 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:44:28.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:44:28.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:44:29.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:44:29.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:44:29.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:44:29.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:44:29.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:44:29.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:44:29.165 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:44:29.165 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:44:29.165 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:44:29.166 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:44:29.166 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:44:29.166 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:44:29.167 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:44:29.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:44:29.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get fsid 2020-11-16T21:44:29.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:44:29.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:44:29.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:44:29.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:44:29.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:44:29.275 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:44:29.276 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:44:29.276 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:44:29.276 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:44:29.276 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:44:29.276 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:44:29.276 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:44:29.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:44:29.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get mon_host 2020-11-16T21:44:29.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:622: TEST_backfill_ec_down_all_out: run_mgr td/osd-backfill-stats x 2020-11-16T21:44:29.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2020-11-16T21:44:29.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:44:29.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:44:29.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:44:29.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2020-11-16T21:44:29.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:44:29.908 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:44:29.908 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:44:29.908 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:44:29.909 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:44:29.909 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:44:29.909 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:44:29.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:44:29.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:44:29.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:44:29.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:623: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 0 2020-11-16T21:44:29.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:44:29.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:44:29.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:44:29.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:44:29.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2020-11-16T21:44:29.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:44:29.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:44:29.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:44:29.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:44:29.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2020-11-16T21:44:29.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2020-11-16T21:44:29.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:44:29.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:44:29.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:44:29.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:44:29.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:44:29.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:44:29.945 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:44:29.945 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:44:29.945 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:44:29.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:44:29.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:44:29.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:44:29.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:44:29.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:44:29.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:44:29.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:44:29.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:44:29.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:44:29.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:44:29.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:44:29.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:44:29.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2020-11-16T21:44:29.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:44:29.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7c9c471c-90bd-4710-a79f-d2d14597d397 2020-11-16T21:44:29.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 7c9c471c-90bd-4710-a79f-d2d14597d397' 2020-11-16T21:44:29.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:44:29.948 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 7c9c471c-90bd-4710-a79f-d2d14597d397 2020-11-16T21:44:29.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC98rJfgRolOhAAI5pqneUitGWkMXdhghVWzQ== 2020-11-16T21:44:29.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC98rJfgRolOhAAI5pqneUitGWkMXdhghVWzQ=="}' 2020-11-16T21:44:29.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7c9c471c-90bd-4710-a79f-d2d14597d397 -i td/osd-backfill-stats/0/new.json 2020-11-16T21:44:30.326 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:44:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2020-11-16T21:44:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC98rJfgRolOhAAI5pqneUitGWkMXdhghVWzQ== --osd-uuid 7c9c471c-90bd-4710-a79f-d2d14597d397 2020-11-16T21:44:30.368 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:30.372+0000 7ff395987d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:30.370 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:30.376+0000 7ff395987d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:30.372 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:30.376+0000 7ff395987d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:30.372 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:30.376+0000 7ff395987d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:44:30.372 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:30.376+0000 7ff395987d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:44:30.373 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:30.376+0000 7ff395987d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:44:30.429 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:30.432+0000 7ff395987d80 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2020-11-16T21:44:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2020-11-16T21:44:34.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:44:34.840 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:44:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:44:34.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:44:35.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:44:35.349 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:44:35.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:44:35.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:44:35.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:44:35.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:44:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:35.384+0000 7fc09d587d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:35.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:35.384+0000 7fc09d587d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:35.388+0000 7fc09d587d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:35.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:44:35.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:44:35.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:44:35.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:44:35.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:44:35.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:44:35.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:44:35.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:44:35.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:44:35.735 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:44:35.915 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:35.920+0000 7fc09d587d80 -1 Falling back to public interface 2020-11-16T21:44:36.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:44:36.784 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:36.784+0000 7fc09d587d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:44:37.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:44:37.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:44:37.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:44:37.135 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:44:37.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:44:37.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:44:37.560 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/493525,v1:127.0.0.1:6803/493525] [v2:127.0.0.1:6804/493525,v1:127.0.0.1:6805/493525] exists,up 7c9c471c-90bd-4710-a79f-d2d14597d397 2020-11-16T21:44:37.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:44:37.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:44:37.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:44:37.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:624: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 1 2020-11-16T21:44:37.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:44:37.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:44:37.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:44:37.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:44:37.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2020-11-16T21:44:37.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:44:37.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:44:37.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:44:37.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:44:37.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2020-11-16T21:44:37.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2020-11-16T21:44:37.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:44:37.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:44:37.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:44:37.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:44:37.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:44:37.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:44:37.565 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:44:37.565 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:44:37.565 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:44:37.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:44:37.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:44:37.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:44:37.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:44:37.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:44:37.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:44:37.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:44:37.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:44:37.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:44:37.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:44:37.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:44:37.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:44:37.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2020-11-16T21:44:37.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:44:37.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=53fbdde4-bae2-46c5-8dcb-e4e201b5420e 2020-11-16T21:44:37.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 53fbdde4-bae2-46c5-8dcb-e4e201b5420e' 2020-11-16T21:44:37.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:44:37.568 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 53fbdde4-bae2-46c5-8dcb-e4e201b5420e 2020-11-16T21:44:37.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDF8rJfwmVoIxAAijgT3ee1bZWn/J4ad4tl9w== 2020-11-16T21:44:37.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDF8rJfwmVoIxAAijgT3ee1bZWn/J4ad4tl9w=="}' 2020-11-16T21:44:37.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 53fbdde4-bae2-46c5-8dcb-e4e201b5420e -i td/osd-backfill-stats/1/new.json 2020-11-16T21:44:38.117 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:44:38.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2020-11-16T21:44:38.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDF8rJfwmVoIxAAijgT3ee1bZWn/J4ad4tl9w== --osd-uuid 53fbdde4-bae2-46c5-8dcb-e4e201b5420e 2020-11-16T21:44:38.162 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:38.168+0000 7f5c2f5f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:38.165 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:38.168+0000 7f5c2f5f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:38.166 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:38.172+0000 7f5c2f5f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:38.172+0000 7f5c2f5f3d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:38.172+0000 7f5c2f5f3d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:44:38.167 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:38.172+0000 7f5c2f5f3d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:44:38.225 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:38.232+0000 7f5c2f5f3d80 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2020-11-16T21:44:42.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2020-11-16T21:44:42.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:44:42.614 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:44:42.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:44:42.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:44:43.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:44:43.117 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:44:43.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:44:43.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:44:43.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:44:43.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:44:43.147 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:43.155+0000 7fbb5c70ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:43.149 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:43.155+0000 7fbb5c70ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:43.150 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:43.159+0000 7fbb5c70ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:43.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:44:43.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:44:43.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:44:43.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:44:43.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:44:43.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:44:43.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:44:43.491 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:44:43.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:44:43.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:44:43.688 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:43.695+0000 7fbb5c70ed80 -1 Falling back to public interface 2020-11-16T21:44:43.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:44:44.557 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:44.563+0000 7fbb5c70ed80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:44:44.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:44:44.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:44:44.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:44:44.875 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:44:44.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:44:44.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:44:45.273 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/494519,v1:127.0.0.1:6811/494519] [v2:127.0.0.1:6812/494519,v1:127.0.0.1:6813/494519] exists,up 53fbdde4-bae2-46c5-8dcb-e4e201b5420e 2020-11-16T21:44:45.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:44:45.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:44:45.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:44:45.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:625: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 2 2020-11-16T21:44:45.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:44:45.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:44:45.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:44:45.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:44:45.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2020-11-16T21:44:45.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:44:45.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:44:45.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:44:45.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:44:45.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2020-11-16T21:44:45.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2020-11-16T21:44:45.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:44:45.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:44:45.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:44:45.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:44:45.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:44:45.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:44:45.278 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:44:45.283 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:44:45.283 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:44:45.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:44:45.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:44:45.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:44:45.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:44:45.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:44:45.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:44:45.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:44:45.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:44:45.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:44:45.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:44:45.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:44:45.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:44:45.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2020-11-16T21:44:45.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:44:45.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2c9eea62-a28d-460a-b664-6b7782fe6855 2020-11-16T21:44:45.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 2c9eea62-a28d-460a-b664-6b7782fe6855' 2020-11-16T21:44:45.285 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 2c9eea62-a28d-460a-b664-6b7782fe6855 2020-11-16T21:44:45.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:44:45.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDN8rJf1I0EEhAA8W9gN2nus1D6DzphQ70SGQ== 2020-11-16T21:44:45.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDN8rJf1I0EEhAA8W9gN2nus1D6DzphQ70SGQ=="}' 2020-11-16T21:44:45.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2c9eea62-a28d-460a-b664-6b7782fe6855 -i td/osd-backfill-stats/2/new.json 2020-11-16T21:44:45.815 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:44:45.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2020-11-16T21:44:45.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDN8rJf1I0EEhAA8W9gN2nus1D6DzphQ70SGQ== --osd-uuid 2c9eea62-a28d-460a-b664-6b7782fe6855 2020-11-16T21:44:45.856 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:45.863+0000 7f370ba60d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:45.858 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:45.867+0000 7f370ba60d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:45.860 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:45.867+0000 7f370ba60d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:45.860 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:45.867+0000 7f370ba60d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:44:45.861 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:45.867+0000 7f370ba60d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:44:45.861 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:45.867+0000 7f370ba60d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:44:45.931 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:45.939+0000 7f370ba60d80 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2020-11-16T21:44:50.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2020-11-16T21:44:50.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:44:50.325 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:44:50.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:44:50.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:44:50.801 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:44:50.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:44:50.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:44:50.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:44:50.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:44:50.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:44:50.832 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:50.839+0000 7fc2c8fe8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:50.834 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:50.843+0000 7fc2c8fe8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:50.836 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:50.843+0000 7fc2c8fe8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:51.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:44:51.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:44:51.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:44:51.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:44:51.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:44:51.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:44:51.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:44:51.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:44:51.189 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:44:51.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:44:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:51.399+0000 7fc2c8fe8d80 -1 Falling back to public interface 2020-11-16T21:44:51.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:44:52.290 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:52.291+0000 7fc2c8fe8d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:44:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:44:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:44:52.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:44:52.634 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:44:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:44:52.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:44:53.032 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/495513,v1:127.0.0.1:6819/495513] [v2:127.0.0.1:6820/495513,v1:127.0.0.1:6821/495513] exists,up 2c9eea62-a28d-460a-b664-6b7782fe6855 2020-11-16T21:44:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:44:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:44:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:44:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:626: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 3 2020-11-16T21:44:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:44:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:44:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:44:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:44:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2020-11-16T21:44:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:44:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:44:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:44:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:44:53.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2020-11-16T21:44:53.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2020-11-16T21:44:53.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:44:53.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:44:53.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:44:53.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:44:53.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:44:53.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:44:53.037 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:44:53.037 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:44:53.037 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:44:53.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:44:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:44:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:44:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:44:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:44:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:44:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:44:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:44:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:44:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:44:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:44:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:44:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2020-11-16T21:44:53.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:44:53.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=67cdb084-65b0-41c3-b19f-354356f5abdb 2020-11-16T21:44:53.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 67cdb084-65b0-41c3-b19f-354356f5abdb' 2020-11-16T21:44:53.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:44:53.041 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 67cdb084-65b0-41c3-b19f-354356f5abdb 2020-11-16T21:44:53.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDV8rJf5E30AxAAKUSCCpUMHrqy7srp0JB0cA== 2020-11-16T21:44:53.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDV8rJf5E30AxAAKUSCCpUMHrqy7srp0JB0cA=="}' 2020-11-16T21:44:53.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 67cdb084-65b0-41c3-b19f-354356f5abdb -i td/osd-backfill-stats/3/new.json 2020-11-16T21:44:53.547 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:44:53.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2020-11-16T21:44:53.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDV8rJf5E30AxAAKUSCCpUMHrqy7srp0JB0cA== --osd-uuid 67cdb084-65b0-41c3-b19f-354356f5abdb 2020-11-16T21:44:53.592 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:53.599+0000 7f3765918d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:53.596 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:53.603+0000 7f3765918d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:53.598 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:53.607+0000 7f3765918d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:53.599 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:53.607+0000 7f3765918d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:44:53.599 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:53.607+0000 7f3765918d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:44:53.599 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:53.607+0000 7f3765918d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:44:53.650 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:53.659+0000 7f3765918d80 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2020-11-16T21:44:57.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2020-11-16T21:44:57.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:44:57.969 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:44:57.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:44:57.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:44:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:44:58.461 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:44:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:44:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:44:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:44:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:44:58.490 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:58.499+0000 7fa457bbcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:58.493 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:58.499+0000 7fa457bbcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:58.494 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:58.503+0000 7fa457bbcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:44:58.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:44:58.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:44:58.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:44:58.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:44:58.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:44:58.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:44:58.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:44:58.840 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:44:58.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:44:58.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:44:59.040 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:59.047+0000 7fa457bbcd80 -1 Falling back to public interface 2020-11-16T21:44:59.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:44:59.948 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:44:59.951+0000 7fa457bbcd80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:45:00.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:45:00.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:45:00.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:45:00.235 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:45:00.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:45:00.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:45:00.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:45:01.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:45:01.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:45:01.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:45:01.653 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:45:01.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:45:01.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:45:02.044 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/496507,v1:127.0.0.1:6827/496507] [v2:127.0.0.1:6828/496507,v1:127.0.0.1:6829/496507] exists,up 67cdb084-65b0-41c3-b19f-354356f5abdb 2020-11-16T21:45:02.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:45:02.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:45:02.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:45:02.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:627: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 4 2020-11-16T21:45:02.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:45:02.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:45:02.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:45:02.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:45:02.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2020-11-16T21:45:02.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:45:02.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:45:02.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:45:02.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:45:02.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2020-11-16T21:45:02.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2020-11-16T21:45:02.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:45:02.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:45:02.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:45:02.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:45:02.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:45:02.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:45:02.049 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:45:02.049 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:45:02.049 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:45:02.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:45:02.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:45:02.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:45:02.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:45:02.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:45:02.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:45:02.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:45:02.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:45:02.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:45:02.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:45:02.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:45:02.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:45:02.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2020-11-16T21:45:02.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:45:02.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7a54c164-2a6e-42a6-9d58-8d284601f592 2020-11-16T21:45:02.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 7a54c164-2a6e-42a6-9d58-8d284601f592' 2020-11-16T21:45:02.052 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 7a54c164-2a6e-42a6-9d58-8d284601f592 2020-11-16T21:45:02.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:45:02.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDe8rJfesZUBBAAEXgeTbbEgVyEAzey+jYV/w== 2020-11-16T21:45:02.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDe8rJfesZUBBAAEXgeTbbEgVyEAzey+jYV/w=="}' 2020-11-16T21:45:02.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7a54c164-2a6e-42a6-9d58-8d284601f592 -i td/osd-backfill-stats/4/new.json 2020-11-16T21:45:02.549 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:45:02.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2020-11-16T21:45:02.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDe8rJfesZUBBAAEXgeTbbEgVyEAzey+jYV/w== --osd-uuid 7a54c164-2a6e-42a6-9d58-8d284601f592 2020-11-16T21:45:02.593 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:02.599+0000 7f4430a68d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:02.627 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:02.635+0000 7f4430a68d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:02.628 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:02.635+0000 7f4430a68d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:02.628 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:02.635+0000 7f4430a68d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:45:02.628 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:02.635+0000 7f4430a68d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:45:02.629 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:02.635+0000 7f4430a68d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:45:02.685 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:02.691+0000 7f4430a68d80 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2020-11-16T21:45:07.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2020-11-16T21:45:07.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:45:07.328 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:45:07.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:45:07.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:45:07.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:45:07.833 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:45:07.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:45:07.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:45:07.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:45:07.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:45:07.862 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:07.871+0000 7fc0c9eb0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:07.865 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:07.875+0000 7fc0c9eb0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:07.867 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:07.875+0000 7fc0c9eb0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:08.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:45:08.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:45:08.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:45:08.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:45:08.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:45:08.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:45:08.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:45:08.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:45:08.225 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:45:08.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:45:08.415 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:08.423+0000 7fc0c9eb0d80 -1 Falling back to public interface 2020-11-16T21:45:08.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:45:09.277 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:09.283+0000 7fc0c9eb0d80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:45:09.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:45:09.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:45:09.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:45:09.640 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:45:09.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:45:09.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:45:10.111 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/497530,v1:127.0.0.1:6835/497530] [v2:127.0.0.1:6836/497530,v1:127.0.0.1:6837/497530] exists,up 7a54c164-2a6e-42a6-9d58-8d284601f592 2020-11-16T21:45:10.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:45:10.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:45:10.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:45:10.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:628: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 5 2020-11-16T21:45:10.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:45:10.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:45:10.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T21:45:10.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:45:10.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2020-11-16T21:45:10.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:45:10.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:45:10.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:45:10.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:45:10.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2020-11-16T21:45:10.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2020-11-16T21:45:10.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:45:10.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:45:10.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:45:10.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:45:10.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:45:10.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:45:10.116 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:45:10.117 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:45:10.117 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:45:10.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:45:10.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:45:10.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:45:10.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:45:10.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:45:10.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:45:10.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:45:10.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:45:10.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:45:10.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:45:10.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:45:10.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:45:10.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2020-11-16T21:45:10.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:45:10.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=dc846c9a-f1d4-4057-bc8b-7635c1a7ee02 2020-11-16T21:45:10.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 dc846c9a-f1d4-4057-bc8b-7635c1a7ee02' 2020-11-16T21:45:10.121 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 dc846c9a-f1d4-4057-bc8b-7635c1a7ee02 2020-11-16T21:45:10.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:45:10.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDm8rJfLYafCBAAnC+MBnDRgpVtSrp+wYzP3Q== 2020-11-16T21:45:10.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDm8rJfLYafCBAAnC+MBnDRgpVtSrp+wYzP3Q=="}' 2020-11-16T21:45:10.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new dc846c9a-f1d4-4057-bc8b-7635c1a7ee02 -i td/osd-backfill-stats/5/new.json 2020-11-16T21:45:10.666 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T21:45:10.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2020-11-16T21:45:10.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDm8rJfLYafCBAAnC+MBnDRgpVtSrp+wYzP3Q== --osd-uuid dc846c9a-f1d4-4057-bc8b-7635c1a7ee02 2020-11-16T21:45:10.714 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:10.723+0000 7faa33ff6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:10.718 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:10.727+0000 7faa33ff6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:10.719 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:10.727+0000 7faa33ff6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:10.719 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:10.727+0000 7faa33ff6d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:45:10.720 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:10.727+0000 7faa33ff6d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:45:10.720 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:10.727+0000 7faa33ff6d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:45:10.759 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:10.767+0000 7faa33ff6d80 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2020-11-16T21:45:15.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2020-11-16T21:45:15.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:45:15.160 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T21:45:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T21:45:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:45:15.660 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T21:45:15.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T21:45:15.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:45:15.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:45:15.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:45:15.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:45:15.690 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:15.699+0000 7f022cf62d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:15.694 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:15.703+0000 7f022cf62d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:15.695 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:15.703+0000 7f022cf62d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:45:16.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T21:45:16.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:45:16.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T21:45:16.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:45:16.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:45:16.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:45:16.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:45:16.084 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:45:16.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:45:16.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:45:16.231 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:16.239+0000 7f022cf62d80 -1 Falling back to public interface 2020-11-16T21:45:16.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:45:17.101 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:45:17.107+0000 7f022cf62d80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T21:45:17.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:45:17.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:45:17.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:45:17.451 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:45:17.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:45:17.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:45:17.897 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/498524,v1:127.0.0.1:6843/498524] [v2:127.0.0.1:6844/498524,v1:127.0.0.1:6845/498524] exists,up dc846c9a-f1d4-4057-bc8b-7635c1a7ee02 2020-11-16T21:45:17.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:45:17.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:45:17.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:45:17.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:630: TEST_backfill_ec_down_all_out: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2020-11-16T21:45:18.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:631: TEST_backfill_ec_down_all_out: create_pool test 1 1 erasure myprofile 2020-11-16T21:45:18.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-11-16T21:45:19.470 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T21:45:19.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:45:20.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:632: TEST_backfill_ec_down_all_out: ceph osd pool set test min_size 2 2020-11-16T21:45:21.322 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 min_size to 2 2020-11-16T21:45:21.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:634: TEST_backfill_ec_down_all_out: wait_for_clean 2020-11-16T21:45:21.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:45:21.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:45:21.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:45:21.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:45:21.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:45:21.335 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:45:21.335 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:45:21.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:45:21.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:45:21.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:45:21.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:45:21.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:45:21.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:45:21.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:45:21.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:45:21.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:45:21.937 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:45:21.937 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:45:21.938 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:45:21.938 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:45:21.938 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:45:21.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:45:21.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:45:21.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:45:22.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2020-11-16T21:45:22.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2020-11-16T21:45:22.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2020-11-16T21:45:22.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:45:22.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:45:22.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672970 2020-11-16T21:45:22.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672970 2020-11-16T21:45:22.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2020-11-16T21:45:22.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:45:22.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:45:22.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542153 2020-11-16T21:45:22.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542153 2020-11-16T21:45:22.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153' 2020-11-16T21:45:22.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:45:22.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:45:22.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411336 2020-11-16T21:45:22.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411336 2020-11-16T21:45:22.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-77309411336' 2020-11-16T21:45:22.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:45:22.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:45:22.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247813 2020-11-16T21:45:22.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247813 2020-11-16T21:45:22.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-77309411336 4-98784247813' 2020-11-16T21:45:22.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:45:22.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:45:23.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084291 2020-11-16T21:45:23.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084291 2020-11-16T21:45:23.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-77309411336 4-98784247813 5-120259084291' 2020-11-16T21:45:23.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:45:23.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2020-11-16T21:45:23.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:45:23.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:45:23.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2020-11-16T21:45:23.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:45:23.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2020-11-16T21:45:23.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2020-11-16T21:45:23.008 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836491 2020-11-16T21:45:23.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:45:23.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836489 -lt 21474836491 2020-11-16T21:45:23.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:45:24.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:45:24.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:45:24.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836491 2020-11-16T21:45:24.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:45:24.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:45:24.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2020-11-16T21:45:24.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:45:24.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672970 2020-11-16T21:45:24.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:45:24.811 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672970 2020-11-16T21:45:24.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672970 2020-11-16T21:45:24.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2020-11-16T21:45:24.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:45:25.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672970 -lt 42949672970 2020-11-16T21:45:25.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:45:25.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542153 2020-11-16T21:45:25.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:45:25.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:45:25.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542153 2020-11-16T21:45:25.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:45:25.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542153 2020-11-16T21:45:25.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2020-11-16T21:45:25.221 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542153 2020-11-16T21:45:25.221 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:45:25.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542154 -lt 60129542153 2020-11-16T21:45:25.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:45:25.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-77309411336 2020-11-16T21:45:25.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:45:25.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:45:25.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411336 2020-11-16T21:45:25.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:45:25.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411336 2020-11-16T21:45:25.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 77309411336' 2020-11-16T21:45:25.598 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 77309411336 2020-11-16T21:45:25.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:45:25.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411336 -lt 77309411336 2020-11-16T21:45:25.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:45:25.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247813 2020-11-16T21:45:25.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:45:25.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:45:26.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247813 2020-11-16T21:45:26.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:45:26.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247813 2020-11-16T21:45:26.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247813' 2020-11-16T21:45:26.002 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247813 2020-11-16T21:45:26.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:45:26.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247814 -lt 98784247813 2020-11-16T21:45:26.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:45:26.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084291 2020-11-16T21:45:26.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:45:26.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:45:26.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:45:26.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084291 2020-11-16T21:45:26.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084291 2020-11-16T21:45:26.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084291' 2020-11-16T21:45:26.380 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 120259084291 2020-11-16T21:45:26.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:45:26.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084292 -lt 120259084291 2020-11-16T21:45:26.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:45:26.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:45:26.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:45:27.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:45:27.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:45:27.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:45:27.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:45:27.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:45:27.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:45:27.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:45:27.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:45:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:45:27.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:45:27.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:45:27.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:45:28.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:45:28.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:45:28.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:45:28.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: seq 1 200 2020-11-16T21:45:28.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:28.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj1 /dev/null 2020-11-16T21:45:28.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:28.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj2 /dev/null 2020-11-16T21:45:28.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:28.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj3 /dev/null 2020-11-16T21:45:28.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:28.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj4 /dev/null 2020-11-16T21:45:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj5 /dev/null 2020-11-16T21:45:29.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:29.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj6 /dev/null 2020-11-16T21:45:29.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:29.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj7 /dev/null 2020-11-16T21:45:29.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:29.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj8 /dev/null 2020-11-16T21:45:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:29.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj9 /dev/null 2020-11-16T21:45:29.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:29.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj10 /dev/null 2020-11-16T21:45:29.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:29.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj11 /dev/null 2020-11-16T21:45:29.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:29.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj12 /dev/null 2020-11-16T21:45:30.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:30.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj13 /dev/null 2020-11-16T21:45:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj14 /dev/null 2020-11-16T21:45:30.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:30.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj15 /dev/null 2020-11-16T21:45:30.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:30.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj16 /dev/null 2020-11-16T21:45:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:30.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj17 /dev/null 2020-11-16T21:45:30.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:30.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj18 /dev/null 2020-11-16T21:45:31.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:31.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj19 /dev/null 2020-11-16T21:45:31.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:31.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj20 /dev/null 2020-11-16T21:45:31.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:31.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj21 /dev/null 2020-11-16T21:45:31.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:31.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj22 /dev/null 2020-11-16T21:45:31.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj23 /dev/null 2020-11-16T21:45:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj24 /dev/null 2020-11-16T21:45:32.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:32.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj25 /dev/null 2020-11-16T21:45:32.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:32.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj26 /dev/null 2020-11-16T21:45:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj27 /dev/null 2020-11-16T21:45:32.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:32.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj28 /dev/null 2020-11-16T21:45:33.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:33.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj29 /dev/null 2020-11-16T21:45:33.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:33.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj30 /dev/null 2020-11-16T21:45:33.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:33.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj31 /dev/null 2020-11-16T21:45:33.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:33.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj32 /dev/null 2020-11-16T21:45:33.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:33.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj33 /dev/null 2020-11-16T21:45:33.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:33.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj34 /dev/null 2020-11-16T21:45:34.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:34.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj35 /dev/null 2020-11-16T21:45:34.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:34.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj36 /dev/null 2020-11-16T21:45:34.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:34.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj37 /dev/null 2020-11-16T21:45:34.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:34.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj38 /dev/null 2020-11-16T21:45:34.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:34.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj39 /dev/null 2020-11-16T21:45:35.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:35.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj40 /dev/null 2020-11-16T21:45:35.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:35.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj41 /dev/null 2020-11-16T21:45:35.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:35.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj42 /dev/null 2020-11-16T21:45:35.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:35.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj43 /dev/null 2020-11-16T21:45:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj44 /dev/null 2020-11-16T21:45:35.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:35.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj45 /dev/null 2020-11-16T21:45:36.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:36.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj46 /dev/null 2020-11-16T21:45:36.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:36.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj47 /dev/null 2020-11-16T21:45:36.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:36.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj48 /dev/null 2020-11-16T21:45:36.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:36.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj49 /dev/null 2020-11-16T21:45:36.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:36.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj50 /dev/null 2020-11-16T21:45:37.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:37.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj51 /dev/null 2020-11-16T21:45:37.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:37.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj52 /dev/null 2020-11-16T21:45:37.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:37.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj53 /dev/null 2020-11-16T21:45:37.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:37.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj54 /dev/null 2020-11-16T21:45:37.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:37.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj55 /dev/null 2020-11-16T21:45:37.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:37.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj56 /dev/null 2020-11-16T21:45:38.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:38.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj57 /dev/null 2020-11-16T21:45:38.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:38.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj58 /dev/null 2020-11-16T21:45:38.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:38.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj59 /dev/null 2020-11-16T21:45:38.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:38.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj60 /dev/null 2020-11-16T21:45:38.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:38.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj61 /dev/null 2020-11-16T21:45:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:38.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj62 /dev/null 2020-11-16T21:45:39.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:39.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj63 /dev/null 2020-11-16T21:45:39.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:39.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj64 /dev/null 2020-11-16T21:45:39.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:39.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj65 /dev/null 2020-11-16T21:45:39.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:39.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj66 /dev/null 2020-11-16T21:45:39.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:39.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj67 /dev/null 2020-11-16T21:45:39.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:39.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj68 /dev/null 2020-11-16T21:45:40.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:40.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj69 /dev/null 2020-11-16T21:45:40.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:40.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj70 /dev/null 2020-11-16T21:45:40.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:40.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj71 /dev/null 2020-11-16T21:45:40.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:40.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj72 /dev/null 2020-11-16T21:45:40.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:40.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj73 /dev/null 2020-11-16T21:45:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj74 /dev/null 2020-11-16T21:45:41.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:41.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj75 /dev/null 2020-11-16T21:45:41.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:41.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj76 /dev/null 2020-11-16T21:45:41.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:41.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj77 /dev/null 2020-11-16T21:45:41.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:41.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj78 /dev/null 2020-11-16T21:45:41.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:41.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj79 /dev/null 2020-11-16T21:45:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:41.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj80 /dev/null 2020-11-16T21:45:42.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:42.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj81 /dev/null 2020-11-16T21:45:42.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:42.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj82 /dev/null 2020-11-16T21:45:42.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:42.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj83 /dev/null 2020-11-16T21:45:42.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:42.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj84 /dev/null 2020-11-16T21:45:42.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:42.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj85 /dev/null 2020-11-16T21:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:42.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj86 /dev/null 2020-11-16T21:45:43.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:43.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj87 /dev/null 2020-11-16T21:45:43.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:43.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj88 /dev/null 2020-11-16T21:45:43.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:43.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj89 /dev/null 2020-11-16T21:45:43.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:43.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj90 /dev/null 2020-11-16T21:45:43.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:43.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj91 /dev/null 2020-11-16T21:45:44.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:44.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj92 /dev/null 2020-11-16T21:45:44.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:44.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj93 /dev/null 2020-11-16T21:45:44.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:44.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj94 /dev/null 2020-11-16T21:45:44.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:44.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj95 /dev/null 2020-11-16T21:45:44.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:44.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj96 /dev/null 2020-11-16T21:45:44.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:44.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj97 /dev/null 2020-11-16T21:45:45.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:45.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj98 /dev/null 2020-11-16T21:45:45.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:45.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj99 /dev/null 2020-11-16T21:45:45.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:45.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj100 /dev/null 2020-11-16T21:45:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj101 /dev/null 2020-11-16T21:45:45.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:45.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj102 /dev/null 2020-11-16T21:45:45.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:45.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj103 /dev/null 2020-11-16T21:45:46.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:46.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj104 /dev/null 2020-11-16T21:45:46.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:46.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj105 /dev/null 2020-11-16T21:45:46.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:46.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj106 /dev/null 2020-11-16T21:45:46.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:46.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj107 /dev/null 2020-11-16T21:45:46.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:46.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj108 /dev/null 2020-11-16T21:45:46.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:46.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj109 /dev/null 2020-11-16T21:45:47.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:47.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj110 /dev/null 2020-11-16T21:45:47.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:47.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj111 /dev/null 2020-11-16T21:45:47.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:47.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj112 /dev/null 2020-11-16T21:45:47.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:47.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj113 /dev/null 2020-11-16T21:45:47.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:47.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj114 /dev/null 2020-11-16T21:45:47.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:47.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj115 /dev/null 2020-11-16T21:45:48.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:48.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj116 /dev/null 2020-11-16T21:45:48.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj117 /dev/null 2020-11-16T21:45:48.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:48.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj118 /dev/null 2020-11-16T21:45:48.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:48.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj119 /dev/null 2020-11-16T21:45:48.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:48.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj120 /dev/null 2020-11-16T21:45:48.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:48.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj121 /dev/null 2020-11-16T21:45:49.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:49.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj122 /dev/null 2020-11-16T21:45:49.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:49.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj123 /dev/null 2020-11-16T21:45:49.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:49.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj124 /dev/null 2020-11-16T21:45:49.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:49.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj125 /dev/null 2020-11-16T21:45:49.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:49.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj126 /dev/null 2020-11-16T21:45:49.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:49.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj127 /dev/null 2020-11-16T21:45:50.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:50.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj128 /dev/null 2020-11-16T21:45:50.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:50.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj129 /dev/null 2020-11-16T21:45:50.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:50.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj130 /dev/null 2020-11-16T21:45:50.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:50.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj131 /dev/null 2020-11-16T21:45:50.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:50.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj132 /dev/null 2020-11-16T21:45:50.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:50.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj133 /dev/null 2020-11-16T21:45:51.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:51.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj134 /dev/null 2020-11-16T21:45:51.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:51.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj135 /dev/null 2020-11-16T21:45:51.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:51.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj136 /dev/null 2020-11-16T21:45:51.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:51.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj137 /dev/null 2020-11-16T21:45:51.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:51.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj138 /dev/null 2020-11-16T21:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj139 /dev/null 2020-11-16T21:45:52.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:52.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj140 /dev/null 2020-11-16T21:45:52.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:52.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj141 /dev/null 2020-11-16T21:45:52.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:52.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj142 /dev/null 2020-11-16T21:45:52.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:52.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj143 /dev/null 2020-11-16T21:45:52.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:52.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj144 /dev/null 2020-11-16T21:45:53.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:53.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj145 /dev/null 2020-11-16T21:45:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:53.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj146 /dev/null 2020-11-16T21:45:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj147 /dev/null 2020-11-16T21:45:53.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:53.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj148 /dev/null 2020-11-16T21:45:53.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:53.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj149 /dev/null 2020-11-16T21:45:53.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:53.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj150 /dev/null 2020-11-16T21:45:54.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:54.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj151 /dev/null 2020-11-16T21:45:54.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:54.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj152 /dev/null 2020-11-16T21:45:54.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:54.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj153 /dev/null 2020-11-16T21:45:54.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:54.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj154 /dev/null 2020-11-16T21:45:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj155 /dev/null 2020-11-16T21:45:54.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:54.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj156 /dev/null 2020-11-16T21:45:55.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:55.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj157 /dev/null 2020-11-16T21:45:55.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:55.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj158 /dev/null 2020-11-16T21:45:55.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:55.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj159 /dev/null 2020-11-16T21:45:55.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:55.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj160 /dev/null 2020-11-16T21:45:55.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:55.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj161 /dev/null 2020-11-16T21:45:56.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:56.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj162 /dev/null 2020-11-16T21:45:56.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:56.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj163 /dev/null 2020-11-16T21:45:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:56.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj164 /dev/null 2020-11-16T21:45:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj165 /dev/null 2020-11-16T21:45:56.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:56.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj166 /dev/null 2020-11-16T21:45:56.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:56.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj167 /dev/null 2020-11-16T21:45:57.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:57.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj168 /dev/null 2020-11-16T21:45:57.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:57.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj169 /dev/null 2020-11-16T21:45:57.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:57.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj170 /dev/null 2020-11-16T21:45:57.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:57.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj171 /dev/null 2020-11-16T21:45:57.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:57.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj172 /dev/null 2020-11-16T21:45:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:57.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj173 /dev/null 2020-11-16T21:45:58.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:58.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj174 /dev/null 2020-11-16T21:45:58.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:58.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj175 /dev/null 2020-11-16T21:45:58.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:58.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj176 /dev/null 2020-11-16T21:45:58.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:58.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj177 /dev/null 2020-11-16T21:45:58.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:58.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj178 /dev/null 2020-11-16T21:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:58.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj179 /dev/null 2020-11-16T21:45:59.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:59.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj180 /dev/null 2020-11-16T21:45:59.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:59.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj181 /dev/null 2020-11-16T21:45:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj182 /dev/null 2020-11-16T21:45:59.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:59.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj183 /dev/null 2020-11-16T21:45:59.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:59.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj184 /dev/null 2020-11-16T21:45:59.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:45:59.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj185 /dev/null 2020-11-16T21:46:00.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:00.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj186 /dev/null 2020-11-16T21:46:00.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:00.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj187 /dev/null 2020-11-16T21:46:00.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:00.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj188 /dev/null 2020-11-16T21:46:00.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:00.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj189 /dev/null 2020-11-16T21:46:00.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:00.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj190 /dev/null 2020-11-16T21:46:01.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:01.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj191 /dev/null 2020-11-16T21:46:01.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:01.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj192 /dev/null 2020-11-16T21:46:01.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:01.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj193 /dev/null 2020-11-16T21:46:01.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:01.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj194 /dev/null 2020-11-16T21:46:01.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:01.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj195 /dev/null 2020-11-16T21:46:01.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:01.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj196 /dev/null 2020-11-16T21:46:02.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:02.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj197 /dev/null 2020-11-16T21:46:02.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:02.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj198 /dev/null 2020-11-16T21:46:02.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:02.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj199 /dev/null 2020-11-16T21:46:02.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2020-11-16T21:46:02.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj200 /dev/null 2020-11-16T21:46:02.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:641: TEST_backfill_ec_down_all_out: get_pg test obj1 2020-11-16T21:46:02.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T21:46:02.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T21:46:02.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T21:46:02.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T21:46:03.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:641: TEST_backfill_ec_down_all_out: local PG=1.0 2020-11-16T21:46:03.230 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:643: TEST_backfill_ec_down_all_out: get_primary test obj1 2020-11-16T21:46:03.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:46:03.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:46:03.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:46:03.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:46:03.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:643: TEST_backfill_ec_down_all_out: local primary=1 2020-11-16T21:46:03.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:644: TEST_backfill_ec_down_all_out: get_not_primary test obj1 2020-11-16T21:46:03.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2020-11-16T21:46:03.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2020-11-16T21:46:03.645 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2020-11-16T21:46:03.645 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:46:03.646 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:46:03.646 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:46:03.646 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:46:04.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T21:46:04.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T21:46:04.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2020-11-16T21:46:04.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:644: TEST_backfill_ec_down_all_out: local otherosd=0 2020-11-16T21:46:04.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:645: TEST_backfill_ec_down_all_out: get_osds test obj1 2020-11-16T21:46:04.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test 2020-11-16T21:46:04.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj1 2020-11-16T21:46:04.428 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-11-16T21:46:04.429 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test obj1 2020-11-16T21:46:04.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=1 2020-11-16T21:46:04.826 INFO:tasks.workunit.client.0.smithi189.stderr:0 2020-11-16T21:46:04.826 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T21:46:04.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 1 0 2 2020-11-16T21:46:04.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:645: TEST_backfill_ec_down_all_out: local 'allosds=1 0 2' 2020-11-16T21:46:04.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:647: TEST_backfill_ec_down_all_out: ceph osd set nobackfill 2020-11-16T21:46:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T21:46:05.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:648: TEST_backfill_ec_down_all_out: cat td/osd-backfill-stats/osd.0.pid 2020-11-16T21:46:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:648: TEST_backfill_ec_down_all_out: kill 493525 2020-11-16T21:46:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:649: TEST_backfill_ec_down_all_out: ceph osd down osd.0 2020-11-16T21:46:05.990 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already down. 2020-11-16T21:46:06.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:650: TEST_backfill_ec_down_all_out: for o in $allosds 2020-11-16T21:46:06.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:652: TEST_backfill_ec_down_all_out: ceph osd out osd.1 2020-11-16T21:46:06.610 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. 2020-11-16T21:46:06.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:650: TEST_backfill_ec_down_all_out: for o in $allosds 2020-11-16T21:46:06.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:652: TEST_backfill_ec_down_all_out: ceph osd out osd.0 2020-11-16T21:46:07.302 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T21:46:07.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:650: TEST_backfill_ec_down_all_out: for o in $allosds 2020-11-16T21:46:07.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:652: TEST_backfill_ec_down_all_out: ceph osd out osd.2 2020-11-16T21:46:07.942 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already out. 2020-11-16T21:46:07.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:655: TEST_backfill_ec_down_all_out: sleep 2 2020-11-16T21:46:09.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:656: TEST_backfill_ec_down_all_out: get_primary test obj1 2020-11-16T21:46:09.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:46:09.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:46:09.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:46:09.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:46:10.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:656: TEST_backfill_ec_down_all_out: primary=1 2020-11-16T21:46:10.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:657: TEST_backfill_ec_down_all_out: ceph osd unset nobackfill 2020-11-16T21:46:11.060 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T21:46:11.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:658: TEST_backfill_ec_down_all_out: ceph tell osd.1 get_latest_osdmap 2020-11-16T21:46:11.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:659: TEST_backfill_ec_down_all_out: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T21:46:11.384 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T21:46:11.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:660: TEST_backfill_ec_down_all_out: sleep 2 2020-11-16T21:46:13.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:661: TEST_backfill_ec_down_all_out: flush_pg_stats 2020-11-16T21:46:13.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:46:13.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:46:13.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:46:13.792 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:46:13.792 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:46:13.793 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:46:13.793 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:46:13.793 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:46:13.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:46:13.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:46:13.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:46:13.914 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.0 2020-11-16T21:46:13.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T21:46:13.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T21:46:13.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T21:46:13.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:46:13.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:46:14.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672982 2020-11-16T21:46:14.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672982 2020-11-16T21:46:14.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982' 2020-11-16T21:46:14.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:46:14.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:46:14.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542166 2020-11-16T21:46:14.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542166 2020-11-16T21:46:14.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-60129542166' 2020-11-16T21:46:14.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:46:14.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:46:14.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411348 2020-11-16T21:46:14.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411348 2020-11-16T21:46:14.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-60129542166 3-77309411348' 2020-11-16T21:46:14.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:46:14.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:46:14.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247825 2020-11-16T21:46:14.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247825 2020-11-16T21:46:14.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-60129542166 3-77309411348 4-98784247825' 2020-11-16T21:46:14.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:46:14.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:46:14.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084304 2020-11-16T21:46:14.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084304 2020-11-16T21:46:14.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-60129542166 3-77309411348 4-98784247825 5-120259084304' 2020-11-16T21:46:14.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:46:14.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672982 2020-11-16T21:46:14.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:46:14.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:46:14.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672982 2020-11-16T21:46:14.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:46:14.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672982 2020-11-16T21:46:14.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2020-11-16T21:46:14.748 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672982 2020-11-16T21:46:14.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:46:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672980 -lt 42949672982 2020-11-16T21:46:15.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:46:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672982 -lt 42949672982 2020-11-16T21:46:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:46:16.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542166 2020-11-16T21:46:16.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:46:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:46:16.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542166 2020-11-16T21:46:16.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:46:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542166 2020-11-16T21:46:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542166' 2020-11-16T21:46:16.560 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542166 2020-11-16T21:46:16.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:46:16.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542166 -lt 60129542166 2020-11-16T21:46:16.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:46:16.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-77309411348 2020-11-16T21:46:17.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:46:17.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:46:17.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411348 2020-11-16T21:46:17.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:46:17.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411348 2020-11-16T21:46:17.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 77309411348' 2020-11-16T21:46:17.005 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 77309411348 2020-11-16T21:46:17.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:46:17.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411348 -lt 77309411348 2020-11-16T21:46:17.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:46:17.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247825 2020-11-16T21:46:17.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:46:17.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:46:17.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247825 2020-11-16T21:46:17.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:46:17.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247825 2020-11-16T21:46:17.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247825' 2020-11-16T21:46:17.398 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247825 2020-11-16T21:46:17.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:46:17.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247826 -lt 98784247825 2020-11-16T21:46:17.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:46:17.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084304 2020-11-16T21:46:17.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:46:17.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:46:17.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084304 2020-11-16T21:46:17.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:46:17.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084304 2020-11-16T21:46:17.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084304' 2020-11-16T21:46:17.774 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 120259084304 2020-11-16T21:46:17.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:46:18.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084304 -lt 120259084304 2020-11-16T21:46:18.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:666: TEST_backfill_ec_down_all_out: true 2020-11-16T21:46:18.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:669: TEST_backfill_ec_down_all_out: ceph --format json pg dump pgs 2020-11-16T21:46:18.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:670: TEST_backfill_ec_down_all_out: jq '.pg_stats | [.[] | .state | select(. == "incomplete")] | length' 2020-11-16T21:46:18.481 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:18.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:669: TEST_backfill_ec_down_all_out: test 0 -ne 0 2020-11-16T21:46:18.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:674: TEST_backfill_ec_down_all_out: break 2020-11-16T21:46:18.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:676: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:18.854 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:46:22.978 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 186 372 0 0 0 0 100 100 active+undersized+degraded+remapped+backfilling 2020-11-16T21:46:08.717878+0000 36'200 48:299 [4,3,5] 4 [1,NONE,2] 1 0'0 2020-11-16T21:45:19.103640+0000 0'0 2020-11-16T21:45:19.103640+0000 0 2020-11-16T21:46:22.978 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:46:22.979 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:46:22.980 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:22.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: seq 1 60 2020-11-16T21:46:22.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:22.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:22.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:22.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:22.982 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:22.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 1 = 60 ']' 2020-11-16T21:46:22.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:22.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:22.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:22.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:22.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:22.983 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:22.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 2 = 60 ']' 2020-11-16T21:46:22.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:22.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:22.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:22.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:22.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:22.984 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:22.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 3 = 60 ']' 2020-11-16T21:46:22.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:23.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:23.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:23.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:23.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:23.392 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:23.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 4 = 60 ']' 2020-11-16T21:46:23.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:24.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:24.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:24.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:24.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:24.787 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:24.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 5 = 60 ']' 2020-11-16T21:46:24.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:25.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:25.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:25.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:25.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:26.184 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:26.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 6 = 60 ']' 2020-11-16T21:46:26.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:27.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:27.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:27.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:27.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:27.578 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:27.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 7 = 60 ']' 2020-11-16T21:46:27.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:28.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:28.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:28.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:28.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:28.970 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:28.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 8 = 60 ']' 2020-11-16T21:46:28.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:29.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:29.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:30.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 9 = 60 ']' 2020-11-16T21:46:30.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:31.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:31.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:31.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:31.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:31.745 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:31.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 10 = 60 ']' 2020-11-16T21:46:31.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:32.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:32.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:32.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:32.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:33.128 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:33.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 11 = 60 ']' 2020-11-16T21:46:33.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:34.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:34.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:34.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:34.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:34.517 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:34.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 12 = 60 ']' 2020-11-16T21:46:34.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:35.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:35.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:35.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:35.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:35.889 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:35.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 13 = 60 ']' 2020-11-16T21:46:35.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:36.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:36.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:37.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 14 = 60 ']' 2020-11-16T21:46:37.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:38.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:38.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:38.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:38.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:38.668 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:38.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 15 = 60 ']' 2020-11-16T21:46:38.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:39.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:39.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:39.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:39.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:40.062 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:40.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 16 = 60 ']' 2020-11-16T21:46:40.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:41.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:41.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:41.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:41.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:41.456 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 17 = 60 ']' 2020-11-16T21:46:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:42.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:42.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:42.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:42.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:42.848 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 18 = 60 ']' 2020-11-16T21:46:42.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:43.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:43.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:43.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:43.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:44.237 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:44.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 19 = 60 ']' 2020-11-16T21:46:44.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:45.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:45.630 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:45.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 20 = 60 ']' 2020-11-16T21:46:45.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:46.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:46.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:46.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:46.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:47.019 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:47.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 21 = 60 ']' 2020-11-16T21:46:47.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:48.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:48.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:48.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:48.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:48.412 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:48.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 22 = 60 ']' 2020-11-16T21:46:48.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:49.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:49.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:49.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:49.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:49.799 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:49.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 23 = 60 ']' 2020-11-16T21:46:49.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:50.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:50.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:50.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:50.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:51.185 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:51.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 24 = 60 ']' 2020-11-16T21:46:51.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:52.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:52.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:52.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:52.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:52.560 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:52.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 25 = 60 ']' 2020-11-16T21:46:52.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:53.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:53.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:53.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:53.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:53.943 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:53.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 26 = 60 ']' 2020-11-16T21:46:53.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:54.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:54.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:54.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:54.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:55.320 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:55.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 27 = 60 ']' 2020-11-16T21:46:55.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:56.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:56.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:56.707 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:56.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 28 = 60 ']' 2020-11-16T21:46:56.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:57.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:57.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:57.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:57.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:58.112 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:58.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 29 = 60 ']' 2020-11-16T21:46:58.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:46:59.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:46:59.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:46:59.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:46:59.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:46:59.507 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:46:59.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 30 = 60 ']' 2020-11-16T21:46:59.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:47:00.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:47:00.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:47:00.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:47:00.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:47:00.884 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:47:00.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 31 = 60 ']' 2020-11-16T21:47:00.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:47:01.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:47:01.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:47:01.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:47:01.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:47:02.318 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:47:02.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 32 = 60 ']' 2020-11-16T21:47:02.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:47:03.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:47:03.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:47:03.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:47:03.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:47:03.700 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:47:03.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 33 = 60 ']' 2020-11-16T21:47:03.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:47:04.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:47:04.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:47:04.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:47:04.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:47:05.098 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:47:05.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 34 = 60 ']' 2020-11-16T21:47:05.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:47:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:47:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:47:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:47:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:47:06.486 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:47:06.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 35 = 60 ']' 2020-11-16T21:47:06.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:47:07.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:47:07.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:47:07.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:47:07.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:47:07.875 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:47:07.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 36 = 60 ']' 2020-11-16T21:47:07.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:47:08.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:47:08.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:47:08.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:47:08.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:47:09.266 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:47:09.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 37 = 60 ']' 2020-11-16T21:47:09.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:47:10.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:47:10.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:47:10.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:47:10.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:47:10.649 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:47:10.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 38 = 60 ']' 2020-11-16T21:47:10.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2020-11-16T21:47:11.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2020-11-16T21:47:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:47:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2020-11-16T21:47:11.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2020-11-16T21:47:12.040 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:47:12.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:681: TEST_backfill_ec_down_all_out: break 2020-11-16T21:47:12.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:691: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2020-11-16T21:47:12.401 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T21:47:12.401 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 0 0 0 0 0 0 10 10 active+clean 2020-11-16T21:47:09.272384+0000 36'200 50:1051 [4,3,5] 4 [4,3,5] 4 0'0 2020-11-16T21:45:19.103640+0000 0'0 2020-11-16T21:45:19.103640+0000 0 2020-11-16T21:47:12.401 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T21:47:12.402 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T21:47:12.403 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T21:47:12.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:693: TEST_backfill_ec_down_all_out: expr 200 '*' 2 2020-11-16T21:47:12.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:693: TEST_backfill_ec_down_all_out: local misplaced=400 2020-11-16T21:47:12.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:694: TEST_backfill_ec_down_all_out: check td/osd-backfill-stats 1.0 1 erasure 200 0 400 0 2020-11-16T21:47:12.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2020-11-16T21:47:12.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2020-11-16T21:47:12.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2020-11-16T21:47:12.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=erasure 2020-11-16T21:47:12.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=200 2020-11-16T21:47:12.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2020-11-16T21:47:12.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=400 2020-11-16T21:47:12.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2020-11-16T21:47:12.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2020-11-16T21:47:12.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2020-11-16T21:47:12.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2020-11-16T21:47:12.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2020-11-16T21:47:12.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2020-11-16T21:47:12.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2020-11-16T21:47:12.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2020-11-16T21:47:12.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2020-11-16T21:47:12.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2020-11-16T21:47:12.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2020-11-16T21:47:12.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2020-11-16T21:47:12.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2020-11-16T21:47:12.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' erasure = erasure ']' 2020-11-16T21:47:12.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:89: check: addp=p 2020-11-16T21:47:12.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2020-11-16T21:47:12.455 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:47:12.455 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2020-11-16T21:47:12.455 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2020-11-16T21:47:12.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[4,3,5]/[1,NONE,2]' 2020-11-16T21:47:12.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[4,3,5]/[1,NONE,2]' ']' 2020-11-16T21:47:12.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2020-11-16T21:47:12.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:47:12.494 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:47:12.494 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [4,3,5]/[1,NONE,2]p' 2020-11-16T21:47:12.494 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2020-11-16T21:47:12.495 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2020-11-16T21:47:12.495 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:47:12.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=200 2020-11-16T21:47:12.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 200 ']' 2020-11-16T21:47:12.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 200 200 2020-11-16T21:47:12.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2020-11-16T21:47:12.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:47:12.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2020-11-16T21:47:12.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:47:12.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:47:12.508 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:47:12.509 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2020-11-16T21:47:12.509 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:47:12.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=1 2020-11-16T21:47:12.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 1 ']' 2020-11-16T21:47:12.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 1 0 2020-11-16T21:47:12.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2020-11-16T21:47:12.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:47:12.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:47:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:47:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2020-11-16T21:47:12.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:47:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:47:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [4,3,5]/[1,NONE,2]p' 2020-11-16T21:47:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2020-11-16T21:47:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2020-11-16T21:47:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:47:12.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=400 2020-11-16T21:47:12.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 400 ']' 2020-11-16T21:47:12.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 400 400 2020-11-16T21:47:12.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2020-11-16T21:47:12.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:47:12.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2020-11-16T21:47:12.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:47:12.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:47:12.538 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:47:12.538 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2020-11-16T21:47:12.538 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:47:12.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=2 2020-11-16T21:47:12.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 2 ']' 2020-11-16T21:47:12.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 2 0 2020-11-16T21:47:12.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=2 2020-11-16T21:47:12.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:47:12.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:47:12.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:47:12.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2020-11-16T21:47:12.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:696: TEST_backfill_ec_down_all_out: delete_pool test 2020-11-16T21:47:12.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T21:47:12.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T21:47:13.068 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T21:47:13.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:697: TEST_backfill_ec_down_all_out: kill_daemons td/osd-backfill-stats 2020-11-16T21:47:13.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:47:13.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:47:13.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:47:13.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:47:13.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:47:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:47:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2020-11-16T21:47:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:47:13.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:47:13.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:47:13.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:47:13.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:47:13.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:47:13.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:47:13.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:47:13.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:47:13.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:47:13.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:47:13.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:47:13.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:47:13.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:47:13.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:47:13.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:47:13.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:47:13.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:47:13.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:47:13.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:47:13.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:47:13.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:47:13.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:47:13.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:13.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:13.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:47:13.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:47:13.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:47:13.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2020-11-16T21:47:13.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2020-11-16T21:47:13.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2020-11-16T21:47:13.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2020-11-16T21:47:13.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:47:13.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:47:13.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:47:13.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:47:13.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:47:13.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:47:13.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:47:13.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:47:13.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:47:13.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:47:13.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:47:13.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:47:13.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:47:13.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:47:13.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:47:13.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:47:13.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:47:13.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:47:13.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:47:13.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:47:13.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:47:13.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:47:13.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:47:13.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:13.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:13.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:47:13.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:47:13.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:47:13.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2020-11-16T21:47:13.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:47:13.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:13.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:13.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.465868 2020-11-16T21:47:13.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:47:13.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:47:13.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:47:13.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2020-11-16T21:47:13.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_ec_down_out td/osd-backfill-stats 2020-11-16T21:47:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:708: TEST_backfill_ec_down_out: local dir=td/osd-backfill-stats 2020-11-16T21:47:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:710: TEST_backfill_ec_down_out: run_mon td/osd-backfill-stats a 2020-11-16T21:47:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2020-11-16T21:47:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:47:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:47:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:47:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2020-11-16T21:47:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2020-11-16T21:47:13.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:47:13.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:47:13.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:47:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:47:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:13.574 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:13.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:13.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:47:13.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:47:13.779 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:47:13.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:47:13.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:47:13.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:47:13.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:47:13.781 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:47:13.781 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:47:13.781 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:47:13.782 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:47:13.782 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:13.782 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:13.783 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:47:13.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:47:13.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get fsid 2020-11-16T21:47:13.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:47:13.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:47:13.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:47:13.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:47:13.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:47:13.891 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:47:13.891 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:47:13.891 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:47:13.891 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:47:13.892 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:13.892 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:13.892 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:47:13.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:47:13.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get mon_host 2020-11-16T21:47:13.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:711: TEST_backfill_ec_down_out: run_mgr td/osd-backfill-stats x 2020-11-16T21:47:13.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2020-11-16T21:47:13.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:47:13.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:47:13.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:47:13.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2020-11-16T21:47:13.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:47:14.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:47:14.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:47:14.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:47:14.507 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:47:14.508 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:14.508 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:14.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:14.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:47:14.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:47:14.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:712: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 0 2020-11-16T21:47:14.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:47:14.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:47:14.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:47:14.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:47:14.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2020-11-16T21:47:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:47:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:47:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:47:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:47:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2020-11-16T21:47:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2020-11-16T21:47:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:47:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:47:14.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:47:14.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:47:14.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:47:14.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:47:14.543 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:47:14.543 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:14.544 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:14.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:14.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:14.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:47:14.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:47:14.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:47:14.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:47:14.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:47:14.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:47:14.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:47:14.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:47:14.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:47:14.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:47:14.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2020-11-16T21:47:14.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:47:14.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a080a3f0-dd37-4265-833f-02add002447b 2020-11-16T21:47:14.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 a080a3f0-dd37-4265-833f-02add002447b' 2020-11-16T21:47:14.547 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 a080a3f0-dd37-4265-833f-02add002447b 2020-11-16T21:47:14.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:47:14.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBi87JfiN8UIhAAQdONjHn2Wl8hGSypw7OPYg== 2020-11-16T21:47:14.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBi87JfiN8UIhAAQdONjHn2Wl8hGSypw7OPYg=="}' 2020-11-16T21:47:14.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a080a3f0-dd37-4265-833f-02add002447b -i td/osd-backfill-stats/0/new.json 2020-11-16T21:47:14.906 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:47:14.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2020-11-16T21:47:14.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBi87JfiN8UIhAAQdONjHn2Wl8hGSypw7OPYg== --osd-uuid a080a3f0-dd37-4265-833f-02add002447b 2020-11-16T21:47:14.945 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:14.948+0000 7f7c54fbfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:14.947 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:14.952+0000 7f7c54fbfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:14.949 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:14.952+0000 7f7c54fbfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:14.949 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:14.952+0000 7f7c54fbfd80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:47:14.950 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:14.952+0000 7f7c54fbfd80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:47:14.950 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:14.952+0000 7f7c54fbfd80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:47:14.994 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:15.000+0000 7f7c54fbfd80 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2020-11-16T21:47:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2020-11-16T21:47:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:47:19.348 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:47:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:47:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:47:19.847 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:47:19.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:47:19.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:47:19.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:47:19.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:47:19.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:47:19.875 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:19.880+0000 7fa2aace3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:19.877 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:19.884+0000 7fa2aace3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:19.879 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:19.884+0000 7fa2aace3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:20.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:47:20.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:47:20.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:47:20.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:47:20.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:47:20.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:20.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:47:20.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:20.242 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:47:20.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:47:20.420 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:20.424+0000 7fa2aace3d80 -1 Falling back to public interface 2020-11-16T21:47:20.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:47:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:21.288+0000 7fa2aace3d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:47:21.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:47:21.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:21.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:47:21.644 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:47:21.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:21.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:47:22.045 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/506340,v1:127.0.0.1:6803/506340] [v2:127.0.0.1:6804/506340,v1:127.0.0.1:6805/506340] exists,up a080a3f0-dd37-4265-833f-02add002447b 2020-11-16T21:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:713: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 1 2020-11-16T21:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:47:22.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2020-11-16T21:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:47:22.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2020-11-16T21:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2020-11-16T21:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:47:22.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:47:22.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:22.050 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:47:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2020-11-16T21:47:22.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f4c4c2ec-f59b-4dc7-aafc-f84563c50fe6 2020-11-16T21:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 f4c4c2ec-f59b-4dc7-aafc-f84563c50fe6' 2020-11-16T21:47:22.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:47:22.053 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 f4c4c2ec-f59b-4dc7-aafc-f84563c50fe6 2020-11-16T21:47:22.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBq87Jf+0/IBBAAe3e1HISJ5ABLWe1qcOXdhw== 2020-11-16T21:47:22.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBq87Jf+0/IBBAAe3e1HISJ5ABLWe1qcOXdhw=="}' 2020-11-16T21:47:22.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f4c4c2ec-f59b-4dc7-aafc-f84563c50fe6 -i td/osd-backfill-stats/1/new.json 2020-11-16T21:47:22.639 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:47:22.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2020-11-16T21:47:22.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBq87Jf+0/IBBAAe3e1HISJ5ABLWe1qcOXdhw== --osd-uuid f4c4c2ec-f59b-4dc7-aafc-f84563c50fe6 2020-11-16T21:47:22.680 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:22.684+0000 7f02268a1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:22.683 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:22.688+0000 7f02268a1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:22.684 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:22.688+0000 7f02268a1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:22.685 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:22.692+0000 7f02268a1d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:47:22.686 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:22.692+0000 7f02268a1d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:47:22.686 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:22.692+0000 7f02268a1d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:47:22.772 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:22.780+0000 7f02268a1d80 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2020-11-16T21:47:27.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2020-11-16T21:47:27.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:47:27.078 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:47:27.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:47:27.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:47:27.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:47:27.570 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:47:27.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:47:27.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:47:27.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:47:27.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:47:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:27.604+0000 7f20541e0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:27.608+0000 7f20541e0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:27.608+0000 7f20541e0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:27.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:47:27.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:47:27.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:47:27.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:47:27.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:47:27.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:27.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:47:27.955 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:47:27.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:27.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:47:28.147 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:28.152+0000 7f20541e0d80 -1 Falling back to public interface 2020-11-16T21:47:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:47:29.075 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:29.076+0000 7f20541e0d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:47:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:47:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:47:29.341 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:47:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:47:29.745 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/507334,v1:127.0.0.1:6811/507334] [v2:127.0.0.1:6812/507334,v1:127.0.0.1:6813/507334] exists,up f4c4c2ec-f59b-4dc7-aafc-f84563c50fe6 2020-11-16T21:47:29.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:47:29.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:47:29.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:47:29.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:714: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 2 2020-11-16T21:47:29.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:47:29.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:47:29.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:47:29.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:47:29.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2020-11-16T21:47:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:47:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:47:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:47:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:47:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2020-11-16T21:47:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2020-11-16T21:47:29.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:47:29.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:47:29.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:47:29.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:47:29.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:47:29.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:47:29.750 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:47:29.750 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:29.750 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:29.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:29.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:29.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:47:29.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:47:29.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:47:29.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:47:29.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:47:29.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:47:29.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:47:29.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:47:29.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:47:29.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:47:29.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2020-11-16T21:47:29.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:47:29.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=aef59f8a-5c24-42b7-b302-6201d4dfcb57 2020-11-16T21:47:29.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 aef59f8a-5c24-42b7-b302-6201d4dfcb57' 2020-11-16T21:47:29.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:47:29.753 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 aef59f8a-5c24-42b7-b302-6201d4dfcb57 2020-11-16T21:47:29.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBx87JfP4pJLhAAOF3jbK78sjMcndBrDrZrnA== 2020-11-16T21:47:29.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBx87JfP4pJLhAAOF3jbK78sjMcndBrDrZrnA=="}' 2020-11-16T21:47:29.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new aef59f8a-5c24-42b7-b302-6201d4dfcb57 -i td/osd-backfill-stats/2/new.json 2020-11-16T21:47:30.320 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:47:30.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2020-11-16T21:47:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBx87JfP4pJLhAAOF3jbK78sjMcndBrDrZrnA== --osd-uuid aef59f8a-5c24-42b7-b302-6201d4dfcb57 2020-11-16T21:47:30.365 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:30.368+0000 7f7688deed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:30.368 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:30.372+0000 7f7688deed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:30.369 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:30.372+0000 7f7688deed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:30.370 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:30.376+0000 7f7688deed80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:47:30.370 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:30.376+0000 7f7688deed80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:47:30.370 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:30.376+0000 7f7688deed80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:47:30.443 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:30.452+0000 7f7688deed80 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2020-11-16T21:47:35.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2020-11-16T21:47:35.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:47:35.082 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:47:35.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:47:35.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:47:35.577 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:47:35.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:47:35.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:47:35.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:47:35.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:47:35.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:47:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:35.612+0000 7f2c4d3d3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:35.609 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:35.616+0000 7f2c4d3d3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:35.611 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:35.616+0000 7f2c4d3d3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:35.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:47:35.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:47:35.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:47:35.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:47:35.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:47:35.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:35.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:47:35.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:47:35.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:35.963 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:47:36.143 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:36.148+0000 7f2c4d3d3d80 -1 Falling back to public interface 2020-11-16T21:47:36.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:47:36.985 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:36.984+0000 7f2c4d3d3d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:47:37.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:47:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:47:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:37.367 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:47:37.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:47:37.784 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/508328,v1:127.0.0.1:6819/508328] [v2:127.0.0.1:6820/508328,v1:127.0.0.1:6821/508328] exists,up aef59f8a-5c24-42b7-b302-6201d4dfcb57 2020-11-16T21:47:37.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:47:37.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:47:37.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:47:37.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:715: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 3 2020-11-16T21:47:37.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:47:37.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:47:37.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:47:37.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:47:37.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2020-11-16T21:47:37.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:47:37.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:47:37.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:47:37.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:47:37.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2020-11-16T21:47:37.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2020-11-16T21:47:37.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:47:37.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:47:37.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:47:37.788 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:47:37.789 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:47:37.789 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:47:37.789 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:47:37.789 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:37.789 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:37.789 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:37.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:37.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:47:37.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:47:37.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:47:37.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:47:37.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:47:37.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:47:37.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:47:37.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:47:37.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:47:37.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:47:37.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2020-11-16T21:47:37.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:47:37.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c008bbef-3ee6-48d2-bad9-34f4550406d6 2020-11-16T21:47:37.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 c008bbef-3ee6-48d2-bad9-34f4550406d6' 2020-11-16T21:47:37.792 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 c008bbef-3ee6-48d2-bad9-34f4550406d6 2020-11-16T21:47:37.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:47:37.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB587JfbqlyMBAAgZhGRShEF7fsViXZtQ/S5g== 2020-11-16T21:47:37.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB587JfbqlyMBAAgZhGRShEF7fsViXZtQ/S5g=="}' 2020-11-16T21:47:37.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c008bbef-3ee6-48d2-bad9-34f4550406d6 -i td/osd-backfill-stats/3/new.json 2020-11-16T21:47:38.397 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:47:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2020-11-16T21:47:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB587JfbqlyMBAAgZhGRShEF7fsViXZtQ/S5g== --osd-uuid c008bbef-3ee6-48d2-bad9-34f4550406d6 2020-11-16T21:47:38.442 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:38.448+0000 7f6635f42d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:38.445 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:38.448+0000 7f6635f42d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:38.446 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:38.452+0000 7f6635f42d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:38.447 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:38.452+0000 7f6635f42d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:47:38.447 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:38.452+0000 7f6635f42d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:47:38.448 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:38.452+0000 7f6635f42d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:47:38.496 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:38.500+0000 7f6635f42d80 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2020-11-16T21:47:42.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2020-11-16T21:47:42.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:47:42.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:47:42.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:47:42.869 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:47:43.360 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:47:43.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:47:43.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:47:43.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:47:43.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:47:43.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:47:43.388 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:43.392+0000 7f6fa5d91d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:43.390 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:43.396+0000 7f6fa5d91d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:43.391 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:43.396+0000 7f6fa5d91d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:43.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:47:43.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:47:43.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:47:43.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:47:43.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:47:43.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:43.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:47:43.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:43.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:47:43.749 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:47:43.932 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:43.936+0000 7f6fa5d91d80 -1 Falling back to public interface 2020-11-16T21:47:44.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:47:44.836 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:44.832+0000 7f6fa5d91d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:47:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:47:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:47:45.155 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:47:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:47:45.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:47:46.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:47:46.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:46.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:47:46.541 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:47:46.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:46.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:47:46.929 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/509322,v1:127.0.0.1:6827/509322] [v2:127.0.0.1:6828/509322,v1:127.0.0.1:6829/509322] exists,up c008bbef-3ee6-48d2-bad9-34f4550406d6 2020-11-16T21:47:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:47:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:47:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:47:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:716: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 4 2020-11-16T21:47:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:47:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:47:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:47:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:47:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2020-11-16T21:47:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:47:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:47:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:47:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:47:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2020-11-16T21:47:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2020-11-16T21:47:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:47:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:47:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:47:46.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:47:46.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:47:46.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:47:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:47:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:46.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:47:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:47:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:47:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:47:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:47:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:47:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:47:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:47:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:47:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:47:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2020-11-16T21:47:46.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:47:46.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=efdb25a5-51d0-4dd8-a65c-117bde9b0d4d 2020-11-16T21:47:46.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 efdb25a5-51d0-4dd8-a65c-117bde9b0d4d' 2020-11-16T21:47:46.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:47:46.937 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 efdb25a5-51d0-4dd8-a65c-117bde9b0d4d 2020-11-16T21:47:46.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCC87JfHiSDORAAMfpGcZa6uWKckPzWhF3bfA== 2020-11-16T21:47:46.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCC87JfHiSDORAAMfpGcZa6uWKckPzWhF3bfA=="}' 2020-11-16T21:47:46.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new efdb25a5-51d0-4dd8-a65c-117bde9b0d4d -i td/osd-backfill-stats/4/new.json 2020-11-16T21:47:47.428 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:47:47.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2020-11-16T21:47:47.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCC87JfHiSDORAAMfpGcZa6uWKckPzWhF3bfA== --osd-uuid efdb25a5-51d0-4dd8-a65c-117bde9b0d4d 2020-11-16T21:47:47.472 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:47.476+0000 7ff9cf0e4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:47.494 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:47.500+0000 7ff9cf0e4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:47.496 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:47.500+0000 7ff9cf0e4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:47.497 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:47.500+0000 7ff9cf0e4d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:47:47.497 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:47.500+0000 7ff9cf0e4d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:47:47.497 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:47.500+0000 7ff9cf0e4d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:47:47.586 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:47.592+0000 7ff9cf0e4d80 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2020-11-16T21:47:51.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2020-11-16T21:47:51.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:47:51.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:47:51.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:47:51.954 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:47:52.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:47:52.446 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:47:52.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:47:52.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:47:52.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:47:52.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:47:52.475 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:52.480+0000 7f0b55a4cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:52.478 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:52.484+0000 7f0b55a4cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:52.480 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:52.484+0000 7f0b55a4cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:52.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:47:52.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:47:52.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:47:52.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:47:52.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:47:52.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:52.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:47:52.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:52.841 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:47:52.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:47:53.019 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:53.024+0000 7f0b55a4cd80 -1 Falling back to public interface 2020-11-16T21:47:53.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:47:53.849 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:53.852+0000 7f0b55a4cd80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:47:54.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:47:54.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:54.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:47:54.218 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:47:54.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:54.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:47:54.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:47:55.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:47:55.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:47:55.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:47:55.640 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:47:55.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:47:55.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:47:56.033 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/510342,v1:127.0.0.1:6835/510342] [v2:127.0.0.1:6836/510342,v1:127.0.0.1:6837/510342] exists,up efdb25a5-51d0-4dd8-a65c-117bde9b0d4d 2020-11-16T21:47:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:47:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:47:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:47:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:717: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 5 2020-11-16T21:47:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:47:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:47:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T21:47:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:47:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2020-11-16T21:47:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:47:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:47:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:47:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:47:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2020-11-16T21:47:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2020-11-16T21:47:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:47:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:47:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:47:56.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:47:56.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:47:56.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:47:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:47:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:47:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:47:56.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:47:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:47:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:47:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:47:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:47:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:47:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:47:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:47:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:47:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:47:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:47:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2020-11-16T21:47:56.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:47:56.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e96f3be0-b0c0-4023-93df-10302ff4d961 2020-11-16T21:47:56.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 e96f3be0-b0c0-4023-93df-10302ff4d961' 2020-11-16T21:47:56.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:47:56.042 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 e96f3be0-b0c0-4023-93df-10302ff4d961 2020-11-16T21:47:56.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCM87JfvbzzAxAAwCpPDeqrlwD0el5/jCtkGg== 2020-11-16T21:47:56.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCM87JfvbzzAxAAwCpPDeqrlwD0el5/jCtkGg=="}' 2020-11-16T21:47:56.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e96f3be0-b0c0-4023-93df-10302ff4d961 -i td/osd-backfill-stats/5/new.json 2020-11-16T21:47:56.550 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T21:47:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2020-11-16T21:47:56.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCM87JfvbzzAxAAwCpPDeqrlwD0el5/jCtkGg== --osd-uuid e96f3be0-b0c0-4023-93df-10302ff4d961 2020-11-16T21:47:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:56.600+0000 7f5e3a147d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:56.604+0000 7f5e3a147d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:56.604+0000 7f5e3a147d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:47:56.602 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:56.608+0000 7f5e3a147d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:47:56.602 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:56.608+0000 7f5e3a147d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:47:56.602 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:56.608+0000 7f5e3a147d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:47:56.651 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:47:56.656+0000 7f5e3a147d80 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2020-11-16T21:48:01.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2020-11-16T21:48:01.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:48:01.024 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T21:48:01.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T21:48:01.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:48:01.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T21:48:01.533 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T21:48:01.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:48:01.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:48:01.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:48:01.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:48:01.563 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:48:01.567+0000 7f407d573d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:48:01.566 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:48:01.571+0000 7f407d573d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:48:01.567 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:48:01.571+0000 7f407d573d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:48:01.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T21:48:01.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:48:01.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T21:48:01.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:48:01.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:48:01.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:48:01.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:48:01.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:48:01.914 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:48:01.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:48:02.115 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:48:02.119+0000 7f407d573d80 -1 Falling back to public interface 2020-11-16T21:48:02.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:48:02.799 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:48:02.803+0000 7f407d573d80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T21:48:03.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:48:03.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:48:03.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:48:03.311 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:48:03.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:48:03.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:48:03.748 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/511362,v1:127.0.0.1:6843/511362] [v2:127.0.0.1:6844/511362,v1:127.0.0.1:6845/511362] exists,up e96f3be0-b0c0-4023-93df-10302ff4d961 2020-11-16T21:48:03.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:48:03.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:48:03.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:48:03.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:719: TEST_backfill_ec_down_out: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2020-11-16T21:48:04.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:720: TEST_backfill_ec_down_out: create_pool test 1 1 erasure myprofile 2020-11-16T21:48:04.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-11-16T21:48:05.102 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T21:48:05.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:48:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:721: TEST_backfill_ec_down_out: ceph osd pool set test min_size 2 2020-11-16T21:48:06.884 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 min_size to 2 2020-11-16T21:48:06.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:723: TEST_backfill_ec_down_out: wait_for_clean 2020-11-16T21:48:06.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:48:06.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:48:06.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:48:06.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:48:06.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:48:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:48:06.903 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:48:06.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:48:06.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:48:06.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:48:07.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:48:07.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:48:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:48:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:48:07.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:48:07.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:48:07.508 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:48:07.508 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:48:07.508 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:48:07.508 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:48:07.508 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:48:07.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:48:07.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:07.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:48:07.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836492 2020-11-16T21:48:07.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836492 2020-11-16T21:48:07.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492' 2020-11-16T21:48:07.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:07.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:48:07.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672971 2020-11-16T21:48:07.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672971 2020-11-16T21:48:07.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971' 2020-11-16T21:48:07.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:07.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:48:08.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542152 2020-11-16T21:48:08.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542152 2020-11-16T21:48:08.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-60129542152' 2020-11-16T21:48:08.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:08.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:48:08.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378631 2020-11-16T21:48:08.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378631 2020-11-16T21:48:08.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-60129542152 3-81604378631' 2020-11-16T21:48:08.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:08.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:48:08.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215110 2020-11-16T21:48:08.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215110 2020-11-16T21:48:08.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-60129542152 3-81604378631 4-103079215110' 2020-11-16T21:48:08.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:08.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:48:08.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051588 2020-11-16T21:48:08.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051588 2020-11-16T21:48:08.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-60129542152 3-81604378631 4-103079215110 5-124554051588' 2020-11-16T21:48:08.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:48:08.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836492 2020-11-16T21:48:08.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:48:08.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:48:08.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836492 2020-11-16T21:48:08.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:48:08.521 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836492 2020-11-16T21:48:08.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836492 2020-11-16T21:48:08.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-11-16T21:48:08.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:48:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836492 2020-11-16T21:48:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:48:08.909 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672971 2020-11-16T21:48:08.909 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:48:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:48:08.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2020-11-16T21:48:08.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:48:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672971 2020-11-16T21:48:08.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-11-16T21:48:08.913 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672971 2020-11-16T21:48:08.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:48:09.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672971 2020-11-16T21:48:09.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:48:09.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542152 2020-11-16T21:48:09.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:48:09.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:48:09.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:48:09.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542152 2020-11-16T21:48:09.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542152 2020-11-16T21:48:09.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2020-11-16T21:48:09.332 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542152 2020-11-16T21:48:09.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:48:09.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542152 -lt 60129542152 2020-11-16T21:48:09.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:48:09.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378631 2020-11-16T21:48:09.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:48:09.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:48:09.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:48:09.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378631 2020-11-16T21:48:09.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378631 2020-11-16T21:48:09.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378631' 2020-11-16T21:48:09.734 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378631 2020-11-16T21:48:09.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:48:10.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378631 -lt 81604378631 2020-11-16T21:48:10.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:48:10.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215110 2020-11-16T21:48:10.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:48:10.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:48:10.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215110 2020-11-16T21:48:10.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:48:10.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215110 2020-11-16T21:48:10.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215110' 2020-11-16T21:48:10.144 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215110 2020-11-16T21:48:10.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:48:10.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215110 -lt 103079215110 2020-11-16T21:48:10.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:48:10.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051588 2020-11-16T21:48:10.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:48:10.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:48:10.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051588 2020-11-16T21:48:10.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:48:10.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051588 2020-11-16T21:48:10.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051588' 2020-11-16T21:48:10.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:48:10.527 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051588 2020-11-16T21:48:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051589 -lt 124554051588 2020-11-16T21:48:10.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:48:10.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:48:10.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:48:11.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:48:11.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:48:11.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:48:11.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:48:11.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:48:11.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:48:11.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:48:11.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:48:11.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:48:11.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:48:11.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:48:11.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:48:12.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:48:12.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:48:12.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:48:12.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: seq 1 200 2020-11-16T21:48:12.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:12.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj1 /dev/null 2020-11-16T21:48:12.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:12.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj2 /dev/null 2020-11-16T21:48:12.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:12.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj3 /dev/null 2020-11-16T21:48:12.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:12.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj4 /dev/null 2020-11-16T21:48:12.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:12.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj5 /dev/null 2020-11-16T21:48:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:13.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj6 /dev/null 2020-11-16T21:48:13.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:13.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj7 /dev/null 2020-11-16T21:48:13.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:13.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj8 /dev/null 2020-11-16T21:48:13.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:13.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj9 /dev/null 2020-11-16T21:48:13.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:13.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj10 /dev/null 2020-11-16T21:48:13.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:13.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj11 /dev/null 2020-11-16T21:48:14.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:14.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj12 /dev/null 2020-11-16T21:48:14.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:14.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj13 /dev/null 2020-11-16T21:48:14.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:14.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj14 /dev/null 2020-11-16T21:48:14.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:14.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj15 /dev/null 2020-11-16T21:48:14.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:14.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj16 /dev/null 2020-11-16T21:48:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:15.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj17 /dev/null 2020-11-16T21:48:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj18 /dev/null 2020-11-16T21:48:15.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:15.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj19 /dev/null 2020-11-16T21:48:15.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:15.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj20 /dev/null 2020-11-16T21:48:15.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:15.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj21 /dev/null 2020-11-16T21:48:15.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:15.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj22 /dev/null 2020-11-16T21:48:16.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:16.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj23 /dev/null 2020-11-16T21:48:16.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:16.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj24 /dev/null 2020-11-16T21:48:16.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:16.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj25 /dev/null 2020-11-16T21:48:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj26 /dev/null 2020-11-16T21:48:16.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:16.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj27 /dev/null 2020-11-16T21:48:17.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:17.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj28 /dev/null 2020-11-16T21:48:17.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:17.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj29 /dev/null 2020-11-16T21:48:17.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:17.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj30 /dev/null 2020-11-16T21:48:17.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:17.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj31 /dev/null 2020-11-16T21:48:17.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:17.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj32 /dev/null 2020-11-16T21:48:17.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:17.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj33 /dev/null 2020-11-16T21:48:18.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:18.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj34 /dev/null 2020-11-16T21:48:18.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:18.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj35 /dev/null 2020-11-16T21:48:18.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:18.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj36 /dev/null 2020-11-16T21:48:18.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:18.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj37 /dev/null 2020-11-16T21:48:18.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:18.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj38 /dev/null 2020-11-16T21:48:18.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:18.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj39 /dev/null 2020-11-16T21:48:19.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:19.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj40 /dev/null 2020-11-16T21:48:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj41 /dev/null 2020-11-16T21:48:19.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:19.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj42 /dev/null 2020-11-16T21:48:19.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:19.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj43 /dev/null 2020-11-16T21:48:19.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:19.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj44 /dev/null 2020-11-16T21:48:20.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:20.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj45 /dev/null 2020-11-16T21:48:20.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:20.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj46 /dev/null 2020-11-16T21:48:20.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:20.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj47 /dev/null 2020-11-16T21:48:20.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:20.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj48 /dev/null 2020-11-16T21:48:20.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:20.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj49 /dev/null 2020-11-16T21:48:20.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:20.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj50 /dev/null 2020-11-16T21:48:21.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:21.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj51 /dev/null 2020-11-16T21:48:21.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj52 /dev/null 2020-11-16T21:48:21.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:21.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj53 /dev/null 2020-11-16T21:48:21.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:21.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj54 /dev/null 2020-11-16T21:48:21.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:21.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj55 /dev/null 2020-11-16T21:48:21.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:21.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj56 /dev/null 2020-11-16T21:48:22.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:22.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj57 /dev/null 2020-11-16T21:48:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj58 /dev/null 2020-11-16T21:48:22.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:22.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj59 /dev/null 2020-11-16T21:48:22.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:22.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj60 /dev/null 2020-11-16T21:48:22.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:22.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj61 /dev/null 2020-11-16T21:48:22.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:22.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj62 /dev/null 2020-11-16T21:48:23.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:23.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj63 /dev/null 2020-11-16T21:48:23.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:23.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj64 /dev/null 2020-11-16T21:48:23.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:23.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj65 /dev/null 2020-11-16T21:48:23.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:23.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj66 /dev/null 2020-11-16T21:48:23.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:23.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj67 /dev/null 2020-11-16T21:48:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj68 /dev/null 2020-11-16T21:48:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj69 /dev/null 2020-11-16T21:48:24.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:24.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj70 /dev/null 2020-11-16T21:48:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj71 /dev/null 2020-11-16T21:48:24.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:24.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj72 /dev/null 2020-11-16T21:48:24.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:24.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj73 /dev/null 2020-11-16T21:48:25.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:25.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj74 /dev/null 2020-11-16T21:48:25.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:25.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj75 /dev/null 2020-11-16T21:48:25.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:25.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj76 /dev/null 2020-11-16T21:48:25.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:25.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj77 /dev/null 2020-11-16T21:48:25.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:25.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj78 /dev/null 2020-11-16T21:48:25.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:25.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj79 /dev/null 2020-11-16T21:48:26.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:26.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj80 /dev/null 2020-11-16T21:48:26.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:26.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj81 /dev/null 2020-11-16T21:48:26.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:26.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj82 /dev/null 2020-11-16T21:48:26.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:26.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj83 /dev/null 2020-11-16T21:48:26.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:26.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj84 /dev/null 2020-11-16T21:48:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj85 /dev/null 2020-11-16T21:48:27.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:27.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj86 /dev/null 2020-11-16T21:48:27.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:27.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj87 /dev/null 2020-11-16T21:48:27.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:27.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj88 /dev/null 2020-11-16T21:48:27.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:27.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj89 /dev/null 2020-11-16T21:48:27.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:27.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj90 /dev/null 2020-11-16T21:48:28.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:28.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj91 /dev/null 2020-11-16T21:48:28.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:28.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj92 /dev/null 2020-11-16T21:48:28.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:28.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj93 /dev/null 2020-11-16T21:48:28.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:28.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj94 /dev/null 2020-11-16T21:48:28.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:28.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj95 /dev/null 2020-11-16T21:48:28.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:28.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj96 /dev/null 2020-11-16T21:48:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj97 /dev/null 2020-11-16T21:48:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:29.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj98 /dev/null 2020-11-16T21:48:29.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:29.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj99 /dev/null 2020-11-16T21:48:29.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:29.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj100 /dev/null 2020-11-16T21:48:29.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:29.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj101 /dev/null 2020-11-16T21:48:29.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:29.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj102 /dev/null 2020-11-16T21:48:30.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:30.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj103 /dev/null 2020-11-16T21:48:30.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:30.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj104 /dev/null 2020-11-16T21:48:30.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:30.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj105 /dev/null 2020-11-16T21:48:30.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:30.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj106 /dev/null 2020-11-16T21:48:30.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:30.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj107 /dev/null 2020-11-16T21:48:30.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:30.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj108 /dev/null 2020-11-16T21:48:31.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:31.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj109 /dev/null 2020-11-16T21:48:31.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:31.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj110 /dev/null 2020-11-16T21:48:31.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:31.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj111 /dev/null 2020-11-16T21:48:31.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:31.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj112 /dev/null 2020-11-16T21:48:31.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:31.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj113 /dev/null 2020-11-16T21:48:31.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:31.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj114 /dev/null 2020-11-16T21:48:32.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:32.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj115 /dev/null 2020-11-16T21:48:32.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:32.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj116 /dev/null 2020-11-16T21:48:32.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:32.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj117 /dev/null 2020-11-16T21:48:32.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:32.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj118 /dev/null 2020-11-16T21:48:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj119 /dev/null 2020-11-16T21:48:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj120 /dev/null 2020-11-16T21:48:33.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:33.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj121 /dev/null 2020-11-16T21:48:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj122 /dev/null 2020-11-16T21:48:33.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:33.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj123 /dev/null 2020-11-16T21:48:33.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:33.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj124 /dev/null 2020-11-16T21:48:33.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:33.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj125 /dev/null 2020-11-16T21:48:34.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:34.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj126 /dev/null 2020-11-16T21:48:34.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:34.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj127 /dev/null 2020-11-16T21:48:34.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:34.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj128 /dev/null 2020-11-16T21:48:34.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:34.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj129 /dev/null 2020-11-16T21:48:34.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:34.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj130 /dev/null 2020-11-16T21:48:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:34.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj131 /dev/null 2020-11-16T21:48:35.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:35.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj132 /dev/null 2020-11-16T21:48:35.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:35.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj133 /dev/null 2020-11-16T21:48:35.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:35.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj134 /dev/null 2020-11-16T21:48:35.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:35.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj135 /dev/null 2020-11-16T21:48:35.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:35.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj136 /dev/null 2020-11-16T21:48:35.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:35.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj137 /dev/null 2020-11-16T21:48:35.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:35.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj138 /dev/null 2020-11-16T21:48:36.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:36.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj139 /dev/null 2020-11-16T21:48:36.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:36.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj140 /dev/null 2020-11-16T21:48:36.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:36.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj141 /dev/null 2020-11-16T21:48:36.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:36.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj142 /dev/null 2020-11-16T21:48:36.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:36.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj143 /dev/null 2020-11-16T21:48:37.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:37.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj144 /dev/null 2020-11-16T21:48:37.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:37.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj145 /dev/null 2020-11-16T21:48:37.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:37.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj146 /dev/null 2020-11-16T21:48:37.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:37.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj147 /dev/null 2020-11-16T21:48:37.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:37.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj148 /dev/null 2020-11-16T21:48:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:38.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj149 /dev/null 2020-11-16T21:48:38.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:38.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj150 /dev/null 2020-11-16T21:48:38.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:38.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj151 /dev/null 2020-11-16T21:48:38.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:38.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj152 /dev/null 2020-11-16T21:48:38.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:38.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj153 /dev/null 2020-11-16T21:48:38.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:38.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj154 /dev/null 2020-11-16T21:48:39.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:39.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj155 /dev/null 2020-11-16T21:48:39.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:39.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj156 /dev/null 2020-11-16T21:48:39.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:39.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj157 /dev/null 2020-11-16T21:48:39.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:39.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj158 /dev/null 2020-11-16T21:48:39.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:39.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj159 /dev/null 2020-11-16T21:48:39.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:39.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj160 /dev/null 2020-11-16T21:48:40.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:40.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj161 /dev/null 2020-11-16T21:48:40.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:40.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj162 /dev/null 2020-11-16T21:48:40.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:40.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj163 /dev/null 2020-11-16T21:48:40.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:40.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj164 /dev/null 2020-11-16T21:48:40.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:40.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj165 /dev/null 2020-11-16T21:48:40.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:40.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj166 /dev/null 2020-11-16T21:48:41.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:41.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj167 /dev/null 2020-11-16T21:48:41.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:41.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj168 /dev/null 2020-11-16T21:48:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj169 /dev/null 2020-11-16T21:48:41.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:41.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj170 /dev/null 2020-11-16T21:48:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj171 /dev/null 2020-11-16T21:48:42.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:42.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj172 /dev/null 2020-11-16T21:48:42.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:42.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj173 /dev/null 2020-11-16T21:48:42.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:42.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj174 /dev/null 2020-11-16T21:48:42.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:42.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj175 /dev/null 2020-11-16T21:48:42.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:42.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj176 /dev/null 2020-11-16T21:48:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:42.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj177 /dev/null 2020-11-16T21:48:43.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:43.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj178 /dev/null 2020-11-16T21:48:43.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:43.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj179 /dev/null 2020-11-16T21:48:43.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:43.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj180 /dev/null 2020-11-16T21:48:43.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:43.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj181 /dev/null 2020-11-16T21:48:43.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:43.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj182 /dev/null 2020-11-16T21:48:43.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:43.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj183 /dev/null 2020-11-16T21:48:44.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:44.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj184 /dev/null 2020-11-16T21:48:44.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:44.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj185 /dev/null 2020-11-16T21:48:44.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:44.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj186 /dev/null 2020-11-16T21:48:44.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:44.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj187 /dev/null 2020-11-16T21:48:44.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:44.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj188 /dev/null 2020-11-16T21:48:44.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj189 /dev/null 2020-11-16T21:48:45.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:45.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj190 /dev/null 2020-11-16T21:48:45.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:45.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj191 /dev/null 2020-11-16T21:48:45.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:45.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj192 /dev/null 2020-11-16T21:48:45.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:45.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj193 /dev/null 2020-11-16T21:48:45.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:45.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj194 /dev/null 2020-11-16T21:48:45.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:45.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj195 /dev/null 2020-11-16T21:48:46.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:46.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj196 /dev/null 2020-11-16T21:48:46.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:46.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj197 /dev/null 2020-11-16T21:48:46.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:46.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj198 /dev/null 2020-11-16T21:48:46.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:46.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj199 /dev/null 2020-11-16T21:48:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2020-11-16T21:48:46.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj200 /dev/null 2020-11-16T21:48:47.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:730: TEST_backfill_ec_down_out: get_pg test obj1 2020-11-16T21:48:47.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T21:48:47.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T21:48:47.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T21:48:47.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T21:48:47.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:730: TEST_backfill_ec_down_out: local PG=1.0 2020-11-16T21:48:47.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:732: TEST_backfill_ec_down_out: get_primary test obj1 2020-11-16T21:48:47.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:48:47.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:48:47.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:48:47.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:48:47.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:732: TEST_backfill_ec_down_out: local primary=1 2020-11-16T21:48:47.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:733: TEST_backfill_ec_down_out: get_not_primary test obj1 2020-11-16T21:48:47.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2020-11-16T21:48:47.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2020-11-16T21:48:47.956 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2020-11-16T21:48:47.956 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:48:47.957 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:48:47.957 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:48:47.957 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:48:48.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T21:48:48.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2020-11-16T21:48:48.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T21:48:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:733: TEST_backfill_ec_down_out: local otherosd=0 2020-11-16T21:48:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:735: TEST_backfill_ec_down_out: ceph osd set nobackfill 2020-11-16T21:48:49.456 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T21:48:49.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:736: TEST_backfill_ec_down_out: cat td/osd-backfill-stats/osd.0.pid 2020-11-16T21:48:49.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:736: TEST_backfill_ec_down_out: kill 506340 2020-11-16T21:48:49.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:737: TEST_backfill_ec_down_out: ceph osd down osd.0 2020-11-16T21:48:49.887 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already down. 2020-11-16T21:48:49.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:738: TEST_backfill_ec_down_out: ceph osd out osd.0 2020-11-16T21:48:50.647 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T21:48:50.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:740: TEST_backfill_ec_down_out: sleep 2 2020-11-16T21:48:52.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:741: TEST_backfill_ec_down_out: get_primary test obj1 2020-11-16T21:48:52.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:48:52.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:48:52.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:48:52.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:48:53.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:741: TEST_backfill_ec_down_out: primary=1 2020-11-16T21:48:53.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:742: TEST_backfill_ec_down_out: ceph osd unset nobackfill 2020-11-16T21:48:53.753 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T21:48:53.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:743: TEST_backfill_ec_down_out: ceph tell osd.1 get_latest_osdmap 2020-11-16T21:48:53.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:744: TEST_backfill_ec_down_out: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T21:48:54.087 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T21:48:54.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:745: TEST_backfill_ec_down_out: sleep 2 2020-11-16T21:48:56.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:747: TEST_backfill_ec_down_out: wait_for_clean 2020-11-16T21:48:56.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:48:56.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:48:56.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:48:56.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:48:56.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:48:56.105 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:48:56.106 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:48:56.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:48:56.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:48:56.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:48:56.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:48:56.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:48:56.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:48:56.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:48:56.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:48:56.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:48:56.714 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:48:56.714 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:48:56.714 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:48:56.715 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:48:56.715 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:48:56.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:48:56.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:56.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:48:56.838 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENXIO: problem getting command descriptions from osd.0 2020-11-16T21:48:56.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-11-16T21:48:56.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-11-16T21:48:56.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-11-16T21:48:56.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:56.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:48:57.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672983 2020-11-16T21:48:57.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672983 2020-11-16T21:48:57.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672983' 2020-11-16T21:48:57.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:57.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:48:57.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542164 2020-11-16T21:48:57.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542164 2020-11-16T21:48:57.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672983 2-60129542164' 2020-11-16T21:48:57.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:57.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:48:57.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378643 2020-11-16T21:48:57.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378643 2020-11-16T21:48:57.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672983 2-60129542164 3-81604378643' 2020-11-16T21:48:57.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:57.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:48:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215122 2020-11-16T21:48:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215122 2020-11-16T21:48:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672983 2-60129542164 3-81604378643 4-103079215122' 2020-11-16T21:48:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:48:57.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:48:57.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051600 2020-11-16T21:48:57.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051600 2020-11-16T21:48:57.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672983 2-60129542164 3-81604378643 4-103079215122 5-124554051600' 2020-11-16T21:48:57.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:48:57.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672983 2020-11-16T21:48:57.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:48:57.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:48:57.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672983 2020-11-16T21:48:57.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:48:57.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672983 2020-11-16T21:48:57.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2020-11-16T21:48:57.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:48:57.675 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672983 2020-11-16T21:48:58.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672981 -lt 42949672983 2020-11-16T21:48:58.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:48:59.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:48:59.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:48:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672983 -lt 42949672983 2020-11-16T21:48:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:48:59.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:48:59.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542164 2020-11-16T21:48:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:48:59.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542164 2020-11-16T21:48:59.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:48:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542164 2020-11-16T21:48:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542164' 2020-11-16T21:48:59.478 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542164 2020-11-16T21:48:59.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:48:59.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542164 -lt 60129542164 2020-11-16T21:48:59.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:48:59.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378643 2020-11-16T21:48:59.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:48:59.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:48:59.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378643 2020-11-16T21:48:59.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:48:59.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378643 2020-11-16T21:48:59.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378643' 2020-11-16T21:48:59.872 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378643 2020-11-16T21:48:59.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:49:00.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378643 -lt 81604378643 2020-11-16T21:49:00.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:49:00.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215122 2020-11-16T21:49:00.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:49:00.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:49:00.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215122 2020-11-16T21:49:00.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:49:00.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215122 2020-11-16T21:49:00.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215122' 2020-11-16T21:49:00.297 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215122 2020-11-16T21:49:00.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:49:00.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215122 -lt 103079215122 2020-11-16T21:49:00.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:49:00.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051600 2020-11-16T21:49:00.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:49:00.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:49:00.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051600 2020-11-16T21:49:00.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:49:00.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051600 2020-11-16T21:49:00.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051600' 2020-11-16T21:49:00.711 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051600 2020-11-16T21:49:00.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:49:01.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051601 -lt 124554051600 2020-11-16T21:49:01.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:49:01.102 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:01.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:01.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:49:01.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:01.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:01.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:01.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:01.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:01.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:01.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:01.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:01.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:01.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:01.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:02.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:02.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T21:49:02.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T21:49:02.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T21:49:02.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:02.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:02.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:02.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:02.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:02.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:02.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:02.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:02.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:02.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:02.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:02.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:02.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:02.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:03.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:03.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:03.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:03.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:03.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:03.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:03.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:03.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:03.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:03.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:49:03.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:49:03.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:03.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:03.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:04.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:04.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:04.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:04.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:04.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:04.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:04.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:04.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:04.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:04.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:04.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:04.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:04.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:04.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:04.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:04.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:04.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:04.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:04.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:04.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:04.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:05.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:49:05.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:49:05.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:05.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:05.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:05.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:05.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:05.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:05.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:05.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:05.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:05.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:05.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:05.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:05.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:05.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:05.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:06.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:06.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:06.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:06.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:06.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:06.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:06.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:06.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:06.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:49:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:49:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:06.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:06.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:06.908 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:06.909 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:06.909 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:06.909 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:06.909 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:06.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:07.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:07.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:07.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:07.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:07.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:07.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:07.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:07.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:07.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:08.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:49:08.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:49:08.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:08.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:08.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:08.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:08.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:08.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:08.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:08.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:08.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:08.307 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:08.307 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:08.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:08.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:08.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:08.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:09.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:09.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:09.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:09.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:09.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:09.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:09.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:09.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:09.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:09.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:49:09.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:49:09.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:09.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:09.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:09.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:09.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:09.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:09.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:09.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:09.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:09.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:10.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:10.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:10.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:10.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:10.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:10.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:10.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:10.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:10.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:10.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:10.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:10.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:10.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:11.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:49:11.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:49:11.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:11.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:11.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:11.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:11.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:11.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:11.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:11.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:11.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:11.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:11.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:11.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:11.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:11.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:12.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:12.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:12.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:12.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:12.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:12.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:12.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:12.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:12.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:12.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:49:12.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:49:12.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:12.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:12.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:12.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:12.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:12.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:12.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:12.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:12.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:12.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:12.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:12.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:12.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:12.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:12.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:13.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:13.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:13.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:13.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:13.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:13.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:13.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:13.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:13.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:13.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:49:13.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:49:13.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:13.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:13.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:14.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:14.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:14.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:14.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:14.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:14.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:14.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:14.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:14.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:14.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:14.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:14.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:14.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:14.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:14.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:14.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:14.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:14.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:14.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:14.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:14.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:15.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:49:15.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:49:15.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:15.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:15.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:15.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:15.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:15.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:15.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:15.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:15.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:15.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:15.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:15.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:15.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:16.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:16.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:16.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:16.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:16.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:16.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:16.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:16.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:16.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:16.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:49:16.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:49:16.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:16.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:16.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:16.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:16.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:16.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:16.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:16.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:16.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:16.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:16.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:17.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:17.308 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:17.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:17.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:17.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:17.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:17.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:17.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:17.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:17.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:17.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:17.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:17.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:49:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:49:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:18.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:18.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:18.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:18.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:18.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:18.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:18.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:18.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:18.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:18.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:18.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:18.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:19.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:19.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:19.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:19.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:19.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:19.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:19.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:19.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:19.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:19.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:49:19.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:49:19.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:19.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:19.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:19.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:19.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:19.988 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:19.988 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:19.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:19.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:19.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:19.989 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:20.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:20.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:20.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:20.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:20.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:20.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:20.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:20.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:20.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:20.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:20.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:20.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:20.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:21.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:49:21.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:49:21.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:21.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:21.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:21.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:21.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:21.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:21.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:21.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:21.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:21.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:21.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:21.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:21.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:21.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:21.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:22.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:22.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:22.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:22.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:22.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:22.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:22.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:22.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:22.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:22.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:49:22.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:49:22.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:22.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:22.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:22.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:22.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:22.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:22.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:22.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:22.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:22.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:22.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:23.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:23.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:23.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:23.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:23.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:23.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:23.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:23.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:23.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:23.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:23.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:23.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:23.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:24.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:49:24.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:49:24.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:24.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:24.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:24.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:24.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:24.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:24.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:24.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:24.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:24.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:24.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:24.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:24.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:24.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:24.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:25.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:25.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:25.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:25.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:25.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:25.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:49:25.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:49:25.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:25.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:25.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:25.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:25.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:25.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:25.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:25.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:25.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:25.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:25.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:26.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:26.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:26.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:26.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:26.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:26.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:26.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:26.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:26.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:26.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:26.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:26.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:49:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:49:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:27.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:27.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:27.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:27.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:27.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:27.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:27.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:27.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:27.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:27.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:27.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:27.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:28.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:28.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:28.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:28.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:28.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:28.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:28.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:28.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:28.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:28.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:49:28.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:49:28.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:28.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:28.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:28.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:28.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:28.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:28.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:28.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:28.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:28.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:28.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:28.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:28.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:28.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:28.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:29.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:29.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:29.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:29.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:29.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:29.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:29.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:29.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:29.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:29.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:49:29.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:49:29.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:29.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:29.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:30.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:30.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:30.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:30.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:30.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:30.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:30.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:30.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:30.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:30.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:30.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:30.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:30.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:30.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:30.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:30.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:30.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:30.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:30.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:30.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:30.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:31.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:49:31.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:49:31.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:31.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:31.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:31.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:31.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:31.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:31.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:31.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:31.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:31.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:31.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:31.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:31.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:31.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:31.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:32.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:32.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:32.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:32.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:32.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:32.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:32.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:32.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:32.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:49:32.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:49:32.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:32.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:32.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:32.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:32.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:32.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:32.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:32.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:32.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:32.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:32.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:33.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:33.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:33.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:33.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:33.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:33.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:33.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:33.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:33.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:33.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:33.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:33.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:33.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:49:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:49:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:34.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:34.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:34.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:34.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:34.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:34.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:34.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:34.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:34.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:34.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:34.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:34.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:35.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:35.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:35.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:35.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:35.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:35.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:35.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:35.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:35.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:35.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:49:35.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:49:35.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:35.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:35.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:35.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:35.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:35.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:35.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:35.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:35.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:35.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:35.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:36.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:49:36.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:36.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:36.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:36.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:49:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:49:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:49:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:49:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:49:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:49:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:49:36.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:49:36.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:49:37.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:49:37.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:49:37.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:49:37.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:49:37.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:49:37.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:49:37.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:49:37.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:49:37.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:49:37.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:49:37.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:49:37.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:49:37.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:49:37.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:49:37.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:49:37.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:49:37.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:49:38.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:49:38.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:49:38.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:49:38.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:749: TEST_backfill_ec_down_out: expr 200 '*' 2 2020-11-16T21:49:38.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:749: TEST_backfill_ec_down_out: local misplaced=400 2020-11-16T21:49:38.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:750: TEST_backfill_ec_down_out: check td/osd-backfill-stats 1.0 1 erasure 200 0 0 0 2020-11-16T21:49:38.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2020-11-16T21:49:38.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2020-11-16T21:49:38.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2020-11-16T21:49:38.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=erasure 2020-11-16T21:49:38.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=200 2020-11-16T21:49:38.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2020-11-16T21:49:38.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=0 2020-11-16T21:49:38.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2020-11-16T21:49:38.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2020-11-16T21:49:38.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2020-11-16T21:49:38.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2020-11-16T21:49:38.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2020-11-16T21:49:38.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2020-11-16T21:49:38.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2020-11-16T21:49:38.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2020-11-16T21:49:38.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2020-11-16T21:49:38.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2020-11-16T21:49:38.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2020-11-16T21:49:38.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2020-11-16T21:49:38.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2020-11-16T21:49:38.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' erasure = erasure ']' 2020-11-16T21:49:38.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:89: check: addp=p 2020-11-16T21:49:38.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2020-11-16T21:49:38.149 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:49:38.150 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2020-11-16T21:49:38.150 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2020-11-16T21:49:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[1,3,2]/[1,NONE,2]' 2020-11-16T21:49:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[1,3,2]/[1,NONE,2]' ']' 2020-11-16T21:49:38.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2020-11-16T21:49:38.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:49:38.174 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:49:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2020-11-16T21:49:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2020-11-16T21:49:38.175 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2020-11-16T21:49:38.176 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:49:38.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=200 2020-11-16T21:49:38.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 200 ']' 2020-11-16T21:49:38.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 200 200 2020-11-16T21:49:38.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2020-11-16T21:49:38.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:49:38.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2020-11-16T21:49:38.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:49:38.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:49:38.188 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:49:38.188 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:49:38.188 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2020-11-16T21:49:38.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=1 2020-11-16T21:49:38.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 1 ']' 2020-11-16T21:49:38.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 1 0 2020-11-16T21:49:38.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2020-11-16T21:49:38.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:49:38.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:49:38.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:49:38.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2020-11-16T21:49:38.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:49:38.201 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:49:38.202 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2020-11-16T21:49:38.202 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2020-11-16T21:49:38.202 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2020-11-16T21:49:38.202 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:49:38.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=0 2020-11-16T21:49:38.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 0 ']' 2020-11-16T21:49:38.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 0 0 2020-11-16T21:49:38.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2020-11-16T21:49:38.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:49:38.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2020-11-16T21:49:38.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:49:38.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:49:38.213 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:49:38.213 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2020-11-16T21:49:38.213 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:49:38.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=0 2020-11-16T21:49:38.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 0 ']' 2020-11-16T21:49:38.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 0 0 2020-11-16T21:49:38.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2020-11-16T21:49:38.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:49:38.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:49:38.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:49:38.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2020-11-16T21:49:38.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:752: TEST_backfill_ec_down_out: delete_pool test 2020-11-16T21:49:38.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T21:49:38.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T21:49:38.739 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T21:49:38.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:753: TEST_backfill_ec_down_out: kill_daemons td/osd-backfill-stats 2020-11-16T21:49:38.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:49:38.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:49:38.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:49:38.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:49:38.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:49:38.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:49:38.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2020-11-16T21:49:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:49:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:49:38.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:49:38.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:49:38.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:49:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:49:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:49:38.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:49:38.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:49:38.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:49:38.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:49:38.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:49:38.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:49:38.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:49:38.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:49:38.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:49:38.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:49:38.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:49:38.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:49:38.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:49:38.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:49:38.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:49:38.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:49:38.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:49:38.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:49:38.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:49:38.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:49:38.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:49:38.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2020-11-16T21:49:38.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2020-11-16T21:49:38.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2020-11-16T21:49:38.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2020-11-16T21:49:38.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:49:38.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:49:38.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:49:38.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:49:38.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:49:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:49:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:49:38.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:49:38.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:49:38.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:49:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:49:38.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:49:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:49:38.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:49:38.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:49:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:49:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:49:38.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:49:38.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:49:38.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:49:38.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:49:38.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:49:38.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:49:38.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:49:38.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:49:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:49:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:49:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:49:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2020-11-16T21:49:38.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:49:38.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:49:38.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:49:38.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.465868 2020-11-16T21:49:38.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:49:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:49:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:49:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2020-11-16T21:49:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_ec_prim_out td/osd-backfill-stats 2020-11-16T21:49:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:569: TEST_backfill_ec_prim_out: local dir=td/osd-backfill-stats 2020-11-16T21:49:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:571: TEST_backfill_ec_prim_out: run_mon td/osd-backfill-stats a 2020-11-16T21:49:38.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2020-11-16T21:49:38.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:49:38.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:49:38.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:49:38.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2020-11-16T21:49:38.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2020-11-16T21:49:39.239 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:49:39.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:49:39.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:49:39.241 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:49:39.241 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:49:39.241 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:49:39.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:49:39.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:49:39.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:49:39.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:49:39.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:49:39.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:49:39.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:49:39.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:49:39.438 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:49:39.438 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:49:39.439 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:49:39.439 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:49:39.439 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:49:39.439 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:49:39.440 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:49:39.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:49:39.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get fsid 2020-11-16T21:49:39.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:49:39.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:49:39.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:49:39.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:49:39.546 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:49:39.546 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:49:39.546 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:49:39.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:49:39.546 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:49:39.546 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:49:39.547 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:49:39.547 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:49:39.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:49:39.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get mon_host 2020-11-16T21:49:39.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:572: TEST_backfill_ec_prim_out: run_mgr td/osd-backfill-stats x 2020-11-16T21:49:39.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2020-11-16T21:49:39.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:49:39.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:49:39.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:49:39.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2020-11-16T21:49:39.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:49:40.173 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:49:40.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:49:40.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:49:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:49:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:49:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:49:40.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:49:40.193 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:49:40.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:49:40.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:573: TEST_backfill_ec_prim_out: run_osd td/osd-backfill-stats 0 2020-11-16T21:49:40.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:49:40.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:49:40.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:49:40.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:49:40.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2020-11-16T21:49:40.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:49:40.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:49:40.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:49:40.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:49:40.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2020-11-16T21:49:40.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2020-11-16T21:49:40.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:49:40.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:49:40.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:49:40.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:49:40.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:49:40.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:49:40.206 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:49:40.206 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:49:40.206 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:49:40.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:49:40.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:49:40.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:49:40.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:49:40.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:49:40.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:49:40.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:49:40.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:49:40.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:49:40.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:49:40.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:49:40.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:49:40.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2020-11-16T21:49:40.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:49:40.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7bdcef3c-5a58-49b1-b403-abca4c09004e 2020-11-16T21:49:40.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 7bdcef3c-5a58-49b1-b403-abca4c09004e' 2020-11-16T21:49:40.209 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 7bdcef3c-5a58-49b1-b403-abca4c09004e 2020-11-16T21:49:40.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:49:40.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD087JfMTrxDRAAGFX83kNIseyLuvJtRc7hOg== 2020-11-16T21:49:40.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD087JfMTrxDRAAGFX83kNIseyLuvJtRc7hOg=="}' 2020-11-16T21:49:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7bdcef3c-5a58-49b1-b403-abca4c09004e -i td/osd-backfill-stats/0/new.json 2020-11-16T21:49:40.573 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:49:40.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2020-11-16T21:49:40.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD087JfMTrxDRAAGFX83kNIseyLuvJtRc7hOg== --osd-uuid 7bdcef3c-5a58-49b1-b403-abca4c09004e 2020-11-16T21:49:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:40.622+0000 7f679ea90d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:40.622+0000 7f679ea90d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:40.626+0000 7f679ea90d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:40.626+0000 7f679ea90d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:49:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:40.626+0000 7f679ea90d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:49:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:40.626+0000 7f679ea90d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:49:40.660 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:40.666+0000 7f679ea90d80 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2020-11-16T21:49:44.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2020-11-16T21:49:44.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:49:44.809 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:49:44.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:49:44.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:49:45.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:49:45.308 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:49:45.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:49:45.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:49:45.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:49:45.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:49:45.338 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:45.345+0000 7fcd7f33cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:45.340 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:45.345+0000 7fcd7f33cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:45.341 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:45.349+0000 7fcd7f33cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:45.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:49:45.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:49:45.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:49:45.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:49:45.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:49:45.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:49:45.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:49:45.690 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:49:45.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:49:45.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:49:45.876 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:45.881+0000 7fcd7f33cd80 -1 Falling back to public interface 2020-11-16T21:49:46.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:49:46.722 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:46.721+0000 7fcd7f33cd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:49:47.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:49:47.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:49:47.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:49:47.067 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:49:47.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:49:47.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:49:47.478 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/520080,v1:127.0.0.1:6803/520080] [v2:127.0.0.1:6804/520080,v1:127.0.0.1:6805/520080] exists,up 7bdcef3c-5a58-49b1-b403-abca4c09004e 2020-11-16T21:49:47.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:49:47.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:49:47.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:49:47.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:574: TEST_backfill_ec_prim_out: run_osd td/osd-backfill-stats 1 2020-11-16T21:49:47.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:49:47.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:49:47.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:49:47.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:49:47.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2020-11-16T21:49:47.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:49:47.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:49:47.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:49:47.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:49:47.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2020-11-16T21:49:47.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2020-11-16T21:49:47.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:49:47.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:49:47.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:49:47.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:49:47.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:49:47.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:49:47.483 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:49:47.483 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:49:47.483 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:49:47.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:49:47.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:49:47.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:49:47.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:49:47.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:49:47.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:49:47.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:49:47.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:49:47.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:49:47.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:49:47.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:49:47.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:49:47.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2020-11-16T21:49:47.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:49:47.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=293744be-da54-4da3-af29-27b6f2b2b768 2020-11-16T21:49:47.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 293744be-da54-4da3-af29-27b6f2b2b768' 2020-11-16T21:49:47.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:49:47.486 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 293744be-da54-4da3-af29-27b6f2b2b768 2020-11-16T21:49:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD787JfoYV9HhAARN7qcnBVVhUyC25GMvK8cg== 2020-11-16T21:49:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD787JfoYV9HhAARN7qcnBVVhUyC25GMvK8cg=="}' 2020-11-16T21:49:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 293744be-da54-4da3-af29-27b6f2b2b768 -i td/osd-backfill-stats/1/new.json 2020-11-16T21:49:47.972 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:49:47.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2020-11-16T21:49:47.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD787JfoYV9HhAARN7qcnBVVhUyC25GMvK8cg== --osd-uuid 293744be-da54-4da3-af29-27b6f2b2b768 2020-11-16T21:49:48.021 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:48.029+0000 7f496411bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:48.024 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:48.029+0000 7f496411bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:48.025 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:48.033+0000 7f496411bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:48.026 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:48.033+0000 7f496411bd80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:49:48.026 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:48.033+0000 7f496411bd80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:49:48.026 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:48.033+0000 7f496411bd80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:49:48.114 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:48.121+0000 7f496411bd80 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2020-11-16T21:49:52.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2020-11-16T21:49:52.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:49:52.133 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:49:52.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:49:52.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:49:52.657 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:49:52.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:49:52.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:49:52.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:49:52.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:49:52.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:49:52.685 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:52.693+0000 7f8ef8b8cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:52.688 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:52.693+0000 7f8ef8b8cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:52.689 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:52.693+0000 7f8ef8b8cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:49:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:49:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:49:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:49:53.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:49:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:49:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:49:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:49:53.022 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:49:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:49:53.223 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:53.229+0000 7f8ef8b8cd80 -1 Falling back to public interface 2020-11-16T21:49:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:49:54.111 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:54.117+0000 7f8ef8b8cd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:49:54.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:49:54.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:49:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:49:54.408 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:49:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:49:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:49:54.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:49:55.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:49:55.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:49:55.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:49:55.803 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:49:55.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:49:55.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:49:56.211 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/521074,v1:127.0.0.1:6811/521074] [v2:127.0.0.1:6812/521074,v1:127.0.0.1:6813/521074] exists,up 293744be-da54-4da3-af29-27b6f2b2b768 2020-11-16T21:49:56.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:49:56.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:49:56.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:49:56.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:575: TEST_backfill_ec_prim_out: run_osd td/osd-backfill-stats 2 2020-11-16T21:49:56.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:49:56.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:49:56.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:49:56.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:49:56.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2020-11-16T21:49:56.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:49:56.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:49:56.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:49:56.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:49:56.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2020-11-16T21:49:56.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2020-11-16T21:49:56.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:49:56.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:49:56.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:49:56.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:49:56.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:49:56.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:49:56.216 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:49:56.217 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:49:56.217 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:49:56.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:49:56.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:49:56.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:49:56.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:49:56.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:49:56.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:49:56.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:49:56.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:49:56.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:49:56.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:49:56.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:49:56.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:49:56.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2020-11-16T21:49:56.221 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:49:56.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a9e19c4a-558c-4683-a9b0-6e26035855f8 2020-11-16T21:49:56.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 a9e19c4a-558c-4683-a9b0-6e26035855f8' 2020-11-16T21:49:56.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:49:56.222 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 a9e19c4a-558c-4683-a9b0-6e26035855f8 2020-11-16T21:49:56.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAE9LJfTy6xDhAABKciOAeByoc3bDP9eRo3aw== 2020-11-16T21:49:56.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAE9LJfTy6xDhAABKciOAeByoc3bDP9eRo3aw=="}' 2020-11-16T21:49:56.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a9e19c4a-558c-4683-a9b0-6e26035855f8 -i td/osd-backfill-stats/2/new.json 2020-11-16T21:49:56.716 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:49:56.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2020-11-16T21:49:56.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAE9LJfTy6xDhAABKciOAeByoc3bDP9eRo3aw== --osd-uuid a9e19c4a-558c-4683-a9b0-6e26035855f8 2020-11-16T21:49:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:56.773+0000 7feee30b8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:56.787 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:56.793+0000 7feee30b8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:56.788 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:56.793+0000 7feee30b8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:49:56.789 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:56.793+0000 7feee30b8d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:49:56.789 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:56.793+0000 7feee30b8d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:49:56.789 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:56.793+0000 7feee30b8d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:49:56.852 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:49:56.857+0000 7feee30b8d80 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2020-11-16T21:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2020-11-16T21:50:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:50:00.985 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:50:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:50:00.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:50:01.489 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:50:01.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:50:01.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:50:01.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:50:01.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:50:01.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:50:01.520 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:01.525+0000 7fb0b61a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:01.523 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:01.529+0000 7fb0b61a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:01.529+0000 7fb0b61a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:01.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:50:01.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:50:01.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:50:01.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:50:01.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:50:01.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:50:01.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:50:01.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:50:01.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:50:01.869 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:50:02.059 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:02.065+0000 7fb0b61a2d80 -1 Falling back to public interface 2020-11-16T21:50:02.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:50:02.933 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:02.933+0000 7fb0b61a2d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:50:03.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:50:03.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:50:03.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:50:03.245 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:50:03.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:50:03.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:50:03.630 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/522095,v1:127.0.0.1:6819/522095] [v2:127.0.0.1:6820/522095,v1:127.0.0.1:6821/522095] exists,up a9e19c4a-558c-4683-a9b0-6e26035855f8 2020-11-16T21:50:03.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:50:03.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:50:03.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:50:03.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:576: TEST_backfill_ec_prim_out: run_osd td/osd-backfill-stats 3 2020-11-16T21:50:03.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:50:03.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:50:03.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:50:03.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:50:03.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2020-11-16T21:50:03.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:50:03.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:50:03.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:50:03.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:50:03.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2020-11-16T21:50:03.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2020-11-16T21:50:03.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:50:03.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:50:03.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:50:03.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:50:03.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:50:03.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:50:03.638 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:50:03.638 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:50:03.638 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:50:03.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:50:03.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:50:03.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:50:03.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:50:03.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:50:03.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:50:03.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:50:03.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:50:03.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:50:03.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:50:03.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:50:03.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:50:03.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2020-11-16T21:50:03.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:50:03.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=730a7395-7ccf-4ddc-b9d5-b2049d31f8e6 2020-11-16T21:50:03.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 730a7395-7ccf-4ddc-b9d5-b2049d31f8e6' 2020-11-16T21:50:03.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:50:03.643 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 730a7395-7ccf-4ddc-b9d5-b2049d31f8e6 2020-11-16T21:50:03.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAL9LJfMrHVJxAA30DEgb0dQdULxI0x3MewAg== 2020-11-16T21:50:03.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAL9LJfMrHVJxAA30DEgb0dQdULxI0x3MewAg=="}' 2020-11-16T21:50:03.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 730a7395-7ccf-4ddc-b9d5-b2049d31f8e6 -i td/osd-backfill-stats/3/new.json 2020-11-16T21:50:04.329 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:50:04.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2020-11-16T21:50:04.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAL9LJfMrHVJxAA30DEgb0dQdULxI0x3MewAg== --osd-uuid 730a7395-7ccf-4ddc-b9d5-b2049d31f8e6 2020-11-16T21:50:04.379 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:04.385+0000 7fa6d0a5cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:04.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:04.389+0000 7fa6d0a5cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:04.389+0000 7fa6d0a5cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:04.389+0000 7fa6d0a5cd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:50:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:04.389+0000 7fa6d0a5cd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:50:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:04.389+0000 7fa6d0a5cd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:50:04.425 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:04.429+0000 7fa6d0a5cd80 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2020-11-16T21:50:08.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2020-11-16T21:50:08.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:50:08.787 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:50:08.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:50:08.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:50:09.289 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:50:09.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:50:09.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:50:09.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:50:09.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:50:09.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:50:09.318 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:09.325+0000 7f13d39a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:09.320 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:09.325+0000 7f13d39a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:09.321 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:09.329+0000 7f13d39a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:09.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:50:09.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:50:09.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:50:09.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:50:09.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:50:09.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:50:09.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:50:09.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:50:09.685 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:50:09.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:50:09.859 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:09.865+0000 7f13d39a2d80 -1 Falling back to public interface 2020-11-16T21:50:10.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:50:10.699 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:10.705+0000 7f13d39a2d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:50:11.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:50:11.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:50:11.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:50:11.078 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:50:11.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:50:11.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:50:11.479 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/523089,v1:127.0.0.1:6827/523089] [v2:127.0.0.1:6828/523089,v1:127.0.0.1:6829/523089] exists,up 730a7395-7ccf-4ddc-b9d5-b2049d31f8e6 2020-11-16T21:50:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:50:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:50:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:50:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:577: TEST_backfill_ec_prim_out: run_osd td/osd-backfill-stats 4 2020-11-16T21:50:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:50:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:50:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:50:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:50:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2020-11-16T21:50:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:50:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:50:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:50:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:50:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2020-11-16T21:50:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2020-11-16T21:50:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:50:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:50:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:50:11.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:50:11.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:50:11.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:50:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:50:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:50:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:50:11.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:50:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:50:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:50:11.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:50:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:50:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:50:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:50:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:50:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:50:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:50:11.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:50:11.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:50:11.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2020-11-16T21:50:11.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:50:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=747c1359-f265-4687-8c24-6f41b425ba9b 2020-11-16T21:50:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 747c1359-f265-4687-8c24-6f41b425ba9b' 2020-11-16T21:50:11.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:50:11.487 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 747c1359-f265-4687-8c24-6f41b425ba9b 2020-11-16T21:50:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAT9LJfpFK/HhAAW502r8z9zNJqX6gYBno/Zw== 2020-11-16T21:50:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAT9LJfpFK/HhAAW502r8z9zNJqX6gYBno/Zw=="}' 2020-11-16T21:50:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 747c1359-f265-4687-8c24-6f41b425ba9b -i td/osd-backfill-stats/4/new.json 2020-11-16T21:50:12.125 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:50:12.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2020-11-16T21:50:12.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAT9LJfpFK/HhAAW502r8z9zNJqX6gYBno/Zw== --osd-uuid 747c1359-f265-4687-8c24-6f41b425ba9b 2020-11-16T21:50:12.170 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:12.177+0000 7fefffd59d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:12.181 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:12.189+0000 7fefffd59d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:12.183 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:12.189+0000 7fefffd59d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:12.184 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:12.189+0000 7fefffd59d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:50:12.184 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:12.189+0000 7fefffd59d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:50:12.184 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:12.189+0000 7fefffd59d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:50:12.254 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:12.261+0000 7fefffd59d80 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2020-11-16T21:50:16.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2020-11-16T21:50:16.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:50:16.842 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:50:16.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:50:16.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:50:17.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:50:17.346 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:50:17.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:50:17.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:50:17.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:50:17.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:50:17.377 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:17.381+0000 7fd536f4bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:17.380 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:17.385+0000 7fd536f4bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:17.382 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:17.389+0000 7fd536f4bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:50:17.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:50:17.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:50:17.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:50:17.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:50:17.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:50:17.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:50:17.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:50:17.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:50:17.725 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:50:17.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:50:17.935 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:17.941+0000 7fd536f4bd80 -1 Falling back to public interface 2020-11-16T21:50:18.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:50:18.769 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:50:18.769+0000 7fd536f4bd80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:50:19.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:50:19.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:50:19.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:50:19.108 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:50:19.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:50:19.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:50:19.508 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/524083,v1:127.0.0.1:6835/524083] [v2:127.0.0.1:6836/524083,v1:127.0.0.1:6837/524083] exists,up 747c1359-f265-4687-8c24-6f41b425ba9b 2020-11-16T21:50:19.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:50:19.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:50:19.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:50:19.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:579: TEST_backfill_ec_prim_out: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2020-11-16T21:50:20.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:580: TEST_backfill_ec_prim_out: create_pool test 1 1 erasure myprofile 2020-11-16T21:50:20.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-11-16T21:50:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T21:50:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:50:21.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:582: TEST_backfill_ec_prim_out: wait_for_clean 2020-11-16T21:50:21.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:50:21.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:50:21.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:50:21.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:50:21.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:50:21.931 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:50:21.931 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:50:21.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:50:21.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:50:21.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:50:22.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:50:22.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:50:22.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:50:22.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:50:22.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:50:22.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:50:22.596 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:50:22.597 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:50:22.597 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:50:22.597 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T21:50:22.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:50:22.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:50:22.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:50:22.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836489 2020-11-16T21:50:22.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836489 2020-11-16T21:50:22.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489' 2020-11-16T21:50:22.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:50:22.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:50:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705673 2020-11-16T21:50:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705673 2020-11-16T21:50:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-38654705673' 2020-11-16T21:50:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:50:22.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:50:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542150 2020-11-16T21:50:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542150 2020-11-16T21:50:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-38654705673 2-60129542150' 2020-11-16T21:50:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:50:23.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:50:23.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378629 2020-11-16T21:50:23.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378629 2020-11-16T21:50:23.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-38654705673 2-60129542150 3-81604378629' 2020-11-16T21:50:23.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:50:23.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:50:23.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215107 2020-11-16T21:50:23.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215107 2020-11-16T21:50:23.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-38654705673 2-60129542150 3-81604378629 4-103079215107' 2020-11-16T21:50:23.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:50:23.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2020-11-16T21:50:23.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:50:23.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:50:23.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836489 2020-11-16T21:50:23.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:50:23.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836489 2020-11-16T21:50:23.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-11-16T21:50:23.455 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836489 2020-11-16T21:50:23.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:50:23.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836489 2020-11-16T21:50:23.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:50:24.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:50:24.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:50:25.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836489 2020-11-16T21:50:25.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:50:25.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705673 2020-11-16T21:50:25.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:50:25.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:50:25.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:50:25.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705673 2020-11-16T21:50:25.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705673 2020-11-16T21:50:25.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2020-11-16T21:50:25.243 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705673 2020-11-16T21:50:25.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:50:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705673 -lt 38654705673 2020-11-16T21:50:25.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:50:25.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542150 2020-11-16T21:50:25.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:50:25.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:50:25.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542150 2020-11-16T21:50:25.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:50:25.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542150 2020-11-16T21:50:25.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2020-11-16T21:50:25.629 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542150 2020-11-16T21:50:25.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:50:26.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542151 -lt 60129542150 2020-11-16T21:50:26.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:50:26.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:50:26.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378629 2020-11-16T21:50:26.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:50:26.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378629 2020-11-16T21:50:26.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:50:26.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378629 2020-11-16T21:50:26.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378629' 2020-11-16T21:50:26.017 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378629 2020-11-16T21:50:26.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:50:26.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378629 -lt 81604378629 2020-11-16T21:50:26.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:50:26.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215107 2020-11-16T21:50:26.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:50:26.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:50:26.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215107 2020-11-16T21:50:26.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:50:26.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215107 2020-11-16T21:50:26.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215107' 2020-11-16T21:50:26.409 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215107 2020-11-16T21:50:26.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:50:26.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215108 -lt 103079215107 2020-11-16T21:50:26.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:50:26.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:50:26.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:50:27.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:50:27.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:50:27.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:50:27.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:50:27.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:50:27.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:50:27.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:50:27.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:50:27.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:50:27.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:50:27.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:50:27.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:50:28.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:50:28.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:50:28.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:50:28.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: seq 1 200 2020-11-16T21:50:28.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:28.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj1 /dev/null 2020-11-16T21:50:28.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:28.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj2 /dev/null 2020-11-16T21:50:28.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:28.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj3 /dev/null 2020-11-16T21:50:28.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:28.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj4 /dev/null 2020-11-16T21:50:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:28.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj5 /dev/null 2020-11-16T21:50:28.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:28.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj6 /dev/null 2020-11-16T21:50:29.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:29.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj7 /dev/null 2020-11-16T21:50:29.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:29.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj8 /dev/null 2020-11-16T21:50:29.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:29.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj9 /dev/null 2020-11-16T21:50:29.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:29.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj10 /dev/null 2020-11-16T21:50:29.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:29.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj11 /dev/null 2020-11-16T21:50:30.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:30.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj12 /dev/null 2020-11-16T21:50:30.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:30.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj13 /dev/null 2020-11-16T21:50:30.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:30.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj14 /dev/null 2020-11-16T21:50:30.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:30.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj15 /dev/null 2020-11-16T21:50:30.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:30.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj16 /dev/null 2020-11-16T21:50:30.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:30.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj17 /dev/null 2020-11-16T21:50:30.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:30.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj18 /dev/null 2020-11-16T21:50:31.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:31.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj19 /dev/null 2020-11-16T21:50:31.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:31.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj20 /dev/null 2020-11-16T21:50:31.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:31.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj21 /dev/null 2020-11-16T21:50:31.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:31.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj22 /dev/null 2020-11-16T21:50:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj23 /dev/null 2020-11-16T21:50:31.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:31.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj24 /dev/null 2020-11-16T21:50:32.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:32.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj25 /dev/null 2020-11-16T21:50:32.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:32.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj26 /dev/null 2020-11-16T21:50:32.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:32.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj27 /dev/null 2020-11-16T21:50:32.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:32.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj28 /dev/null 2020-11-16T21:50:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:32.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj29 /dev/null 2020-11-16T21:50:33.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:33.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj30 /dev/null 2020-11-16T21:50:33.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:33.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj31 /dev/null 2020-11-16T21:50:33.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:33.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj32 /dev/null 2020-11-16T21:50:33.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:33.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj33 /dev/null 2020-11-16T21:50:33.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:33.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj34 /dev/null 2020-11-16T21:50:33.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:33.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj35 /dev/null 2020-11-16T21:50:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj36 /dev/null 2020-11-16T21:50:34.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:34.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj37 /dev/null 2020-11-16T21:50:34.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:34.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj38 /dev/null 2020-11-16T21:50:34.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:34.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj39 /dev/null 2020-11-16T21:50:34.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:34.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj40 /dev/null 2020-11-16T21:50:34.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:34.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj41 /dev/null 2020-11-16T21:50:35.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:35.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj42 /dev/null 2020-11-16T21:50:35.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:35.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj43 /dev/null 2020-11-16T21:50:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj44 /dev/null 2020-11-16T21:50:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:35.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj45 /dev/null 2020-11-16T21:50:35.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj46 /dev/null 2020-11-16T21:50:35.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:35.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj47 /dev/null 2020-11-16T21:50:36.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:36.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj48 /dev/null 2020-11-16T21:50:36.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:36.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj49 /dev/null 2020-11-16T21:50:36.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:36.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj50 /dev/null 2020-11-16T21:50:36.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:36.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj51 /dev/null 2020-11-16T21:50:36.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:36.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj52 /dev/null 2020-11-16T21:50:37.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:37.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj53 /dev/null 2020-11-16T21:50:37.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:37.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj54 /dev/null 2020-11-16T21:50:37.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:37.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj55 /dev/null 2020-11-16T21:50:37.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:37.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj56 /dev/null 2020-11-16T21:50:37.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:37.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj57 /dev/null 2020-11-16T21:50:37.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:37.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj58 /dev/null 2020-11-16T21:50:37.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:37.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj59 /dev/null 2020-11-16T21:50:38.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:38.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj60 /dev/null 2020-11-16T21:50:38.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:38.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj61 /dev/null 2020-11-16T21:50:38.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:38.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj62 /dev/null 2020-11-16T21:50:38.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:38.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj63 /dev/null 2020-11-16T21:50:38.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:38.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj64 /dev/null 2020-11-16T21:50:38.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:38.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj65 /dev/null 2020-11-16T21:50:39.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:39.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj66 /dev/null 2020-11-16T21:50:39.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:39.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj67 /dev/null 2020-11-16T21:50:39.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:39.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj68 /dev/null 2020-11-16T21:50:39.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:39.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj69 /dev/null 2020-11-16T21:50:39.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:39.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj70 /dev/null 2020-11-16T21:50:40.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:40.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj71 /dev/null 2020-11-16T21:50:40.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:40.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj72 /dev/null 2020-11-16T21:50:40.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:40.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj73 /dev/null 2020-11-16T21:50:40.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:40.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj74 /dev/null 2020-11-16T21:50:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj75 /dev/null 2020-11-16T21:50:41.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:41.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj76 /dev/null 2020-11-16T21:50:41.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:41.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj77 /dev/null 2020-11-16T21:50:41.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:41.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj78 /dev/null 2020-11-16T21:50:41.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:41.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj79 /dev/null 2020-11-16T21:50:41.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:41.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj80 /dev/null 2020-11-16T21:50:41.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:41.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj81 /dev/null 2020-11-16T21:50:41.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:41.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj82 /dev/null 2020-11-16T21:50:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj83 /dev/null 2020-11-16T21:50:42.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:42.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj84 /dev/null 2020-11-16T21:50:42.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:42.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj85 /dev/null 2020-11-16T21:50:42.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:42.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj86 /dev/null 2020-11-16T21:50:42.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:42.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj87 /dev/null 2020-11-16T21:50:42.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:42.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj88 /dev/null 2020-11-16T21:50:43.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:43.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj89 /dev/null 2020-11-16T21:50:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:43.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj90 /dev/null 2020-11-16T21:50:43.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:43.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj91 /dev/null 2020-11-16T21:50:43.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:43.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj92 /dev/null 2020-11-16T21:50:43.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:43.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj93 /dev/null 2020-11-16T21:50:43.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:43.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj94 /dev/null 2020-11-16T21:50:44.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:44.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj95 /dev/null 2020-11-16T21:50:44.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:44.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj96 /dev/null 2020-11-16T21:50:44.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:44.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj97 /dev/null 2020-11-16T21:50:44.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:44.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj98 /dev/null 2020-11-16T21:50:45.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:45.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj99 /dev/null 2020-11-16T21:50:45.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:45.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj100 /dev/null 2020-11-16T21:50:45.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:45.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj101 /dev/null 2020-11-16T21:50:45.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:45.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj102 /dev/null 2020-11-16T21:50:45.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:45.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj103 /dev/null 2020-11-16T21:50:45.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:45.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj104 /dev/null 2020-11-16T21:50:45.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:45.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj105 /dev/null 2020-11-16T21:50:46.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:46.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj106 /dev/null 2020-11-16T21:50:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj107 /dev/null 2020-11-16T21:50:46.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:46.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj108 /dev/null 2020-11-16T21:50:46.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:46.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj109 /dev/null 2020-11-16T21:50:46.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:46.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj110 /dev/null 2020-11-16T21:50:46.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:46.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj111 /dev/null 2020-11-16T21:50:47.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:47.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj112 /dev/null 2020-11-16T21:50:47.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:47.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj113 /dev/null 2020-11-16T21:50:47.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:47.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj114 /dev/null 2020-11-16T21:50:47.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:47.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj115 /dev/null 2020-11-16T21:50:47.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:47.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj116 /dev/null 2020-11-16T21:50:48.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:48.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj117 /dev/null 2020-11-16T21:50:48.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:48.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj118 /dev/null 2020-11-16T21:50:48.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:48.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj119 /dev/null 2020-11-16T21:50:48.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:48.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj120 /dev/null 2020-11-16T21:50:48.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:48.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj121 /dev/null 2020-11-16T21:50:48.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:48.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj122 /dev/null 2020-11-16T21:50:49.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:49.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj123 /dev/null 2020-11-16T21:50:49.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:49.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj124 /dev/null 2020-11-16T21:50:49.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:49.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj125 /dev/null 2020-11-16T21:50:49.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:49.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj126 /dev/null 2020-11-16T21:50:49.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:49.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj127 /dev/null 2020-11-16T21:50:49.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:49.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj128 /dev/null 2020-11-16T21:50:50.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj129 /dev/null 2020-11-16T21:50:50.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:50.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj130 /dev/null 2020-11-16T21:50:50.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:50.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj131 /dev/null 2020-11-16T21:50:50.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:50.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj132 /dev/null 2020-11-16T21:50:50.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:50.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj133 /dev/null 2020-11-16T21:50:50.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:50.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj134 /dev/null 2020-11-16T21:50:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj135 /dev/null 2020-11-16T21:50:51.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:51.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj136 /dev/null 2020-11-16T21:50:51.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:51.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj137 /dev/null 2020-11-16T21:50:51.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:51.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj138 /dev/null 2020-11-16T21:50:51.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:51.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj139 /dev/null 2020-11-16T21:50:51.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:51.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj140 /dev/null 2020-11-16T21:50:52.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:52.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj141 /dev/null 2020-11-16T21:50:52.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:52.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj142 /dev/null 2020-11-16T21:50:52.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:52.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj143 /dev/null 2020-11-16T21:50:52.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:52.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj144 /dev/null 2020-11-16T21:50:52.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:52.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj145 /dev/null 2020-11-16T21:50:52.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:52.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj146 /dev/null 2020-11-16T21:50:53.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:53.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj147 /dev/null 2020-11-16T21:50:53.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:53.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj148 /dev/null 2020-11-16T21:50:53.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:53.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj149 /dev/null 2020-11-16T21:50:53.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:53.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj150 /dev/null 2020-11-16T21:50:53.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:53.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj151 /dev/null 2020-11-16T21:50:57.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj152 /dev/null 2020-11-16T21:50:57.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj153 /dev/null 2020-11-16T21:50:57.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj154 /dev/null 2020-11-16T21:50:57.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj155 /dev/null 2020-11-16T21:50:57.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj156 /dev/null 2020-11-16T21:50:57.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj157 /dev/null 2020-11-16T21:50:57.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj158 /dev/null 2020-11-16T21:50:57.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj159 /dev/null 2020-11-16T21:50:57.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj160 /dev/null 2020-11-16T21:50:57.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj161 /dev/null 2020-11-16T21:50:57.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj162 /dev/null 2020-11-16T21:50:57.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj163 /dev/null 2020-11-16T21:50:57.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj164 /dev/null 2020-11-16T21:50:57.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj165 /dev/null 2020-11-16T21:50:57.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj166 /dev/null 2020-11-16T21:50:57.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj167 /dev/null 2020-11-16T21:50:57.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj168 /dev/null 2020-11-16T21:50:57.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj169 /dev/null 2020-11-16T21:50:57.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj170 /dev/null 2020-11-16T21:50:57.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj171 /dev/null 2020-11-16T21:50:57.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj172 /dev/null 2020-11-16T21:50:57.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj173 /dev/null 2020-11-16T21:50:57.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj174 /dev/null 2020-11-16T21:50:57.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj175 /dev/null 2020-11-16T21:50:57.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:57.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj176 /dev/null 2020-11-16T21:50:58.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:58.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj177 /dev/null 2020-11-16T21:50:58.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:58.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj178 /dev/null 2020-11-16T21:50:58.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:58.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj179 /dev/null 2020-11-16T21:50:58.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:58.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj180 /dev/null 2020-11-16T21:50:58.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:58.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj181 /dev/null 2020-11-16T21:50:58.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:58.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj182 /dev/null 2020-11-16T21:50:59.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:59.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj183 /dev/null 2020-11-16T21:50:59.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:59.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj184 /dev/null 2020-11-16T21:50:59.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:59.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj185 /dev/null 2020-11-16T21:50:59.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:59.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj186 /dev/null 2020-11-16T21:50:59.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:59.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj187 /dev/null 2020-11-16T21:50:59.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:50:59.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj188 /dev/null 2020-11-16T21:51:00.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:00.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj189 /dev/null 2020-11-16T21:51:00.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:00.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj190 /dev/null 2020-11-16T21:51:00.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj191 /dev/null 2020-11-16T21:51:00.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:00.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj192 /dev/null 2020-11-16T21:51:00.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:00.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj193 /dev/null 2020-11-16T21:51:00.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:00.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj194 /dev/null 2020-11-16T21:51:01.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:01.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj195 /dev/null 2020-11-16T21:51:01.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:01.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj196 /dev/null 2020-11-16T21:51:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj197 /dev/null 2020-11-16T21:51:01.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:01.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj198 /dev/null 2020-11-16T21:51:01.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:01.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj199 /dev/null 2020-11-16T21:51:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2020-11-16T21:51:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj200 /dev/null 2020-11-16T21:51:02.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:589: TEST_backfill_ec_prim_out: get_pg test obj1 2020-11-16T21:51:02.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T21:51:02.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T21:51:02.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T21:51:02.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T21:51:02.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:589: TEST_backfill_ec_prim_out: local PG=1.0 2020-11-16T21:51:02.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:591: TEST_backfill_ec_prim_out: get_primary test obj1 2020-11-16T21:51:02.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:51:02.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:51:02.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:51:02.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:51:02.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:591: TEST_backfill_ec_prim_out: local primary=1 2020-11-16T21:51:02.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:593: TEST_backfill_ec_prim_out: ceph osd set nobackfill 2020-11-16T21:51:03.661 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T21:51:03.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:594: TEST_backfill_ec_prim_out: ceph osd out osd.1 2020-11-16T21:51:04.258 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. 2020-11-16T21:51:04.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:596: TEST_backfill_ec_prim_out: sleep 2 2020-11-16T21:51:06.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:597: TEST_backfill_ec_prim_out: get_primary test obj1 2020-11-16T21:51:06.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:51:06.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:51:06.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:51:06.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:51:06.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:597: TEST_backfill_ec_prim_out: primary=1 2020-11-16T21:51:06.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:598: TEST_backfill_ec_prim_out: ceph osd unset nobackfill 2020-11-16T21:51:07.379 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T21:51:07.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:599: TEST_backfill_ec_prim_out: ceph tell osd.1 get_latest_osdmap 2020-11-16T21:51:07.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:600: TEST_backfill_ec_prim_out: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T21:51:07.691 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T21:51:07.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:601: TEST_backfill_ec_prim_out: sleep 2 2020-11-16T21:51:09.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:603: TEST_backfill_ec_prim_out: wait_for_clean 2020-11-16T21:51:09.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:51:09.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:51:09.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:51:09.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:51:09.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:51:09.711 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:51:09.712 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:51:09.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:51:09.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:51:09.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:51:09.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:51:09.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:51:09.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:51:09.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:51:09.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:51:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:51:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:51:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:51:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:51:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T21:51:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:51:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:51:10.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:51:10.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836501 2020-11-16T21:51:10.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836501 2020-11-16T21:51:10.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501' 2020-11-16T21:51:10.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:51:10.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:51:10.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705685 2020-11-16T21:51:10.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705685 2020-11-16T21:51:10.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-38654705685' 2020-11-16T21:51:10.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:51:10.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:51:10.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542162 2020-11-16T21:51:10.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542162 2020-11-16T21:51:10.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-38654705685 2-60129542162' 2020-11-16T21:51:10.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:51:10.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:51:11.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378640 2020-11-16T21:51:11.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378640 2020-11-16T21:51:11.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-38654705685 2-60129542162 3-81604378640' 2020-11-16T21:51:11.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:51:11.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:51:11.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215119 2020-11-16T21:51:11.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215119 2020-11-16T21:51:11.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-38654705685 2-60129542162 3-81604378640 4-103079215119' 2020-11-16T21:51:11.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:51:11.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836501 2020-11-16T21:51:11.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:51:11.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:51:11.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836501 2020-11-16T21:51:11.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:51:11.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836501 2020-11-16T21:51:11.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2020-11-16T21:51:11.184 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836501 2020-11-16T21:51:11.185 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:51:11.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836499 -lt 21474836501 2020-11-16T21:51:11.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:51:12.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:51:12.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:51:12.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836501 -lt 21474836501 2020-11-16T21:51:12.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:51:12.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705685 2020-11-16T21:51:12.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:51:12.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:51:12.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705685 2020-11-16T21:51:12.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:51:12.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705685 2020-11-16T21:51:12.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705685' 2020-11-16T21:51:12.973 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705685 2020-11-16T21:51:12.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:51:13.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705685 -lt 38654705685 2020-11-16T21:51:13.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:51:13.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542162 2020-11-16T21:51:13.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:51:13.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:51:13.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542162 2020-11-16T21:51:13.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:51:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542162 2020-11-16T21:51:13.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542162' 2020-11-16T21:51:13.376 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542162 2020-11-16T21:51:13.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:51:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542162 -lt 60129542162 2020-11-16T21:51:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:51:13.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378640 2020-11-16T21:51:13.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:51:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:51:13.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378640 2020-11-16T21:51:13.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:51:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378640 2020-11-16T21:51:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378640' 2020-11-16T21:51:13.766 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378640 2020-11-16T21:51:13.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:51:14.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378641 -lt 81604378640 2020-11-16T21:51:14.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:51:14.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215119 2020-11-16T21:51:14.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:51:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:51:14.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215119 2020-11-16T21:51:14.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:51:14.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215119 2020-11-16T21:51:14.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215119' 2020-11-16T21:51:14.170 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215119 2020-11-16T21:51:14.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:51:14.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215119 -lt 103079215119 2020-11-16T21:51:14.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:51:14.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:14.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:15.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:51:15.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:15.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:15.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:15.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:15.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:15.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:15.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:15.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:15.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:15.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:15.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:15.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T21:51:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T21:51:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T21:51:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:15.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:15.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:15.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:15.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:15.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:15.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:15.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:15.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:16.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:16.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:16.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:16.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:16.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:16.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:17.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:17.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:17.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:17.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:17.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:17.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:17.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:17.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:17.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:17.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:17.778 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:17.778 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:18.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:18.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:18.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:18.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:18.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:18.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:18.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:51:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:51:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:18.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:18.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:18.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:18.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:18.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:18.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:18.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:18.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:19.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:19.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:19.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:19.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:19.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:20.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:51:20.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:51:20.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:20.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:20.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:20.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:20.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:20.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:20.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:20.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:20.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:20.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:20.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:20.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:20.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:20.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:20.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:21.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:21.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:21.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:21.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:21.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:21.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:21.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:21.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:21.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:21.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:51:21.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:51:21.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:21.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:21.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:21.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:21.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:21.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:21.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:21.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:21.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:21.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:21.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:22.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:22.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:22.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:22.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:22.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:22.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:22.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:22.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:22.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:22.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:22.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:22.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:22.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:23.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:51:23.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:51:23.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:23.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:23.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:23.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:23.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:23.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:23.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:23.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:23.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:23.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:23.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:23.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:23.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:23.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:23.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:23.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:23.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:23.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:23.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:23.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:23.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:23.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:23.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:23.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:24.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:24.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:24.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:24.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:24.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:24.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:24.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:24.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:24.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:24.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:24.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:24.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:24.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:24.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:24.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:24.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:25.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:25.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:25.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:25.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:25.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:25.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:25.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:25.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:25.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:25.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:25.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:25.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:25.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:25.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:25.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:25.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:25.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:25.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:25.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:26.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:26.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:26.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:26.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:26.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:26.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:26.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:27.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:51:27.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:51:27.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:27.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:27.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:27.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:27.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:27.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:27.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:27.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:27.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:27.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:27.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:27.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:27.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:27.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:27.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:28.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:28.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:28.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:28.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:28.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:28.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:28.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:28.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:28.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:28.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:28.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:28.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:28.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:28.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:28.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:28.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:28.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:28.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:28.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:28.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:28.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:28.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:29.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:29.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:29.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:29.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:29.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:29.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:29.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:29.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:29.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:29.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:29.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:29.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:29.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:30.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:30.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:30.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:30.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:30.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:30.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:30.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:30.235 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:30.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:30.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:30.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:30.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:30.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:30.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:30.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:30.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:30.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:31.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:31.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:31.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:31.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:31.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:31.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:31.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:31.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:31.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:31.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:31.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:31.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:31.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:31.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:31.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:31.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:31.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:31.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:31.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:31.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:31.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:31.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:31.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:32.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:32.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:32.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:32.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:32.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:32.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:32.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:32.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:32.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:32.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:51:32.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:51:32.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:32.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:32.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:33.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:33.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:33.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:33.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:33.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:33.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:33.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:33.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:33.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:33.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:33.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:33.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:33.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:33.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:33.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:33.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:33.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:33.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:33.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:33.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:33.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:34.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:34.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:34.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:34.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:34.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:34.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:34.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:34.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:34.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:34.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:34.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:34.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:35.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:35.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:35.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:35.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:35.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:35.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:35.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:35.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:35.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:35.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:35.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:36.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:36.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:36.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:36.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:36.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:36.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:36.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:36.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:36.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:36.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:36.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:36.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:36.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:51:37.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:51:37.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:37.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:37.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:37.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:37.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:37.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:37.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:37.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:37.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:37.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:37.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:37.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:37.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:37.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:38.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:38.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:38.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:38.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:38.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:38.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:38.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:38.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:38.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:38.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:38.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:38.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:38.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:38.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:38.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:38.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:38.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:38.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:38.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:38.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:38.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:38.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:39.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:39.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:39.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:39.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:39.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:39.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:39.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:39.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:39.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:39.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:39.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:39.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:39.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:40.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:40.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:40.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:40.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:40.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:40.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:40.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:40.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:40.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:40.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:40.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:40.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:40.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:40.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:40.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:40.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:40.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:41.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:41.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:41.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:41.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:41.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:41.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:41.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:41.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:41.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:41.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:41.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:41.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:41.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:41.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:41.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:41.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:41.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:41.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:41.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:41.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:41.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:41.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:42.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:42.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:42.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:42.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:42.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:42.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:42.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:42.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:42.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:42.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:42.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:42.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:42.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:42.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:51:42.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:51:42.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:42.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:42.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:43.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:43.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:43.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:43.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:43.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:43.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:43.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:43.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:43.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:43.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:43.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:43.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:43.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:43.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:43.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:43.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:43.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:43.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:43.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:43.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:44.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:44.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:44.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:44.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:44.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:44.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:44.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:44.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:44.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:44.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:44.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:44.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:44.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:44.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:44.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:44.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:44.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:45.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:45.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:45.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:45.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:45.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:45.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:45.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:45.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:45.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:45.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:51:45.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:51:45.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:45.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:45.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:45.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:45.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:45.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:45.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:45.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:45.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:45.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:45.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:46.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:46.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:46.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:46.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:46.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:46.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:46.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:46.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:46.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:46.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:46.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:46.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:46.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:47.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:51:47.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:51:47.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:47.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:47.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:47.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:47.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:47.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:47.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:47.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:47.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:47.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:47.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:47.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:47.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:47.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:47.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:48.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:48.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:48.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:48.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:48.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:48.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:48.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:48.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:48.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:48.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:51:48.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:51:48.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:48.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:48.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:48.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:48.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:48.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:48.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:48.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:48.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:49.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:49.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:49.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:49.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:49.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:49.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:49.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:49.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:49.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:49.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:49.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:49.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:49.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:50.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:51:50.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:51:50.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:50.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:50.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:50.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:50.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:50.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:50.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:50.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:50.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:50.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:50.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:50.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:51:50.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:50.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:50.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:51.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:51:51.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:51:51.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:51:51.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:51:51.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:51:51.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:51:51.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:51:51.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:51:51.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:51:51.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:51:51.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:51:51.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:51:51.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:51:51.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:51:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:51:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:51:51.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:51:51.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:51:51.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:51:51.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:51:51.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:51:51.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:51:52.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:51:52.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:51:52.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:51:52.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:51:52.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:51:52.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:51:52.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:51:52.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:605: TEST_backfill_ec_prim_out: expr 200 '*' 3 2020-11-16T21:51:52.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:605: TEST_backfill_ec_prim_out: local misplaced=600 2020-11-16T21:51:52.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:606: TEST_backfill_ec_prim_out: check td/osd-backfill-stats 1.0 1 erasure 0 0 200 0 2020-11-16T21:51:52.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2020-11-16T21:51:52.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2020-11-16T21:51:52.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2020-11-16T21:51:52.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=erasure 2020-11-16T21:51:52.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=0 2020-11-16T21:51:52.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2020-11-16T21:51:52.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=200 2020-11-16T21:51:52.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2020-11-16T21:51:52.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2020-11-16T21:51:52.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2020-11-16T21:51:52.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2020-11-16T21:51:52.682 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2020-11-16T21:51:52.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2020-11-16T21:51:52.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2020-11-16T21:51:52.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log 2020-11-16T21:51:52.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2020-11-16T21:51:52.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2020-11-16T21:51:52.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2020-11-16T21:51:52.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2020-11-16T21:51:52.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2020-11-16T21:51:52.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' erasure = erasure ']' 2020-11-16T21:51:52.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:89: check: addp=p 2020-11-16T21:51:52.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2020-11-16T21:51:52.703 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:51:52.703 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2020-11-16T21:51:52.704 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2020-11-16T21:51:52.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[4,0,2]/[1,0,2]' 2020-11-16T21:51:52.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[4,0,2]/[1,0,2]' ']' 2020-11-16T21:51:52.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2020-11-16T21:51:52.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:51:52.722 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:51:52.723 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2020-11-16T21:51:52.723 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2020-11-16T21:51:52.723 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [4,0,2]/[1,0,2]p' 2020-11-16T21:51:52.723 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:51:52.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=0 2020-11-16T21:51:52.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 0 ']' 2020-11-16T21:51:52.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 0 0 2020-11-16T21:51:52.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2020-11-16T21:51:52.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:51:52.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2020-11-16T21:51:52.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:51:52.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:51:52.736 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:51:52.736 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:51:52.736 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2020-11-16T21:51:52.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=0 2020-11-16T21:51:52.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 0 ']' 2020-11-16T21:51:52.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 0 0 2020-11-16T21:51:52.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2020-11-16T21:51:52.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:51:52.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:51:52.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:51:52.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2020-11-16T21:51:52.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:51:52.750 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:51:52.750 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [4,0,2]/[1,0,2]p' 2020-11-16T21:51:52.751 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2020-11-16T21:51:52.751 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2020-11-16T21:51:52.751 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:51:52.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=200 2020-11-16T21:51:52.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 200 ']' 2020-11-16T21:51:52.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 200 200 2020-11-16T21:51:52.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2020-11-16T21:51:52.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:51:52.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2020-11-16T21:51:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:51:52.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:51:52.758 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:51:52.759 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2020-11-16T21:51:52.759 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:51:52.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=1 2020-11-16T21:51:52.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 1 ']' 2020-11-16T21:51:52.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 1 0 2020-11-16T21:51:52.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2020-11-16T21:51:52.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:51:52.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:51:52.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:51:52.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2020-11-16T21:51:52.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:608: TEST_backfill_ec_prim_out: delete_pool test 2020-11-16T21:51:52.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T21:51:52.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T21:51:53.278 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T21:51:53.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:609: TEST_backfill_ec_prim_out: kill_daemons td/osd-backfill-stats 2020-11-16T21:51:53.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:51:53.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:51:53.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:51:53.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:51:53.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:51:53.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:51:53.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2020-11-16T21:51:53.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:51:53.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:51:53.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:51:53.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:51:53.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:51:53.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:51:53.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:51:53.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:51:53.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:51:53.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:51:53.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:51:53.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:51:53.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:51:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:51:53.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:51:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:51:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:51:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:51:53.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:51:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:51:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:51:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:51:53.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:51:53.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:51:53.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:51:53.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:51:53.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:51:53.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:51:53.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2020-11-16T21:51:53.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2020-11-16T21:51:53.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2020-11-16T21:51:53.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2020-11-16T21:51:53.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:51:53.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:51:53.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:51:53.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:51:53.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:51:53.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:51:53.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:51:53.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:51:53.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:51:53.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:51:53.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:51:53.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:51:53.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:51:53.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:51:53.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:51:53.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:51:53.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:51:53.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:51:53.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:51:53.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:51:53.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:51:53.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:51:53.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:51:53.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:51:53.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:51:53.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:51:53.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:51:53.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:51:53.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2020-11-16T21:51:53.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:51:53.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:51:53.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:51:53.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.465868 2020-11-16T21:51:53.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:51:53.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:51:53.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:51:53.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2020-11-16T21:51:53.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_out td/osd-backfill-stats 2020-11-16T21:51:53.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:230: TEST_backfill_out: local dir=td/osd-backfill-stats 2020-11-16T21:51:53.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:232: TEST_backfill_out: run_mon td/osd-backfill-stats a 2020-11-16T21:51:53.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2020-11-16T21:51:53.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:51:53.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:51:53.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:51:53.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2020-11-16T21:51:53.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2020-11-16T21:51:53.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:51:53.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:51:53.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:51:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:51:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:51:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:51:53.782 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:51:53.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:51:53.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:51:53.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:51:53.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:51:53.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:51:53.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:51:53.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:51:53.994 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:51:53.994 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:51:53.994 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:51:53.995 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:51:53.995 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:51:53.995 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:51:53.996 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:51:53.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:51:53.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get fsid 2020-11-16T21:51:54.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:51:54.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:51:54.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:51:54.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:51:54.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:51:54.109 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:51:54.109 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:51:54.109 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:51:54.109 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:51:54.110 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:51:54.110 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:51:54.110 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:51:54.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:51:54.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get mon_host 2020-11-16T21:51:54.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:233: TEST_backfill_out: run_mgr td/osd-backfill-stats x 2020-11-16T21:51:54.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2020-11-16T21:51:54.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:51:54.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:51:54.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:51:54.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2020-11-16T21:51:54.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:51:54.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:51:54.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:51:54.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:51:54.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:51:54.732 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:51:54.732 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:51:54.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:51:54.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:51:54.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:51:54.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:234: TEST_backfill_out: run_osd td/osd-backfill-stats 0 2020-11-16T21:51:54.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:51:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:51:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:51:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:51:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2020-11-16T21:51:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:51:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:51:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:51:54.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:51:54.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2020-11-16T21:51:54.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2020-11-16T21:51:54.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:51:54.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:51:54.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:51:54.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:51:54.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:51:54.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:51:54.769 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:51:54.769 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:51:54.770 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:51:54.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:51:54.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:51:54.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:51:54.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:51:54.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:51:54.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:51:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:51:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:51:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:51:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:51:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:51:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:51:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2020-11-16T21:51:54.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:51:54.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7c723c3f-bf2c-467d-a7f5-256202f59968 2020-11-16T21:51:54.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 7c723c3f-bf2c-467d-a7f5-256202f59968' 2020-11-16T21:51:54.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:51:54.773 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 7c723c3f-bf2c-467d-a7f5-256202f59968 2020-11-16T21:51:54.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB69LJf26+DLxAAepQzt00xgb313XD5IBU3+A== 2020-11-16T21:51:54.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB69LJf26+DLxAAepQzt00xgb313XD5IBU3+A=="}' 2020-11-16T21:51:54.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7c723c3f-bf2c-467d-a7f5-256202f59968 -i td/osd-backfill-stats/0/new.json 2020-11-16T21:51:55.151 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:51:55.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2020-11-16T21:51:55.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB69LJf26+DLxAAepQzt00xgb313XD5IBU3+A== --osd-uuid 7c723c3f-bf2c-467d-a7f5-256202f59968 2020-11-16T21:51:55.198 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:51:55.207+0000 7f8095809d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:51:55.200 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:51:55.207+0000 7f8095809d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:51:55.202 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:51:55.211+0000 7f8095809d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:51:55.202 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:51:55.211+0000 7f8095809d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:51:55.203 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:51:55.211+0000 7f8095809d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:51:55.203 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:51:55.211+0000 7f8095809d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:51:55.259 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:51:55.267+0000 7f8095809d80 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2020-11-16T21:51:59.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2020-11-16T21:51:59.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:51:59.574 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:51:59.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:51:59.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:52:00.062 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:52:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:52:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:52:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:52:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:52:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:52:00.093 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:00.099+0000 7f12386efd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:00.095 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:00.103+0000 7f12386efd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:00.097 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:00.103+0000 7f12386efd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:00.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:52:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:52:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:52:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:52:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:52:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:00.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:52:00.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:00.459 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:52:00.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:52:00.635 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:00.643+0000 7f12386efd80 -1 Falling back to public interface 2020-11-16T21:52:00.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:52:01.484 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:01.491+0000 7f12386efd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:52:01.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:52:01.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:01.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:52:01.837 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:52:01.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:01.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:52:02.245 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/532721,v1:127.0.0.1:6803/532721] [v2:127.0.0.1:6804/532721,v1:127.0.0.1:6805/532721] exists,up 7c723c3f-bf2c-467d-a7f5-256202f59968 2020-11-16T21:52:02.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:52:02.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:52:02.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:52:02.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:235: TEST_backfill_out: run_osd td/osd-backfill-stats 1 2020-11-16T21:52:02.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:52:02.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:52:02.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:52:02.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:52:02.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2020-11-16T21:52:02.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:52:02.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:52:02.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:52:02.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:52:02.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2020-11-16T21:52:02.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2020-11-16T21:52:02.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:52:02.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:52:02.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:52:02.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:52:02.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:52:02.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:52:02.250 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:52:02.250 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:52:02.250 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:52:02.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:52:02.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:52:02.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:52:02.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:52:02.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:52:02.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:52:02.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:52:02.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:52:02.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:52:02.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:52:02.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:52:02.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:52:02.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2020-11-16T21:52:02.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:52:02.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c25f8fc7-2387-4c98-b4c4-4baf8aafbe3b 2020-11-16T21:52:02.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 c25f8fc7-2387-4c98-b4c4-4baf8aafbe3b' 2020-11-16T21:52:02.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:52:02.254 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 c25f8fc7-2387-4c98-b4c4-4baf8aafbe3b 2020-11-16T21:52:02.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCC9LJfGayZEBAAxlcWFsf34CQlfwGPgekVnw== 2020-11-16T21:52:02.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCC9LJfGayZEBAAxlcWFsf34CQlfwGPgekVnw=="}' 2020-11-16T21:52:02.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c25f8fc7-2387-4c98-b4c4-4baf8aafbe3b -i td/osd-backfill-stats/1/new.json 2020-11-16T21:52:02.760 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:52:02.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2020-11-16T21:52:02.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCC9LJfGayZEBAAxlcWFsf34CQlfwGPgekVnw== --osd-uuid c25f8fc7-2387-4c98-b4c4-4baf8aafbe3b 2020-11-16T21:52:02.803 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:02.811+0000 7fb29bfe7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:02.807 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:02.815+0000 7fb29bfe7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:02.809 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:02.815+0000 7fb29bfe7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:02.809 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:02.819+0000 7fb29bfe7d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:52:02.810 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:02.819+0000 7fb29bfe7d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:52:02.810 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:02.819+0000 7fb29bfe7d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:52:02.912 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:02.919+0000 7fb29bfe7d80 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2020-11-16T21:52:06.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2020-11-16T21:52:06.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:52:06.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:52:06.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:52:06.945 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:52:07.443 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:52:07.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:52:07.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:52:07.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:52:07.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:52:07.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:52:07.474 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:07.483+0000 7fa4bb6f1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:07.476 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:07.483+0000 7fa4bb6f1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:07.477 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:07.487+0000 7fa4bb6f1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:07.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:52:07.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:52:07.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:52:07.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:52:07.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:52:07.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:07.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:52:07.820 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:52:07.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:07.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:52:08.023 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:08.031+0000 7fa4bb6f1d80 -1 Falling back to public interface 2020-11-16T21:52:08.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:52:08.645 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:08.655+0000 7fa4bb6f1d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:52:09.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:52:09.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:09.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:52:09.219 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:52:09.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:09.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:52:09.614 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/533715,v1:127.0.0.1:6811/533715] [v2:127.0.0.1:6812/533715,v1:127.0.0.1:6813/533715] exists,up c25f8fc7-2387-4c98-b4c4-4baf8aafbe3b 2020-11-16T21:52:09.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:52:09.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:52:09.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:52:09.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:236: TEST_backfill_out: run_osd td/osd-backfill-stats 2 2020-11-16T21:52:09.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:52:09.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:52:09.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:52:09.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:52:09.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2020-11-16T21:52:09.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:52:09.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:52:09.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:52:09.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:52:09.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2020-11-16T21:52:09.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2020-11-16T21:52:09.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:52:09.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:52:09.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:52:09.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:52:09.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:52:09.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:52:09.619 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:52:09.619 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:52:09.619 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:52:09.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:52:09.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:52:09.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:52:09.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:52:09.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:52:09.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:52:09.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:52:09.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:52:09.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:52:09.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:52:09.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:52:09.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:52:09.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2020-11-16T21:52:09.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:52:09.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=462d24b8-fe58-4a27-8ba7-cf80ae96f39e 2020-11-16T21:52:09.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 462d24b8-fe58-4a27-8ba7-cf80ae96f39e' 2020-11-16T21:52:09.622 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 462d24b8-fe58-4a27-8ba7-cf80ae96f39e 2020-11-16T21:52:09.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:52:09.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCJ9LJfJU+4JhAACe+/f+9FYID/dkPepY3YRw== 2020-11-16T21:52:09.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCJ9LJfJU+4JhAACe+/f+9FYID/dkPepY3YRw=="}' 2020-11-16T21:52:09.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 462d24b8-fe58-4a27-8ba7-cf80ae96f39e -i td/osd-backfill-stats/2/new.json 2020-11-16T21:52:10.096 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:52:10.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2020-11-16T21:52:10.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCJ9LJfJU+4JhAACe+/f+9FYID/dkPepY3YRw== --osd-uuid 462d24b8-fe58-4a27-8ba7-cf80ae96f39e 2020-11-16T21:52:10.143 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:10.151+0000 7fbd5fbb7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:10.162 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:10.171+0000 7fbd5fbb7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:10.164 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:10.171+0000 7fbd5fbb7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:10.164 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:10.171+0000 7fbd5fbb7d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:52:10.164 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:10.171+0000 7fbd5fbb7d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:52:10.164 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:10.171+0000 7fbd5fbb7d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:52:10.228 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:10.235+0000 7fbd5fbb7d80 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2020-11-16T21:52:14.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2020-11-16T21:52:14.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:52:14.517 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:52:14.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:52:14.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:52:15.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:52:15.023 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:52:15.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:52:15.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:52:15.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:52:15.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:52:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:15.063+0000 7fbd2d589d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:15.063+0000 7fbd2d589d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:15.063+0000 7fbd2d589d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:52:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:52:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:52:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:52:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:52:15.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:15.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:52:15.433 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:52:15.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:15.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:52:15.595 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:15.603+0000 7fbd2d589d80 -1 Falling back to public interface 2020-11-16T21:52:15.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:52:16.434 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:16.434+0000 7fbd2d589d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:52:16.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:52:16.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:16.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:52:16.787 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:52:16.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:16.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:52:17.162 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/534709,v1:127.0.0.1:6819/534709] [v2:127.0.0.1:6820/534709,v1:127.0.0.1:6821/534709] exists,up 462d24b8-fe58-4a27-8ba7-cf80ae96f39e 2020-11-16T21:52:17.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:52:17.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:52:17.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:52:17.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:237: TEST_backfill_out: run_osd td/osd-backfill-stats 3 2020-11-16T21:52:17.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:52:17.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:52:17.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:52:17.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:52:17.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2020-11-16T21:52:17.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:52:17.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:52:17.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:52:17.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:52:17.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2020-11-16T21:52:17.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2020-11-16T21:52:17.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:52:17.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:52:17.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:52:17.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:52:17.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:52:17.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:52:17.167 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:52:17.167 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:52:17.168 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:52:17.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:52:17.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:52:17.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:52:17.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:52:17.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:52:17.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:52:17.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:52:17.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:52:17.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:52:17.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:52:17.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:52:17.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:52:17.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2020-11-16T21:52:17.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:52:17.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0f3ac286-5495-4f40-8a6d-036a212f3e9b 2020-11-16T21:52:17.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 0f3ac286-5495-4f40-8a6d-036a212f3e9b' 2020-11-16T21:52:17.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:52:17.171 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 0f3ac286-5495-4f40-8a6d-036a212f3e9b 2020-11-16T21:52:17.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCR9LJfNTGLCxAAI0xL3MSw0QAYSog3FqXr9g== 2020-11-16T21:52:17.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCR9LJfNTGLCxAAI0xL3MSw0QAYSog3FqXr9g=="}' 2020-11-16T21:52:17.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0f3ac286-5495-4f40-8a6d-036a212f3e9b -i td/osd-backfill-stats/3/new.json 2020-11-16T21:52:17.666 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:52:17.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2020-11-16T21:52:17.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCR9LJfNTGLCxAAI0xL3MSw0QAYSog3FqXr9g== --osd-uuid 0f3ac286-5495-4f40-8a6d-036a212f3e9b 2020-11-16T21:52:17.711 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:17.718+0000 7fb7d6d7ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:17.713 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:17.722+0000 7fb7d6d7ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:17.715 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:17.722+0000 7fb7d6d7ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:17.716 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:17.722+0000 7fb7d6d7ed80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:52:17.716 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:17.722+0000 7fb7d6d7ed80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:52:17.716 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:17.722+0000 7fb7d6d7ed80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:52:17.775 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:17.782+0000 7fb7d6d7ed80 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2020-11-16T21:52:21.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2020-11-16T21:52:21.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:52:21.955 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:52:21.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:52:21.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:52:22.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:52:22.454 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:52:22.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:52:22.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:52:22.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:52:22.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:52:22.485 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:22.490+0000 7f7d06782d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:22.487 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:22.494+0000 7f7d06782d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:22.489 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:22.494+0000 7f7d06782d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:22.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:52:22.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:52:22.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:52:22.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:52:22.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:52:22.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:22.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:52:22.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:52:22.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:22.849 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:52:23.035 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:23.042+0000 7f7d06782d80 -1 Falling back to public interface 2020-11-16T21:52:23.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:52:23.865 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:23.874+0000 7f7d06782d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:52:24.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:52:24.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:24.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:52:24.246 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:52:24.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:24.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:52:24.654 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/535703,v1:127.0.0.1:6827/535703] [v2:127.0.0.1:6828/535703,v1:127.0.0.1:6829/535703] exists,up 0f3ac286-5495-4f40-8a6d-036a212f3e9b 2020-11-16T21:52:24.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:52:24.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:52:24.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:52:24.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:238: TEST_backfill_out: run_osd td/osd-backfill-stats 4 2020-11-16T21:52:24.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:52:24.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:52:24.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:52:24.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:52:24.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2020-11-16T21:52:24.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:52:24.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:52:24.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:52:24.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:52:24.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2020-11-16T21:52:24.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2020-11-16T21:52:24.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:52:24.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:52:24.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:52:24.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:52:24.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:52:24.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:52:24.659 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:52:24.659 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:52:24.659 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:52:24.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:52:24.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:52:24.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:52:24.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:52:24.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:52:24.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:52:24.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:52:24.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:52:24.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:52:24.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:52:24.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:52:24.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:52:24.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2020-11-16T21:52:24.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:52:24.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=808c552e-cc99-4c53-ba4e-0b7917f8646a 2020-11-16T21:52:24.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 808c552e-cc99-4c53-ba4e-0b7917f8646a' 2020-11-16T21:52:24.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:52:24.663 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 808c552e-cc99-4c53-ba4e-0b7917f8646a 2020-11-16T21:52:24.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCY9LJfKfMuKRAAj8gAyyQ/KxIRkrIQqT5ZbQ== 2020-11-16T21:52:24.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCY9LJfKfMuKRAAj8gAyyQ/KxIRkrIQqT5ZbQ=="}' 2020-11-16T21:52:24.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 808c552e-cc99-4c53-ba4e-0b7917f8646a -i td/osd-backfill-stats/4/new.json 2020-11-16T21:52:25.249 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:52:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2020-11-16T21:52:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCY9LJfKfMuKRAAj8gAyyQ/KxIRkrIQqT5ZbQ== --osd-uuid 808c552e-cc99-4c53-ba4e-0b7917f8646a 2020-11-16T21:52:25.291 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:25.298+0000 7ff7ad872d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:25.302+0000 7ff7ad872d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:25.302+0000 7ff7ad872d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:25.302+0000 7ff7ad872d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:52:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:25.302+0000 7ff7ad872d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:52:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:25.302+0000 7ff7ad872d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:52:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:25.346+0000 7ff7ad872d80 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2020-11-16T21:52:29.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2020-11-16T21:52:29.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:52:29.678 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:52:29.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:52:29.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:52:30.186 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:52:30.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:52:30.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:52:30.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:52:30.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:52:30.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:52:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:30.222+0000 7f631435ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:30.226+0000 7f631435ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:30.226+0000 7f631435ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:30.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:52:30.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:52:30.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:52:30.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:52:30.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:52:30.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:30.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:52:30.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:30.578 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:52:30.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:52:30.763 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:30.770+0000 7f631435ad80 -1 Falling back to public interface 2020-11-16T21:52:30.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:52:31.637 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:31.638+0000 7f631435ad80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:52:31.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:52:31.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:31.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:52:31.976 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:52:31.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:31.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:52:32.409 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/536697,v1:127.0.0.1:6835/536697] [v2:127.0.0.1:6836/536697,v1:127.0.0.1:6837/536697] exists,up 808c552e-cc99-4c53-ba4e-0b7917f8646a 2020-11-16T21:52:32.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:52:32.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:52:32.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:52:32.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:239: TEST_backfill_out: run_osd td/osd-backfill-stats 5 2020-11-16T21:52:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:52:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:52:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T21:52:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:52:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2020-11-16T21:52:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:52:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:52:32.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:52:32.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:52:32.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2020-11-16T21:52:32.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2020-11-16T21:52:32.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:52:32.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:52:32.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:52:32.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:52:32.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:52:32.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:52:32.413 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:52:32.413 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:52:32.413 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:52:32.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:52:32.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:52:32.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:52:32.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:52:32.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:52:32.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:52:32.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:52:32.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:52:32.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:52:32.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:52:32.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:52:32.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:52:32.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2020-11-16T21:52:32.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:52:32.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8524a800-8dfb-460c-b65f-96e12b51c42b 2020-11-16T21:52:32.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 8524a800-8dfb-460c-b65f-96e12b51c42b' 2020-11-16T21:52:32.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:52:32.417 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 8524a800-8dfb-460c-b65f-96e12b51c42b 2020-11-16T21:52:32.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCg9LJfLFdiGhAAKJLcTq/1dr7x8BxKTXqLCQ== 2020-11-16T21:52:32.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCg9LJfLFdiGhAAKJLcTq/1dr7x8BxKTXqLCQ=="}' 2020-11-16T21:52:32.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8524a800-8dfb-460c-b65f-96e12b51c42b -i td/osd-backfill-stats/5/new.json 2020-11-16T21:52:33.056 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T21:52:33.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2020-11-16T21:52:33.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCg9LJfLFdiGhAAKJLcTq/1dr7x8BxKTXqLCQ== --osd-uuid 8524a800-8dfb-460c-b65f-96e12b51c42b 2020-11-16T21:52:33.101 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:33.106+0000 7fa1b7ca7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:33.104 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:33.110+0000 7fa1b7ca7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:33.105 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:33.114+0000 7fa1b7ca7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:33.106 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:33.114+0000 7fa1b7ca7d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:52:33.106 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:33.114+0000 7fa1b7ca7d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:52:33.106 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:33.114+0000 7fa1b7ca7d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:52:33.196 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:33.202+0000 7fa1b7ca7d80 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2020-11-16T21:52:37.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2020-11-16T21:52:37.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:52:37.276 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T21:52:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T21:52:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:52:37.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T21:52:37.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:52:37.779 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T21:52:37.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:52:37.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:52:37.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:52:37.799 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:37.806+0000 7ffaff816d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:37.802 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:37.810+0000 7ffaff816d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:37.804 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:37.810+0000 7ffaff816d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:52:38.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T21:52:38.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:52:38.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T21:52:38.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:52:38.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:52:38.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:38.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:52:38.143 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:52:38.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:38.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:52:38.351 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:38.358+0000 7ffaff816d80 -1 Falling back to public interface 2020-11-16T21:52:38.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:52:39.134 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:52:39.138+0000 7ffaff816d80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T21:52:39.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:52:39.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:52:39.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:52:39.554 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:52:39.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:52:39.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:52:39.956 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/537691,v1:127.0.0.1:6843/537691] [v2:127.0.0.1:6844/537691,v1:127.0.0.1:6845/537691] exists,up 8524a800-8dfb-460c-b65f-96e12b51c42b 2020-11-16T21:52:39.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:52:39.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:52:39.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:52:39.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:241: TEST_backfill_out: create_pool test 1 1 2020-11-16T21:52:39.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T21:52:40.624 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T21:52:40.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:52:41.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:242: TEST_backfill_out: ceph osd pool set test size 2 2020-11-16T21:52:42.608 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T21:52:42.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:243: TEST_backfill_out: sleep 5 2020-11-16T21:52:47.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:245: TEST_backfill_out: wait_for_clean 2020-11-16T21:52:47.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:52:47.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:52:47.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:52:47.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:52:47.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:52:47.627 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:52:47.627 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:52:47.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:52:47.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:52:47.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:52:47.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:52:47.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:52:47.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:52:47.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:52:47.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:52:48.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:52:48.261 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:52:48.261 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:52:48.261 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:52:48.261 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:52:48.261 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:52:48.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:52:48.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:52:48.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:52:48.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836493 2020-11-16T21:52:48.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836493 2020-11-16T21:52:48.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493' 2020-11-16T21:52:48.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:52:48.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:52:48.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705674 2020-11-16T21:52:48.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705674 2020-11-16T21:52:48.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-38654705674' 2020-11-16T21:52:48.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:52:48.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:52:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542153 2020-11-16T21:52:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542153 2020-11-16T21:52:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-38654705674 2-60129542153' 2020-11-16T21:52:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:52:48.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:52:48.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411336 2020-11-16T21:52:48.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411336 2020-11-16T21:52:48.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-38654705674 2-60129542153 3-77309411336' 2020-11-16T21:52:48.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:52:48.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:52:49.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247814 2020-11-16T21:52:49.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247814 2020-11-16T21:52:49.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-38654705674 2-60129542153 3-77309411336 4-98784247814' 2020-11-16T21:52:49.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:52:49.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:52:49.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084292 2020-11-16T21:52:49.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084292 2020-11-16T21:52:49.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-38654705674 2-60129542153 3-77309411336 4-98784247814 5-120259084292' 2020-11-16T21:52:49.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:52:49.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836493 2020-11-16T21:52:49.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:52:49.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:52:49.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836493 2020-11-16T21:52:49.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:52:49.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836493 2020-11-16T21:52:49.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-11-16T21:52:49.260 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836493 2020-11-16T21:52:49.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:52:49.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836493 -lt 21474836493 2020-11-16T21:52:49.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:52:49.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705674 2020-11-16T21:52:49.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:52:49.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:52:49.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705674 2020-11-16T21:52:49.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:52:49.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705674 2020-11-16T21:52:49.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2020-11-16T21:52:49.657 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705674 2020-11-16T21:52:49.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:52:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705672 -lt 38654705674 2020-11-16T21:52:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:52:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:52:51.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:52:51.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705675 -lt 38654705674 2020-11-16T21:52:51.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:52:51.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542153 2020-11-16T21:52:51.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:52:51.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:52:51.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:52:51.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542153 2020-11-16T21:52:51.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542153 2020-11-16T21:52:51.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2020-11-16T21:52:51.454 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542153 2020-11-16T21:52:51.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:52:51.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542153 -lt 60129542153 2020-11-16T21:52:51.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:52:51.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-77309411336 2020-11-16T21:52:51.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:52:51.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:52:51.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:52:51.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411336 2020-11-16T21:52:51.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411336 2020-11-16T21:52:51.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 77309411336' 2020-11-16T21:52:51.837 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 77309411336 2020-11-16T21:52:51.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:52:52.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411337 -lt 77309411336 2020-11-16T21:52:52.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:52:52.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247814 2020-11-16T21:52:52.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:52:52.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:52:52.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247814 2020-11-16T21:52:52.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:52:52.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247814 2020-11-16T21:52:52.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247814' 2020-11-16T21:52:52.224 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247814 2020-11-16T21:52:52.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:52:52.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247814 -lt 98784247814 2020-11-16T21:52:52.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:52:52.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084292 2020-11-16T21:52:52.602 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:52:52.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:52:52.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084292 2020-11-16T21:52:52.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:52:52.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084292 2020-11-16T21:52:52.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084292' 2020-11-16T21:52:52.606 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 120259084292 2020-11-16T21:52:52.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:52:53.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084293 -lt 120259084292 2020-11-16T21:52:53.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:52:53.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:52:53.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:52:53.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:52:53.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:52:53.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:52:53.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:52:53.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:52:53.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:52:53.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:52:53.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:52:53.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:52:53.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:52:53.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:52:53.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:52:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:52:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:52:54.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:52:54.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: seq 1 200 2020-11-16T21:52:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:54.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj1 /dev/null 2020-11-16T21:52:54.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:54.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj2 /dev/null 2020-11-16T21:52:54.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:54.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj3 /dev/null 2020-11-16T21:52:54.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:54.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj4 /dev/null 2020-11-16T21:52:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:54.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj5 /dev/null 2020-11-16T21:52:54.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:54.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj6 /dev/null 2020-11-16T21:52:55.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:55.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj7 /dev/null 2020-11-16T21:52:55.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:55.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj8 /dev/null 2020-11-16T21:52:55.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:55.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj9 /dev/null 2020-11-16T21:52:55.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:55.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj10 /dev/null 2020-11-16T21:52:55.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:55.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj11 /dev/null 2020-11-16T21:52:55.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:55.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj12 /dev/null 2020-11-16T21:52:55.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:55.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj13 /dev/null 2020-11-16T21:52:55.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:55.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj14 /dev/null 2020-11-16T21:52:55.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:55.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj15 /dev/null 2020-11-16T21:52:56.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:56.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj16 /dev/null 2020-11-16T21:52:56.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:56.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj17 /dev/null 2020-11-16T21:52:56.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:56.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj18 /dev/null 2020-11-16T21:52:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:56.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj19 /dev/null 2020-11-16T21:52:56.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:56.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj20 /dev/null 2020-11-16T21:52:56.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:56.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj21 /dev/null 2020-11-16T21:52:56.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:56.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj22 /dev/null 2020-11-16T21:52:56.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:56.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj23 /dev/null 2020-11-16T21:52:57.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:57.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj24 /dev/null 2020-11-16T21:52:57.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:57.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj25 /dev/null 2020-11-16T21:52:57.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:57.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj26 /dev/null 2020-11-16T21:52:57.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:57.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj27 /dev/null 2020-11-16T21:52:57.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:57.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj28 /dev/null 2020-11-16T21:52:57.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:57.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj29 /dev/null 2020-11-16T21:52:57.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:57.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj30 /dev/null 2020-11-16T21:52:57.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:57.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj31 /dev/null 2020-11-16T21:52:58.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:58.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj32 /dev/null 2020-11-16T21:52:58.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:58.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj33 /dev/null 2020-11-16T21:52:58.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:58.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj34 /dev/null 2020-11-16T21:52:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:58.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj35 /dev/null 2020-11-16T21:52:58.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:58.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj36 /dev/null 2020-11-16T21:52:58.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:58.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj37 /dev/null 2020-11-16T21:52:58.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:58.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj38 /dev/null 2020-11-16T21:52:58.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:58.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj39 /dev/null 2020-11-16T21:52:58.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:58.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj40 /dev/null 2020-11-16T21:52:59.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:59.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj41 /dev/null 2020-11-16T21:52:59.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:59.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj42 /dev/null 2020-11-16T21:52:59.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:59.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj43 /dev/null 2020-11-16T21:52:59.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:59.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj44 /dev/null 2020-11-16T21:52:59.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:59.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj45 /dev/null 2020-11-16T21:52:59.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:59.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj46 /dev/null 2020-11-16T21:52:59.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:59.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj47 /dev/null 2020-11-16T21:52:59.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:52:59.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj48 /dev/null 2020-11-16T21:53:00.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:00.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj49 /dev/null 2020-11-16T21:53:00.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:00.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj50 /dev/null 2020-11-16T21:53:00.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:00.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj51 /dev/null 2020-11-16T21:53:00.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:00.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj52 /dev/null 2020-11-16T21:53:00.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:00.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj53 /dev/null 2020-11-16T21:53:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:00.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj54 /dev/null 2020-11-16T21:53:00.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:00.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj55 /dev/null 2020-11-16T21:53:00.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:00.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj56 /dev/null 2020-11-16T21:53:00.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:00.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj57 /dev/null 2020-11-16T21:53:01.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:01.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj58 /dev/null 2020-11-16T21:53:01.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:01.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj59 /dev/null 2020-11-16T21:53:01.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:01.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj60 /dev/null 2020-11-16T21:53:01.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:01.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj61 /dev/null 2020-11-16T21:53:01.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:01.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj62 /dev/null 2020-11-16T21:53:01.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:01.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj63 /dev/null 2020-11-16T21:53:01.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:01.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj64 /dev/null 2020-11-16T21:53:01.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:01.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj65 /dev/null 2020-11-16T21:53:01.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:01.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj66 /dev/null 2020-11-16T21:53:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj67 /dev/null 2020-11-16T21:53:02.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:02.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj68 /dev/null 2020-11-16T21:53:02.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:02.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj69 /dev/null 2020-11-16T21:53:02.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:02.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj70 /dev/null 2020-11-16T21:53:02.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:02.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj71 /dev/null 2020-11-16T21:53:02.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:02.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj72 /dev/null 2020-11-16T21:53:02.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:02.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj73 /dev/null 2020-11-16T21:53:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:02.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj74 /dev/null 2020-11-16T21:53:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:02.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj75 /dev/null 2020-11-16T21:53:03.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:03.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj76 /dev/null 2020-11-16T21:53:03.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:03.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj77 /dev/null 2020-11-16T21:53:03.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:03.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj78 /dev/null 2020-11-16T21:53:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj79 /dev/null 2020-11-16T21:53:03.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:03.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj80 /dev/null 2020-11-16T21:53:03.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:03.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj81 /dev/null 2020-11-16T21:53:03.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:03.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj82 /dev/null 2020-11-16T21:53:03.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:03.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj83 /dev/null 2020-11-16T21:53:03.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj84 /dev/null 2020-11-16T21:53:04.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:04.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj85 /dev/null 2020-11-16T21:53:04.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:04.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj86 /dev/null 2020-11-16T21:53:04.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:04.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj87 /dev/null 2020-11-16T21:53:04.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:04.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj88 /dev/null 2020-11-16T21:53:04.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:04.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj89 /dev/null 2020-11-16T21:53:04.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:04.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj90 /dev/null 2020-11-16T21:53:04.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:04.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj91 /dev/null 2020-11-16T21:53:04.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:04.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj92 /dev/null 2020-11-16T21:53:05.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:05.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj93 /dev/null 2020-11-16T21:53:05.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:05.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj94 /dev/null 2020-11-16T21:53:05.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:05.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj95 /dev/null 2020-11-16T21:53:05.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:05.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj96 /dev/null 2020-11-16T21:53:05.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:05.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj97 /dev/null 2020-11-16T21:53:05.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:05.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj98 /dev/null 2020-11-16T21:53:05.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:05.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj99 /dev/null 2020-11-16T21:53:05.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:05.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj100 /dev/null 2020-11-16T21:53:05.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:05.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj101 /dev/null 2020-11-16T21:53:06.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:06.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj102 /dev/null 2020-11-16T21:53:06.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:06.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj103 /dev/null 2020-11-16T21:53:06.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:06.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj104 /dev/null 2020-11-16T21:53:06.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:06.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj105 /dev/null 2020-11-16T21:53:06.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:06.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj106 /dev/null 2020-11-16T21:53:06.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:06.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj107 /dev/null 2020-11-16T21:53:06.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:06.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj108 /dev/null 2020-11-16T21:53:06.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:06.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj109 /dev/null 2020-11-16T21:53:06.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:06.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj110 /dev/null 2020-11-16T21:53:07.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:07.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj111 /dev/null 2020-11-16T21:53:07.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:07.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj112 /dev/null 2020-11-16T21:53:07.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:07.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj113 /dev/null 2020-11-16T21:53:07.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:07.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj114 /dev/null 2020-11-16T21:53:07.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:07.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj115 /dev/null 2020-11-16T21:53:07.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:07.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj116 /dev/null 2020-11-16T21:53:07.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:07.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj117 /dev/null 2020-11-16T21:53:07.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:07.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj118 /dev/null 2020-11-16T21:53:08.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:08.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj119 /dev/null 2020-11-16T21:53:08.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:08.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj120 /dev/null 2020-11-16T21:53:08.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:08.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj121 /dev/null 2020-11-16T21:53:08.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:08.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj122 /dev/null 2020-11-16T21:53:08.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:08.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj123 /dev/null 2020-11-16T21:53:08.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:08.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj124 /dev/null 2020-11-16T21:53:08.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:08.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj125 /dev/null 2020-11-16T21:53:08.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:08.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj126 /dev/null 2020-11-16T21:53:08.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:08.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj127 /dev/null 2020-11-16T21:53:09.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:09.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj128 /dev/null 2020-11-16T21:53:09.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:09.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj129 /dev/null 2020-11-16T21:53:09.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:09.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj130 /dev/null 2020-11-16T21:53:09.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:09.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj131 /dev/null 2020-11-16T21:53:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj132 /dev/null 2020-11-16T21:53:09.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:09.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj133 /dev/null 2020-11-16T21:53:09.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:09.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj134 /dev/null 2020-11-16T21:53:09.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:09.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj135 /dev/null 2020-11-16T21:53:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj136 /dev/null 2020-11-16T21:53:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:10.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj137 /dev/null 2020-11-16T21:53:10.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:10.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj138 /dev/null 2020-11-16T21:53:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj139 /dev/null 2020-11-16T21:53:10.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:10.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj140 /dev/null 2020-11-16T21:53:10.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj141 /dev/null 2020-11-16T21:53:10.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:10.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj142 /dev/null 2020-11-16T21:53:10.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:10.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj143 /dev/null 2020-11-16T21:53:10.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:10.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj144 /dev/null 2020-11-16T21:53:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj145 /dev/null 2020-11-16T21:53:11.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:11.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj146 /dev/null 2020-11-16T21:53:11.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:11.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj147 /dev/null 2020-11-16T21:53:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:11.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj148 /dev/null 2020-11-16T21:53:11.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:11.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj149 /dev/null 2020-11-16T21:53:11.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj150 /dev/null 2020-11-16T21:53:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj151 /dev/null 2020-11-16T21:53:11.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:11.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj152 /dev/null 2020-11-16T21:53:12.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:12.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj153 /dev/null 2020-11-16T21:53:12.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:12.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj154 /dev/null 2020-11-16T21:53:12.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:12.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj155 /dev/null 2020-11-16T21:53:12.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:12.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj156 /dev/null 2020-11-16T21:53:12.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:12.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj157 /dev/null 2020-11-16T21:53:12.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:12.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj158 /dev/null 2020-11-16T21:53:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj159 /dev/null 2020-11-16T21:53:12.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:12.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj160 /dev/null 2020-11-16T21:53:12.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:12.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj161 /dev/null 2020-11-16T21:53:13.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:13.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj162 /dev/null 2020-11-16T21:53:13.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:13.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj163 /dev/null 2020-11-16T21:53:13.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:13.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj164 /dev/null 2020-11-16T21:53:13.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:13.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj165 /dev/null 2020-11-16T21:53:13.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:13.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj166 /dev/null 2020-11-16T21:53:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj167 /dev/null 2020-11-16T21:53:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj168 /dev/null 2020-11-16T21:53:13.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:13.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj169 /dev/null 2020-11-16T21:53:14.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:14.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj170 /dev/null 2020-11-16T21:53:14.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:14.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj171 /dev/null 2020-11-16T21:53:14.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:14.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj172 /dev/null 2020-11-16T21:53:14.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:14.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj173 /dev/null 2020-11-16T21:53:14.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:14.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj174 /dev/null 2020-11-16T21:53:14.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:14.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj175 /dev/null 2020-11-16T21:53:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:14.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj176 /dev/null 2020-11-16T21:53:14.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:14.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj177 /dev/null 2020-11-16T21:53:14.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:14.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj178 /dev/null 2020-11-16T21:53:15.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:15.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj179 /dev/null 2020-11-16T21:53:15.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:15.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj180 /dev/null 2020-11-16T21:53:15.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:15.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj181 /dev/null 2020-11-16T21:53:15.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:15.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj182 /dev/null 2020-11-16T21:53:15.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:15.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj183 /dev/null 2020-11-16T21:53:15.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:15.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj184 /dev/null 2020-11-16T21:53:15.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:15.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj185 /dev/null 2020-11-16T21:53:15.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:15.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj186 /dev/null 2020-11-16T21:53:15.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:15.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj187 /dev/null 2020-11-16T21:53:16.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:16.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj188 /dev/null 2020-11-16T21:53:16.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:16.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj189 /dev/null 2020-11-16T21:53:16.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:16.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj190 /dev/null 2020-11-16T21:53:16.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:16.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj191 /dev/null 2020-11-16T21:53:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj192 /dev/null 2020-11-16T21:53:16.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:16.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj193 /dev/null 2020-11-16T21:53:16.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:16.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj194 /dev/null 2020-11-16T21:53:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj195 /dev/null 2020-11-16T21:53:16.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:16.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj196 /dev/null 2020-11-16T21:53:17.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:17.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj197 /dev/null 2020-11-16T21:53:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj198 /dev/null 2020-11-16T21:53:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj199 /dev/null 2020-11-16T21:53:17.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2020-11-16T21:53:17.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj200 /dev/null 2020-11-16T21:53:17.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:252: TEST_backfill_out: get_pg test obj1 2020-11-16T21:53:17.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T21:53:17.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T21:53:17.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T21:53:17.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T21:53:17.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:252: TEST_backfill_out: local PG=1.0 2020-11-16T21:53:17.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:254: TEST_backfill_out: get_primary test obj1 2020-11-16T21:53:17.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:53:17.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:53:17.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:53:17.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:53:18.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:254: TEST_backfill_out: local primary=1 2020-11-16T21:53:18.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:256: TEST_backfill_out: ceph osd set nobackfill 2020-11-16T21:53:19.119 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T21:53:19.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:257: TEST_backfill_out: get_not_primary test obj1 2020-11-16T21:53:19.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2020-11-16T21:53:19.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2020-11-16T21:53:19.133 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2020-11-16T21:53:19.133 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:53:19.133 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:53:19.134 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:53:19.134 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:53:19.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T21:53:19.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2020-11-16T21:53:19.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T21:53:19.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:257: TEST_backfill_out: ceph osd out osd.0 2020-11-16T21:53:20.535 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T21:53:20.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:258: TEST_backfill_out: sleep 2 2020-11-16T21:53:22.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:259: TEST_backfill_out: ceph osd unset nobackfill 2020-11-16T21:53:23.343 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T21:53:23.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:261: TEST_backfill_out: wait_for_clean 2020-11-16T21:53:23.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:53:23.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:53:23.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:53:23.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:53:23.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:53:23.357 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:53:23.357 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:53:23.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:53:23.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:53:23.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:53:23.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:53:23.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:53:23.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:53:23.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:53:23.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:53:23.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:53:23.950 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:53:23.950 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:53:23.950 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:53:23.950 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:53:23.950 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:53:23.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:53:23.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:53:23.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:53:24.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836502 2020-11-16T21:53:24.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836502 2020-11-16T21:53:24.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502' 2020-11-16T21:53:24.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:53:24.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:53:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705683 2020-11-16T21:53:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705683 2020-11-16T21:53:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-38654705683' 2020-11-16T21:53:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:53:24.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:53:24.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542162 2020-11-16T21:53:24.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542162 2020-11-16T21:53:24.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-38654705683 2-60129542162' 2020-11-16T21:53:24.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:53:24.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:53:24.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411345 2020-11-16T21:53:24.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411345 2020-11-16T21:53:24.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-38654705683 2-60129542162 3-77309411345' 2020-11-16T21:53:24.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:53:24.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:53:24.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247823 2020-11-16T21:53:24.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247823 2020-11-16T21:53:24.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-38654705683 2-60129542162 3-77309411345 4-98784247823' 2020-11-16T21:53:24.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:53:24.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:53:24.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084301 2020-11-16T21:53:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084301 2020-11-16T21:53:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-38654705683 2-60129542162 3-77309411345 4-98784247823 5-120259084301' 2020-11-16T21:53:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:53:24.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836502 2020-11-16T21:53:24.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:53:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:53:24.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836502 2020-11-16T21:53:24.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:53:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836502 2020-11-16T21:53:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2020-11-16T21:53:24.974 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836502 2020-11-16T21:53:24.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:53:25.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836502 -lt 21474836502 2020-11-16T21:53:25.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:53:25.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705683 2020-11-16T21:53:25.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:53:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:53:25.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705683 2020-11-16T21:53:25.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:53:25.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705683 2020-11-16T21:53:25.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705683' 2020-11-16T21:53:25.377 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705683 2020-11-16T21:53:25.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:53:25.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705683 -lt 38654705683 2020-11-16T21:53:25.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:53:25.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542162 2020-11-16T21:53:25.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:53:25.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:53:25.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542162 2020-11-16T21:53:25.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:53:25.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542162 2020-11-16T21:53:25.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542162' 2020-11-16T21:53:25.779 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542162 2020-11-16T21:53:25.779 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:53:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542160 -lt 60129542162 2020-11-16T21:53:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:53:27.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:53:27.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:53:27.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542162 -lt 60129542162 2020-11-16T21:53:27.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:53:27.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-77309411345 2020-11-16T21:53:27.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:53:27.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:53:27.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411345 2020-11-16T21:53:27.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:53:27.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411345 2020-11-16T21:53:27.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 77309411345' 2020-11-16T21:53:27.571 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 77309411345 2020-11-16T21:53:27.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:53:27.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411346 -lt 77309411345 2020-11-16T21:53:27.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:53:27.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247823 2020-11-16T21:53:27.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:53:27.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:53:27.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247823 2020-11-16T21:53:27.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:53:27.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247823 2020-11-16T21:53:27.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247823' 2020-11-16T21:53:27.983 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247823 2020-11-16T21:53:27.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:53:28.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247823 -lt 98784247823 2020-11-16T21:53:28.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:53:28.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084301 2020-11-16T21:53:28.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:53:28.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:53:28.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084301 2020-11-16T21:53:28.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:53:28.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084301 2020-11-16T21:53:28.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084301' 2020-11-16T21:53:28.397 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 120259084301 2020-11-16T21:53:28.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:53:28.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084302 -lt 120259084301 2020-11-16T21:53:28.782 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:53:28.782 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:28.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:29.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:53:29.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:29.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:29.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:29.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:29.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:29.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:29.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:29.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:29.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:29.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:29.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:30.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:30.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T21:53:30.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T21:53:30.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T21:53:30.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:30.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:30.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:30.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:30.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:30.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:30.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:30.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:30.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:30.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:30.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:30.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:31.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:31.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:31.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:31.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:31.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:31.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:31.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:31.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:31.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:31.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:53:31.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:53:31.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:31.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:31.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:31.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:31.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:31.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:31.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:31.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:31.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:31.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:31.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:32.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:32.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:32.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:32.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:32.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:32.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:32.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:32.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:32.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:32.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:32.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:32.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:32.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:33.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:53:33.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:53:33.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:33.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:33.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:33.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:33.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:33.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:33.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:33.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:33.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:33.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:33.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:33.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:33.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:33.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:33.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:34.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:34.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:34.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:53:34.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:53:34.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:34.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:34.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:34.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:34.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:34.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:34.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:34.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:34.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:34.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:34.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:35.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:35.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:35.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:35.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:35.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:35.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:35.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:35.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:35.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:35.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:35.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:35.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:35.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:36.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:53:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:53:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:36.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:36.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:36.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:36.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:36.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:36.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:36.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:36.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:36.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:36.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:36.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:36.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:36.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:36.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:36.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:36.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:36.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:36.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:36.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:36.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:36.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:37.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:53:37.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:53:37.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:37.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:37.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:37.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:37.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:37.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:37.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:37.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:37.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:37.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:37.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:37.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:37.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:37.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:37.926 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:38.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:38.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:38.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:38.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:38.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:38.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:38.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:38.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:38.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:53:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:53:38.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:38.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:38.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:38.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:38.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:38.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:38.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:38.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:38.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:38.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:39.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:39.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:39.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:39.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:39.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:39.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:39.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:39.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:39.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:39.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:39.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:39.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:39.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:40.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:53:40.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:53:40.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:40.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:40.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:40.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:40.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:40.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:40.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:40.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:40.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:40.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:40.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:40.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:40.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:40.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:40.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:41.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:41.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:41.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:41.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:41.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:41.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:41.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:41.263 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:41.263 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:41.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2020-11-16T21:53:41.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2020-11-16T21:53:41.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:41.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:41.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:41.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:41.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:41.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:41.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:41.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:41.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:41.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:41.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:42.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:42.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:42.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:42.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:42.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:42.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:42.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:42.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:42.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:42.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:42.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:42.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:42.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:43.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:53:43.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:53:43.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:43.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:43.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:43.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:43.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:43.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:43.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:43.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:43.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:43.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:43.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:43.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:43.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:43.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:43.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:44.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:44.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:44.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:44.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:44.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:44.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:44.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:44.071 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:44.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:53:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:53:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:44.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:44.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:44.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:44.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:44.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:44.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:44.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:44.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:44.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:44.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:45.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:45.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:45.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:45.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:45.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:45.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:45.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:45.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:45.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:45.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:45.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:45.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:45.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:53:45.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:53:45.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:45.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:45.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:46.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:46.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:46.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:46.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:46.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:46.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:46.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:46.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:46.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:46.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:46.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:46.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:46.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:46.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:47.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2020-11-16T21:53:47.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2020-11-16T21:53:47.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:47.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:47.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:47.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:47.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:47.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:47.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:47.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:47.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:47.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:47.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:47.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:47.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:48.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:48.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:48.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:48.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:48.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:48.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:48.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:48.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:48.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:48.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:53:48.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:53:48.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:48.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:48.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:48.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:48.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:48.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:48.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:48.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:48.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:48.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:48.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:49.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:49.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:49.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:49.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:49.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:49.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:49.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:49.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:49.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:49.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:49.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:49.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:49.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:50.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:53:50.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:53:50.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:50.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:50.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:50.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:50.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:50.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:50.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:50.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:50.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:50.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:50.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:50.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:50.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:50.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:50.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:51.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:51.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:51.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:51.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:51.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:51.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:51.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:51.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:51.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:51.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2020-11-16T21:53:51.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2020-11-16T21:53:51.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:51.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:51.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:51.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:51.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:51.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:51.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:51.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:51.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:51.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:51.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:52.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:52.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:52.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:52.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:52.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:52.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:52.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:52.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:52.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:52.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:52.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:52.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:52.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:53.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:53:53.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:53:53.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:53.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:53.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:53.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:53.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:53.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:53.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:53.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:53.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:53.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:53.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:53.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:53.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:53.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:53.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:54.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:54.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:54.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:53:54.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:53:54.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:54.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:54.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:54.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:54.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:54.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:54.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:54.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:54.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:54.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:54.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:55.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:53:55.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:55.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:55.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:55.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:53:55.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:53:55.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:53:55.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:53:55.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:53:55.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:53:55.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:53:55.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:53:55.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:53:56.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:53:56.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:53:56.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:53:56.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:53:56.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:53:56.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:53:56.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:53:56.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:53:56.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:53:56.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:53:56.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:53:56.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:53:56.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:53:56.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:53:56.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:53:56.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:53:56.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:53:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:53:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:53:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:53:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:263: TEST_backfill_out: check td/osd-backfill-stats 1.0 1 replicated 0 0 200 0 2020-11-16T21:53:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2020-11-16T21:53:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2020-11-16T21:53:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2020-11-16T21:53:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2020-11-16T21:53:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=0 2020-11-16T21:53:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2020-11-16T21:53:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=200 2020-11-16T21:53:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2020-11-16T21:53:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2020-11-16T21:53:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2020-11-16T21:53:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2020-11-16T21:53:57.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2020-11-16T21:53:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2020-11-16T21:53:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2020-11-16T21:53:57.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2020-11-16T21:53:57.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2020-11-16T21:53:57.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2020-11-16T21:53:57.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2020-11-16T21:53:57.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2020-11-16T21:53:57.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2020-11-16T21:53:57.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2020-11-16T21:53:57.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2020-11-16T21:53:57.069 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:53:57.069 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2020-11-16T21:53:57.070 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2020-11-16T21:53:57.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[1,2]/[1,0]' 2020-11-16T21:53:57.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[1,2]/[1,0]' ']' 2020-11-16T21:53:57.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2020-11-16T21:53:57.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:53:57.090 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:53:57.090 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [1,2]/[1,0] ' 2020-11-16T21:53:57.090 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2020-11-16T21:53:57.090 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2020-11-16T21:53:57.091 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:53:57.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=0 2020-11-16T21:53:57.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 0 ']' 2020-11-16T21:53:57.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 0 0 2020-11-16T21:53:57.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2020-11-16T21:53:57.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:53:57.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2020-11-16T21:53:57.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:53:57.098 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:53:57.099 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:53:57.099 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2020-11-16T21:53:57.099 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:53:57.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=0 2020-11-16T21:53:57.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 0 ']' 2020-11-16T21:53:57.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 0 0 2020-11-16T21:53:57.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2020-11-16T21:53:57.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:53:57.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:53:57.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:53:57.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2020-11-16T21:53:57.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:53:57.112 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:53:57.112 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [1,2]/[1,0] ' 2020-11-16T21:53:57.113 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2020-11-16T21:53:57.113 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2020-11-16T21:53:57.113 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:53:57.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=200 2020-11-16T21:53:57.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 200 ']' 2020-11-16T21:53:57.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 200 200 2020-11-16T21:53:57.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2020-11-16T21:53:57.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:53:57.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2020-11-16T21:53:57.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:53:57.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:53:57.122 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2020-11-16T21:53:57.122 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:53:57.122 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:53:57.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=1 2020-11-16T21:53:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 1 ']' 2020-11-16T21:53:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 1 0 2020-11-16T21:53:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2020-11-16T21:53:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:53:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:53:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:53:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2020-11-16T21:53:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:265: TEST_backfill_out: delete_pool test 2020-11-16T21:53:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T21:53:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T21:53:57.621 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T21:53:57.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:266: TEST_backfill_out: kill_daemons td/osd-backfill-stats 2020-11-16T21:53:57.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:53:57.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:53:57.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:53:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:53:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:53:57.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:53:57.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2020-11-16T21:53:57.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:53:57.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:53:57.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:53:57.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:53:57.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:53:57.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:53:57.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:53:57.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:53:57.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:53:57.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:53:57.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:53:57.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:53:57.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:53:57.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:53:57.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:53:57.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:53:57.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:53:57.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:53:57.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:53:57.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:53:57.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:53:57.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:53:57.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:53:57.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:53:57.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:53:57.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:53:57.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:53:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:53:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2020-11-16T21:53:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2020-11-16T21:53:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2020-11-16T21:53:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2020-11-16T21:53:57.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:53:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:53:57.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:53:57.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:53:57.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:53:57.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:53:57.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:53:57.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:53:57.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:53:57.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:53:57.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:53:57.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:53:57.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:53:57.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:53:57.805 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:53:57.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:53:57.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:53:57.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:53:57.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:53:57.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:53:57.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:53:57.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:53:57.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:53:57.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:53:57.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:53:57.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:53:57.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:53:57.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:53:57.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2020-11-16T21:53:57.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:53:57.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:53:57.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:53:57.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.465868 2020-11-16T21:53:57.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:53:57.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:53:57.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:53:57.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2020-11-16T21:53:57.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_out2 td/osd-backfill-stats 2020-11-16T21:53:57.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:329: TEST_backfill_out2: local dir=td/osd-backfill-stats 2020-11-16T21:53:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:331: TEST_backfill_out2: run_mon td/osd-backfill-stats a 2020-11-16T21:53:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2020-11-16T21:53:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:53:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:53:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:53:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2020-11-16T21:53:57.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2020-11-16T21:53:58.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:53:58.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:53:58.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:53:58.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:53:58.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:53:58.152 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:53:58.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:53:58.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:53:58.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:53:58.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:53:58.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:53:58.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:53:58.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:53:58.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:53:58.358 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:53:58.358 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:53:58.358 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:53:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:53:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:53:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:53:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:53:58.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:53:58.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get fsid 2020-11-16T21:53:58.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:53:58.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:53:58.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:53:58.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:53:58.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:53:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:53:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:53:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:53:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:53:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:53:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:53:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:53:58.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:53:58.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get mon_host 2020-11-16T21:53:58.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:332: TEST_backfill_out2: run_mgr td/osd-backfill-stats x 2020-11-16T21:53:58.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2020-11-16T21:53:58.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:53:58.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:53:58.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:53:58.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2020-11-16T21:53:58.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:53:59.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:53:59.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:53:59.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:53:59.160 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:53:59.160 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:53:59.160 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:53:59.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:53:59.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:53:59.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:53:59.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:333: TEST_backfill_out2: run_osd td/osd-backfill-stats 0 2020-11-16T21:53:59.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:53:59.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:53:59.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:53:59.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:53:59.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2020-11-16T21:53:59.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:53:59.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:53:59.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:53:59.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:53:59.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2020-11-16T21:53:59.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2020-11-16T21:53:59.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:53:59.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:53:59.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:53:59.194 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:53:59.194 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:53:59.194 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:53:59.194 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:53:59.194 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:53:59.195 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:53:59.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:53:59.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:53:59.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:53:59.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:53:59.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:53:59.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:53:59.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:53:59.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:53:59.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:53:59.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:53:59.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:53:59.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:53:59.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2020-11-16T21:53:59.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:53:59.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=221bf916-5b41-44cd-8fa8-5d85f6fd788d 2020-11-16T21:53:59.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 221bf916-5b41-44cd-8fa8-5d85f6fd788d' 2020-11-16T21:53:59.198 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 221bf916-5b41-44cd-8fa8-5d85f6fd788d 2020-11-16T21:53:59.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:53:59.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD39LJfwgtoDRAAaS+FL1xMAuLZv5glVgaBxg== 2020-11-16T21:53:59.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD39LJfwgtoDRAAaS+FL1xMAuLZv5glVgaBxg=="}' 2020-11-16T21:53:59.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 221bf916-5b41-44cd-8fa8-5d85f6fd788d -i td/osd-backfill-stats/0/new.json 2020-11-16T21:53:59.589 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:53:59.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2020-11-16T21:53:59.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD39LJfwgtoDRAAaS+FL1xMAuLZv5glVgaBxg== --osd-uuid 221bf916-5b41-44cd-8fa8-5d85f6fd788d 2020-11-16T21:53:59.631 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:53:59.636+0000 7f24860f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:53:59.633 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:53:59.640+0000 7f24860f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:53:59.634 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:53:59.640+0000 7f24860f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:53:59.635 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:53:59.640+0000 7f24860f3d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:53:59.635 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:53:59.640+0000 7f24860f3d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:53:59.635 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:53:59.640+0000 7f24860f3d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:53:59.696 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:53:59.700+0000 7f24860f3d80 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2020-11-16T21:54:03.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2020-11-16T21:54:03.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:54:03.986 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:54:03.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:54:03.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:54:04.486 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:54:04.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:54:04.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:54:04.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:54:04.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:54:04.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:54:04.515 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:04.520+0000 7fcc204c9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:04.517 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:04.524+0000 7fcc204c9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:04.519 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:04.524+0000 7fcc204c9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:04.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:54:04.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:54:04.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:54:04.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:54:04.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:54:04.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:04.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:54:04.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:04.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:54:04.856 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:54:05.059 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:05.064+0000 7fcc204c9d80 -1 Falling back to public interface 2020-11-16T21:54:05.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:54:05.900 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:05.896+0000 7fcc204c9d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:54:06.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:54:06.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:06.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:54:06.243 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:54:06.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:06.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:54:06.646 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/545870,v1:127.0.0.1:6803/545870] [v2:127.0.0.1:6804/545870,v1:127.0.0.1:6805/545870] exists,up 221bf916-5b41-44cd-8fa8-5d85f6fd788d 2020-11-16T21:54:06.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:54:06.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:54:06.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:54:06.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:334: TEST_backfill_out2: run_osd td/osd-backfill-stats 1 2020-11-16T21:54:06.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:54:06.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:54:06.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:54:06.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:54:06.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2020-11-16T21:54:06.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:54:06.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:54:06.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:54:06.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:54:06.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2020-11-16T21:54:06.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2020-11-16T21:54:06.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:54:06.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:54:06.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:54:06.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:54:06.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:54:06.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:54:06.651 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:54:06.651 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:54:06.651 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:54:06.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:54:06.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:54:06.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:54:06.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:54:06.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:54:06.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:54:06.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:54:06.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:54:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:54:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:54:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:54:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:54:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2020-11-16T21:54:06.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:54:06.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2f502eb2-9f76-46fb-acd3-bfe593d77532 2020-11-16T21:54:06.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 2f502eb2-9f76-46fb-acd3-bfe593d77532' 2020-11-16T21:54:06.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:54:06.654 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 2f502eb2-9f76-46fb-acd3-bfe593d77532 2020-11-16T21:54:06.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD+9LJfNQjvKBAAuBBV57Q97TwYuBuMGv4r2g== 2020-11-16T21:54:06.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD+9LJfNQjvKBAAuBBV57Q97TwYuBuMGv4r2g=="}' 2020-11-16T21:54:06.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2f502eb2-9f76-46fb-acd3-bfe593d77532 -i td/osd-backfill-stats/1/new.json 2020-11-16T21:54:07.216 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:54:07.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2020-11-16T21:54:07.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD+9LJfNQjvKBAAuBBV57Q97TwYuBuMGv4r2g== --osd-uuid 2f502eb2-9f76-46fb-acd3-bfe593d77532 2020-11-16T21:54:07.260 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:07.264+0000 7fc3b6f42d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:07.263 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:07.268+0000 7fc3b6f42d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:07.264 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:07.268+0000 7fc3b6f42d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:07.265 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:07.272+0000 7fc3b6f42d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:54:07.265 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:07.272+0000 7fc3b6f42d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:54:07.265 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:07.272+0000 7fc3b6f42d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:54:07.323 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:07.328+0000 7fc3b6f42d80 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2020-11-16T21:54:11.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2020-11-16T21:54:11.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:54:11.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:54:11.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:54:11.401 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:54:11.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:54:11.903 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:54:11.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:54:11.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:54:11.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:54:11.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:54:11.933 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:11.940+0000 7f2f9fbb5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:11.936 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:11.940+0000 7f2f9fbb5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:11.937 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:11.944+0000 7f2f9fbb5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:12.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:54:12.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:54:12.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:54:12.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:54:12.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:54:12.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:12.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:54:12.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:12.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:54:12.274 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:54:12.479 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:12.484+0000 7f2f9fbb5d80 -1 Falling back to public interface 2020-11-16T21:54:12.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:54:13.402 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:13.400+0000 7f2f9fbb5d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:54:13.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:54:13.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:13.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:54:13.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:13.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:54:13.665 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:54:14.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:54:15.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:54:15.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:15.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T21:54:15.071 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:54:15.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:15.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:54:15.471 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/546864,v1:127.0.0.1:6811/546864] [v2:127.0.0.1:6812/546864,v1:127.0.0.1:6813/546864] exists,up 2f502eb2-9f76-46fb-acd3-bfe593d77532 2020-11-16T21:54:15.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:54:15.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:54:15.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:54:15.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:335: TEST_backfill_out2: run_osd td/osd-backfill-stats 2 2020-11-16T21:54:15.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:54:15.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:54:15.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:54:15.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:54:15.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2020-11-16T21:54:15.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:54:15.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:54:15.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:54:15.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:54:15.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2020-11-16T21:54:15.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2020-11-16T21:54:15.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:54:15.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:54:15.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:54:15.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:54:15.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:54:15.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:54:15.476 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:54:15.476 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:54:15.476 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:54:15.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:54:15.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:54:15.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:54:15.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:54:15.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:54:15.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:54:15.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:54:15.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:54:15.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:54:15.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:54:15.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:54:15.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:54:15.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2020-11-16T21:54:15.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:54:15.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=cbd844aa-2475-4306-80ad-298a3dcfc509 2020-11-16T21:54:15.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 cbd844aa-2475-4306-80ad-298a3dcfc509' 2020-11-16T21:54:15.479 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 cbd844aa-2475-4306-80ad-298a3dcfc509 2020-11-16T21:54:15.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:54:15.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAH9bJfF7paHhAA2P6855apKCNmBv81I3CUbQ== 2020-11-16T21:54:15.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAH9bJfF7paHhAA2P6855apKCNmBv81I3CUbQ=="}' 2020-11-16T21:54:15.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new cbd844aa-2475-4306-80ad-298a3dcfc509 -i td/osd-backfill-stats/2/new.json 2020-11-16T21:54:16.000 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:54:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2020-11-16T21:54:16.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAH9bJfF7paHhAA2P6855apKCNmBv81I3CUbQ== --osd-uuid cbd844aa-2475-4306-80ad-298a3dcfc509 2020-11-16T21:54:16.040 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:16.044+0000 7fb2e2358d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:16.042 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:16.048+0000 7fb2e2358d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:16.044 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:16.048+0000 7fb2e2358d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:16.052+0000 7fb2e2358d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:54:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:16.052+0000 7fb2e2358d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:54:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:16.052+0000 7fb2e2358d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:54:16.116 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:16.124+0000 7fb2e2358d80 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2020-11-16T21:54:20.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2020-11-16T21:54:20.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:54:20.424 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:54:20.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:54:20.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:54:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:54:20.933 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:54:20.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:54:20.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:54:20.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:54:20.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:54:20.963 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:20.968+0000 7f6e23563d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:20.966 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:20.972+0000 7f6e23563d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:20.968 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:20.972+0000 7f6e23563d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:21.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:54:21.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:54:21.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:54:21.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:54:21.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:54:21.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:21.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:54:21.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:54:21.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:21.343 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:54:21.507 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:21.512+0000 7f6e23563d80 -1 Falling back to public interface 2020-11-16T21:54:21.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:54:22.353 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:22.356+0000 7f6e23563d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:54:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:54:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:22.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:54:22.736 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:54:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:22.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:54:23.144 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/547884,v1:127.0.0.1:6819/547884] [v2:127.0.0.1:6820/547884,v1:127.0.0.1:6821/547884] exists,up cbd844aa-2475-4306-80ad-298a3dcfc509 2020-11-16T21:54:23.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:54:23.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:54:23.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:54:23.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:336: TEST_backfill_out2: run_osd td/osd-backfill-stats 3 2020-11-16T21:54:23.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:54:23.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:54:23.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:54:23.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:54:23.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2020-11-16T21:54:23.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:54:23.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:54:23.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:54:23.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:54:23.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2020-11-16T21:54:23.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2020-11-16T21:54:23.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:54:23.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:54:23.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:54:23.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:54:23.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:54:23.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:54:23.149 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:54:23.149 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:54:23.149 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:54:23.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:54:23.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:54:23.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:54:23.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:54:23.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:54:23.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:54:23.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:54:23.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:54:23.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:54:23.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:54:23.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:54:23.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:54:23.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2020-11-16T21:54:23.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:54:23.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d3cd3af7-327d-4d01-97b9-e9a322f6810a 2020-11-16T21:54:23.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 d3cd3af7-327d-4d01-97b9-e9a322f6810a' 2020-11-16T21:54:23.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:54:23.153 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 d3cd3af7-327d-4d01-97b9-e9a322f6810a 2020-11-16T21:54:23.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAP9bJfam3FChAA/bZYavT8+IywZWXrsmRV7Q== 2020-11-16T21:54:23.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAP9bJfam3FChAA/bZYavT8+IywZWXrsmRV7Q=="}' 2020-11-16T21:54:23.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d3cd3af7-327d-4d01-97b9-e9a322f6810a -i td/osd-backfill-stats/3/new.json 2020-11-16T21:54:23.704 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:54:23.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2020-11-16T21:54:23.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAP9bJfam3FChAA/bZYavT8+IywZWXrsmRV7Q== --osd-uuid d3cd3af7-327d-4d01-97b9-e9a322f6810a 2020-11-16T21:54:23.747 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:23.752+0000 7f3f3f9c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:23.750 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:23.756+0000 7f3f3f9c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:23.751 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:23.756+0000 7f3f3f9c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:23.756+0000 7f3f3f9c0d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:54:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:23.756+0000 7f3f3f9c0d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:54:23.752 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:23.756+0000 7f3f3f9c0d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:54:23.790 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:23.796+0000 7f3f3f9c0d80 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2020-11-16T21:54:28.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2020-11-16T21:54:28.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:54:28.362 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:54:28.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:54:28.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:54:28.865 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:54:28.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:54:28.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:54:28.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:54:28.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:54:28.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:54:28.893 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:28.896+0000 7f17e6e44d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:28.895 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:28.900+0000 7f17e6e44d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:28.896 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:28.900+0000 7f17e6e44d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:29.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:54:29.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:54:29.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:54:29.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:54:29.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:54:29.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:29.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:54:29.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:29.246 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:54:29.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:54:29.434 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:29.440+0000 7f17e6e44d80 -1 Falling back to public interface 2020-11-16T21:54:29.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:54:30.154 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:30.152+0000 7f17e6e44d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:54:30.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:54:30.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:30.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:54:30.659 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:54:30.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:30.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:54:31.088 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/548878,v1:127.0.0.1:6827/548878] [v2:127.0.0.1:6828/548878,v1:127.0.0.1:6829/548878] exists,up d3cd3af7-327d-4d01-97b9-e9a322f6810a 2020-11-16T21:54:31.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:54:31.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:54:31.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:54:31.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:337: TEST_backfill_out2: run_osd td/osd-backfill-stats 4 2020-11-16T21:54:31.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:54:31.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:54:31.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:54:31.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:54:31.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2020-11-16T21:54:31.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:54:31.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:54:31.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:54:31.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:54:31.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2020-11-16T21:54:31.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2020-11-16T21:54:31.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:54:31.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:54:31.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:54:31.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:54:31.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:54:31.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:54:31.092 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:54:31.093 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:54:31.093 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:54:31.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:54:31.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:54:31.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:54:31.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:54:31.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:54:31.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:54:31.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:54:31.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:54:31.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:54:31.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:54:31.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:54:31.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:54:31.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2020-11-16T21:54:31.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:54:31.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=89c9a0c8-4760-4935-aaf9-8a24ad88e767 2020-11-16T21:54:31.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 89c9a0c8-4760-4935-aaf9-8a24ad88e767' 2020-11-16T21:54:31.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:54:31.096 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 89c9a0c8-4760-4935-aaf9-8a24ad88e767 2020-11-16T21:54:31.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAX9bJfI/IQBxAAglnQjvji7mPhFk5gEb79sw== 2020-11-16T21:54:31.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAX9bJfI/IQBxAAglnQjvji7mPhFk5gEb79sw=="}' 2020-11-16T21:54:31.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 89c9a0c8-4760-4935-aaf9-8a24ad88e767 -i td/osd-backfill-stats/4/new.json 2020-11-16T21:54:31.619 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:54:31.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2020-11-16T21:54:31.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAX9bJfI/IQBxAAglnQjvji7mPhFk5gEb79sw== --osd-uuid 89c9a0c8-4760-4935-aaf9-8a24ad88e767 2020-11-16T21:54:31.664 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:31.668+0000 7f058268ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:31.667 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:31.672+0000 7f058268ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:31.668 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:31.672+0000 7f058268ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:31.668 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:31.672+0000 7f058268ad80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:54:31.668 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:31.672+0000 7f058268ad80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:54:31.668 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:31.672+0000 7f058268ad80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:54:31.720 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:31.724+0000 7f058268ad80 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2020-11-16T21:54:36.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2020-11-16T21:54:36.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:54:36.316 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:54:36.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:54:36.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:54:36.816 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:54:36.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:54:36.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:54:36.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:54:36.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:54:36.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:54:36.846 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:36.852+0000 7f50dcaf8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:36.848 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:36.852+0000 7f50dcaf8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:36.849 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:36.856+0000 7f50dcaf8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:37.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:54:37.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:54:37.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:54:37.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:54:37.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:54:37.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:37.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:54:37.198 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:54:37.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:37.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:54:37.395 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:37.400+0000 7f50dcaf8d80 -1 Falling back to public interface 2020-11-16T21:54:37.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:54:38.274 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:38.272+0000 7f50dcaf8d80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:54:38.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:54:38.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:38.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:54:38.621 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:54:38.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:38.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:54:39.056 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/549872,v1:127.0.0.1:6835/549872] [v2:127.0.0.1:6836/549872,v1:127.0.0.1:6837/549872] exists,up 89c9a0c8-4760-4935-aaf9-8a24ad88e767 2020-11-16T21:54:39.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:54:39.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:54:39.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:54:39.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:338: TEST_backfill_out2: run_osd td/osd-backfill-stats 5 2020-11-16T21:54:39.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:54:39.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:54:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T21:54:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:54:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2020-11-16T21:54:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:54:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:54:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:54:39.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:54:39.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2020-11-16T21:54:39.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2020-11-16T21:54:39.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:54:39.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:54:39.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:54:39.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:54:39.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:54:39.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:54:39.061 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:54:39.061 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:54:39.061 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:54:39.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:54:39.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:54:39.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:54:39.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:54:39.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:54:39.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:54:39.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:54:39.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:54:39.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:54:39.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:54:39.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:54:39.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:54:39.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2020-11-16T21:54:39.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:54:39.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=33d59bcc-9163-475e-a088-d4388aac3de8 2020-11-16T21:54:39.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 33d59bcc-9163-475e-a088-d4388aac3de8' 2020-11-16T21:54:39.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:54:39.065 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 33d59bcc-9163-475e-a088-d4388aac3de8 2020-11-16T21:54:39.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAf9bJfRaSSBRAA86QYUkvV9cEtZpztLo39Uw== 2020-11-16T21:54:39.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAf9bJfRaSSBRAA86QYUkvV9cEtZpztLo39Uw=="}' 2020-11-16T21:54:39.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 33d59bcc-9163-475e-a088-d4388aac3de8 -i td/osd-backfill-stats/5/new.json 2020-11-16T21:54:39.672 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T21:54:39.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2020-11-16T21:54:39.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAf9bJfRaSSBRAA86QYUkvV9cEtZpztLo39Uw== --osd-uuid 33d59bcc-9163-475e-a088-d4388aac3de8 2020-11-16T21:54:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:39.720+0000 7f4d17105d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:39.734 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:39.740+0000 7f4d17105d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:39.736 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:39.740+0000 7f4d17105d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:43.926 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:39.740+0000 7f4d17105d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:54:43.927 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:39.740+0000 7f4d17105d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:54:43.927 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:39.740+0000 7f4d17105d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:54:43.927 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:39.816+0000 7f4d17105d80 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2020-11-16T21:54:44.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2020-11-16T21:54:44.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:54:44.180 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T21:54:44.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T21:54:44.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:54:44.678 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T21:54:44.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T21:54:44.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:54:44.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:54:44.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:54:44.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:54:44.707 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:44.712+0000 7ff8882e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:44.709 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:44.716+0000 7ff8882e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:44.710 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:44.716+0000 7ff8882e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:54:45.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T21:54:45.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:54:45.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T21:54:45.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:54:45.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:54:45.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:45.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:54:45.058 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:54:45.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:45.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:54:45.251 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:45.256+0000 7ff8882e9d80 -1 Falling back to public interface 2020-11-16T21:54:45.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:54:46.134 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:54:46.140+0000 7ff8882e9d80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T21:54:46.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:54:46.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:54:46.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:54:46.522 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:54:46.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:54:46.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:54:46.905 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/550866,v1:127.0.0.1:6843/550866] [v2:127.0.0.1:6844/550866,v1:127.0.0.1:6845/550866] exists,up 33d59bcc-9163-475e-a088-d4388aac3de8 2020-11-16T21:54:46.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:54:46.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:54:46.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:54:46.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:340: TEST_backfill_out2: create_pool test 1 1 2020-11-16T21:54:46.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T21:54:47.523 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T21:54:47.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:54:48.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:341: TEST_backfill_out2: ceph osd pool set test size 2 2020-11-16T21:54:49.271 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T21:54:49.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:342: TEST_backfill_out2: sleep 5 2020-11-16T21:54:54.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:344: TEST_backfill_out2: wait_for_clean 2020-11-16T21:54:54.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:54:54.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:54:54.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:54:54.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:54:54.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:54:54.285 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:54:54.285 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:54:54.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:54:54.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:54:54.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:54:54.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:54:54.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:54:54.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:54:54.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:54:54.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:54:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:54:54.916 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:54:54.917 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:54:54.917 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:54:54.917 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:54:54.917 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:54:54.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:54:54.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:54:54.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:54:55.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836492 2020-11-16T21:54:55.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836492 2020-11-16T21:54:55.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492' 2020-11-16T21:54:55.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:54:55.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:54:55.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672971 2020-11-16T21:54:55.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672971 2020-11-16T21:54:55.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971' 2020-11-16T21:54:55.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:54:55.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:54:55.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509450 2020-11-16T21:54:55.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509450 2020-11-16T21:54:55.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509450' 2020-11-16T21:54:55.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:54:55.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:54:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345927 2020-11-16T21:54:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345927 2020-11-16T21:54:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509450 3-85899345927' 2020-11-16T21:54:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:54:55.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:54:55.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182407 2020-11-16T21:54:55.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182407 2020-11-16T21:54:55.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509450 3-85899345927 4-107374182407' 2020-11-16T21:54:55.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:54:55.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:54:55.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018884 2020-11-16T21:54:55.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018884 2020-11-16T21:54:55.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509450 3-85899345927 4-107374182407 5-128849018884' 2020-11-16T21:54:55.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:54:55.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836492 2020-11-16T21:54:55.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:54:55.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:54:55.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836492 2020-11-16T21:54:55.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:54:55.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836492 2020-11-16T21:54:55.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-11-16T21:54:55.924 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836492 2020-11-16T21:54:55.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:54:57.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836492 2020-11-16T21:54:57.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:54:58.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:54:58.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:54:58.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836492 2020-11-16T21:54:58.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:54:58.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672971 2020-11-16T21:54:58.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:54:58.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:54:58.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2020-11-16T21:54:58.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:54:58.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672971 2020-11-16T21:54:58.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-11-16T21:54:58.561 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672971 2020-11-16T21:54:58.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:54:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672971 2020-11-16T21:54:58.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:54:58.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509450 2020-11-16T21:54:58.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:54:58.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:54:58.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509450 2020-11-16T21:54:58.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:54:58.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509450 2020-11-16T21:54:58.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2020-11-16T21:54:58.965 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509450 2020-11-16T21:54:58.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:54:59.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509451 -lt 64424509450 2020-11-16T21:54:59.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:54:59.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345927 2020-11-16T21:54:59.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:54:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:54:59.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345927 2020-11-16T21:54:59.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:54:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345927 2020-11-16T21:54:59.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2020-11-16T21:54:59.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:54:59.382 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 85899345927 2020-11-16T21:54:59.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345928 -lt 85899345927 2020-11-16T21:54:59.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:54:59.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182407 2020-11-16T21:54:59.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:54:59.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:54:59.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182407 2020-11-16T21:54:59.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:54:59.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182407 2020-11-16T21:54:59.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182407' 2020-11-16T21:54:59.750 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 107374182407 2020-11-16T21:54:59.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:55:00.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182407 -lt 107374182407 2020-11-16T21:55:00.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:55:00.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:55:00.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018884 2020-11-16T21:55:00.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:55:00.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:55:00.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018884 2020-11-16T21:55:00.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018884 2020-11-16T21:55:00.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018884' 2020-11-16T21:55:00.138 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 128849018884 2020-11-16T21:55:00.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:55:00.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018885 -lt 128849018884 2020-11-16T21:55:00.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:55:00.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:00.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:01.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:55:01.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:01.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:01.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:01.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:01.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:01.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:01.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:55:01.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:01.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:01.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:01.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:55:01.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:55:01.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:55:01.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: seq 1 200 2020-11-16T21:55:01.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:01.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj1 /dev/null 2020-11-16T21:55:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj2 /dev/null 2020-11-16T21:55:02.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:02.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj3 /dev/null 2020-11-16T21:55:02.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:02.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj4 /dev/null 2020-11-16T21:55:02.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:02.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj5 /dev/null 2020-11-16T21:55:02.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:02.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj6 /dev/null 2020-11-16T21:55:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:02.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj7 /dev/null 2020-11-16T21:55:02.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:02.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj8 /dev/null 2020-11-16T21:55:02.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:02.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj9 /dev/null 2020-11-16T21:55:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:02.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj10 /dev/null 2020-11-16T21:55:02.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:02.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj11 /dev/null 2020-11-16T21:55:03.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:03.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj12 /dev/null 2020-11-16T21:55:03.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:03.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj13 /dev/null 2020-11-16T21:55:03.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:03.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj14 /dev/null 2020-11-16T21:55:03.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:03.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj15 /dev/null 2020-11-16T21:55:03.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:03.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj16 /dev/null 2020-11-16T21:55:03.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:03.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj17 /dev/null 2020-11-16T21:55:03.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:03.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj18 /dev/null 2020-11-16T21:55:03.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:03.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj19 /dev/null 2020-11-16T21:55:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj20 /dev/null 2020-11-16T21:55:04.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:04.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj21 /dev/null 2020-11-16T21:55:04.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:04.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj22 /dev/null 2020-11-16T21:55:04.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:04.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj23 /dev/null 2020-11-16T21:55:04.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:04.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj24 /dev/null 2020-11-16T21:55:04.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:04.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj25 /dev/null 2020-11-16T21:55:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:04.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj26 /dev/null 2020-11-16T21:55:04.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:04.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj27 /dev/null 2020-11-16T21:55:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj28 /dev/null 2020-11-16T21:55:04.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:04.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj29 /dev/null 2020-11-16T21:55:04.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:04.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj30 /dev/null 2020-11-16T21:55:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj31 /dev/null 2020-11-16T21:55:05.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:05.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj32 /dev/null 2020-11-16T21:55:05.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:05.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj33 /dev/null 2020-11-16T21:55:05.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:05.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj34 /dev/null 2020-11-16T21:55:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj35 /dev/null 2020-11-16T21:55:05.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:05.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj36 /dev/null 2020-11-16T21:55:05.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:05.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj37 /dev/null 2020-11-16T21:55:05.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:05.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj38 /dev/null 2020-11-16T21:55:05.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:05.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj39 /dev/null 2020-11-16T21:55:06.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:06.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj40 /dev/null 2020-11-16T21:55:06.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:06.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj41 /dev/null 2020-11-16T21:55:06.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:06.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj42 /dev/null 2020-11-16T21:55:06.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:06.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj43 /dev/null 2020-11-16T21:55:06.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:06.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj44 /dev/null 2020-11-16T21:55:06.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:06.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj45 /dev/null 2020-11-16T21:55:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:06.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj46 /dev/null 2020-11-16T21:55:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj47 /dev/null 2020-11-16T21:55:06.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:06.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj48 /dev/null 2020-11-16T21:55:07.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:07.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj49 /dev/null 2020-11-16T21:55:07.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:07.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj50 /dev/null 2020-11-16T21:55:07.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:07.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj51 /dev/null 2020-11-16T21:55:07.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:07.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj52 /dev/null 2020-11-16T21:55:07.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:07.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj53 /dev/null 2020-11-16T21:55:07.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:07.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj54 /dev/null 2020-11-16T21:55:07.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:07.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj55 /dev/null 2020-11-16T21:55:07.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:07.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj56 /dev/null 2020-11-16T21:55:07.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:07.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj57 /dev/null 2020-11-16T21:55:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj58 /dev/null 2020-11-16T21:55:08.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:08.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj59 /dev/null 2020-11-16T21:55:08.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:08.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj60 /dev/null 2020-11-16T21:55:08.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:08.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj61 /dev/null 2020-11-16T21:55:08.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:08.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj62 /dev/null 2020-11-16T21:55:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj63 /dev/null 2020-11-16T21:55:08.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:08.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj64 /dev/null 2020-11-16T21:55:08.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:08.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj65 /dev/null 2020-11-16T21:55:08.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:08.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj66 /dev/null 2020-11-16T21:55:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:08.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj67 /dev/null 2020-11-16T21:55:09.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:09.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj68 /dev/null 2020-11-16T21:55:09.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:09.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj69 /dev/null 2020-11-16T21:55:09.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:09.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj70 /dev/null 2020-11-16T21:55:09.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:09.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj71 /dev/null 2020-11-16T21:55:09.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:09.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj72 /dev/null 2020-11-16T21:55:09.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:09.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj73 /dev/null 2020-11-16T21:55:09.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:09.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj74 /dev/null 2020-11-16T21:55:09.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:09.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj75 /dev/null 2020-11-16T21:55:09.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:09.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj76 /dev/null 2020-11-16T21:55:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj77 /dev/null 2020-11-16T21:55:10.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:10.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj78 /dev/null 2020-11-16T21:55:10.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:10.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj79 /dev/null 2020-11-16T21:55:10.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:10.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj80 /dev/null 2020-11-16T21:55:10.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:10.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj81 /dev/null 2020-11-16T21:55:10.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:10.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj82 /dev/null 2020-11-16T21:55:10.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:10.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj83 /dev/null 2020-11-16T21:55:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj84 /dev/null 2020-11-16T21:55:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj85 /dev/null 2020-11-16T21:55:10.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:10.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj86 /dev/null 2020-11-16T21:55:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj87 /dev/null 2020-11-16T21:55:11.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:11.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj88 /dev/null 2020-11-16T21:55:11.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:11.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj89 /dev/null 2020-11-16T21:55:11.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:11.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj90 /dev/null 2020-11-16T21:55:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj91 /dev/null 2020-11-16T21:55:11.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:11.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj92 /dev/null 2020-11-16T21:55:11.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:11.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj93 /dev/null 2020-11-16T21:55:11.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:11.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj94 /dev/null 2020-11-16T21:55:11.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:11.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj95 /dev/null 2020-11-16T21:55:12.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:12.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj96 /dev/null 2020-11-16T21:55:12.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:12.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj97 /dev/null 2020-11-16T21:55:12.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:12.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj98 /dev/null 2020-11-16T21:55:12.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:12.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj99 /dev/null 2020-11-16T21:55:12.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:12.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj100 /dev/null 2020-11-16T21:55:12.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:12.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj101 /dev/null 2020-11-16T21:55:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj102 /dev/null 2020-11-16T21:55:12.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:12.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj103 /dev/null 2020-11-16T21:55:12.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:12.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj104 /dev/null 2020-11-16T21:55:12.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:12.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj105 /dev/null 2020-11-16T21:55:13.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:13.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj106 /dev/null 2020-11-16T21:55:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj107 /dev/null 2020-11-16T21:55:13.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:13.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj108 /dev/null 2020-11-16T21:55:13.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:13.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj109 /dev/null 2020-11-16T21:55:13.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:13.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj110 /dev/null 2020-11-16T21:55:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:13.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj111 /dev/null 2020-11-16T21:55:13.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:13.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj112 /dev/null 2020-11-16T21:55:13.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:13.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj113 /dev/null 2020-11-16T21:55:13.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:13.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj114 /dev/null 2020-11-16T21:55:14.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:14.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj115 /dev/null 2020-11-16T21:55:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj116 /dev/null 2020-11-16T21:55:14.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:14.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj117 /dev/null 2020-11-16T21:55:14.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:14.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj118 /dev/null 2020-11-16T21:55:14.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:14.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj119 /dev/null 2020-11-16T21:55:14.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:14.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj120 /dev/null 2020-11-16T21:55:14.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:14.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj121 /dev/null 2020-11-16T21:55:14.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:14.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj122 /dev/null 2020-11-16T21:55:14.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:14.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj123 /dev/null 2020-11-16T21:55:15.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:15.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj124 /dev/null 2020-11-16T21:55:15.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:15.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj125 /dev/null 2020-11-16T21:55:15.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:15.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj126 /dev/null 2020-11-16T21:55:15.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:15.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj127 /dev/null 2020-11-16T21:55:15.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:15.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj128 /dev/null 2020-11-16T21:55:15.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:15.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj129 /dev/null 2020-11-16T21:55:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:15.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj130 /dev/null 2020-11-16T21:55:15.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:15.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj131 /dev/null 2020-11-16T21:55:15.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:15.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj132 /dev/null 2020-11-16T21:55:15.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:15.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj133 /dev/null 2020-11-16T21:55:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj134 /dev/null 2020-11-16T21:55:16.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:16.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj135 /dev/null 2020-11-16T21:55:16.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:16.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj136 /dev/null 2020-11-16T21:55:16.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:16.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj137 /dev/null 2020-11-16T21:55:16.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:16.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj138 /dev/null 2020-11-16T21:55:16.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:16.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj139 /dev/null 2020-11-16T21:55:16.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:16.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj140 /dev/null 2020-11-16T21:55:16.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:16.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj141 /dev/null 2020-11-16T21:55:16.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:16.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj142 /dev/null 2020-11-16T21:55:17.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:17.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj143 /dev/null 2020-11-16T21:55:17.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:17.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj144 /dev/null 2020-11-16T21:55:17.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:17.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj145 /dev/null 2020-11-16T21:55:17.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:17.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj146 /dev/null 2020-11-16T21:55:17.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:17.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj147 /dev/null 2020-11-16T21:55:17.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:17.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj148 /dev/null 2020-11-16T21:55:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj149 /dev/null 2020-11-16T21:55:17.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:17.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj150 /dev/null 2020-11-16T21:55:17.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:17.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj151 /dev/null 2020-11-16T21:55:17.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:17.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj152 /dev/null 2020-11-16T21:55:18.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:18.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj153 /dev/null 2020-11-16T21:55:18.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:18.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj154 /dev/null 2020-11-16T21:55:18.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:18.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj155 /dev/null 2020-11-16T21:55:18.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:18.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj156 /dev/null 2020-11-16T21:55:18.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:18.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj157 /dev/null 2020-11-16T21:55:18.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:18.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj158 /dev/null 2020-11-16T21:55:18.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:18.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj159 /dev/null 2020-11-16T21:55:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj160 /dev/null 2020-11-16T21:55:18.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:18.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj161 /dev/null 2020-11-16T21:55:18.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:18.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj162 /dev/null 2020-11-16T21:55:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj163 /dev/null 2020-11-16T21:55:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj164 /dev/null 2020-11-16T21:55:19.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:19.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj165 /dev/null 2020-11-16T21:55:19.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:19.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj166 /dev/null 2020-11-16T21:55:19.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:19.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj167 /dev/null 2020-11-16T21:55:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj168 /dev/null 2020-11-16T21:55:19.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:19.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj169 /dev/null 2020-11-16T21:55:19.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:19.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj170 /dev/null 2020-11-16T21:55:19.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:19.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj171 /dev/null 2020-11-16T21:55:20.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:20.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj172 /dev/null 2020-11-16T21:55:20.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:20.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj173 /dev/null 2020-11-16T21:55:20.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:20.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj174 /dev/null 2020-11-16T21:55:20.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:20.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj175 /dev/null 2020-11-16T21:55:20.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:20.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj176 /dev/null 2020-11-16T21:55:20.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:20.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj177 /dev/null 2020-11-16T21:55:20.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:20.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj178 /dev/null 2020-11-16T21:55:20.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:20.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj179 /dev/null 2020-11-16T21:55:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj180 /dev/null 2020-11-16T21:55:21.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:21.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj181 /dev/null 2020-11-16T21:55:21.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:21.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj182 /dev/null 2020-11-16T21:55:21.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:21.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj183 /dev/null 2020-11-16T21:55:21.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:21.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj184 /dev/null 2020-11-16T21:55:21.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:21.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj185 /dev/null 2020-11-16T21:55:21.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:21.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj186 /dev/null 2020-11-16T21:55:21.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:21.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj187 /dev/null 2020-11-16T21:55:21.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:21.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj188 /dev/null 2020-11-16T21:55:21.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:21.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj189 /dev/null 2020-11-16T21:55:22.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:22.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj190 /dev/null 2020-11-16T21:55:22.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:22.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj191 /dev/null 2020-11-16T21:55:22.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:22.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj192 /dev/null 2020-11-16T21:55:22.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:22.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj193 /dev/null 2020-11-16T21:55:22.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:22.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj194 /dev/null 2020-11-16T21:55:22.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:22.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj195 /dev/null 2020-11-16T21:55:22.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:22.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj196 /dev/null 2020-11-16T21:55:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:22.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj197 /dev/null 2020-11-16T21:55:22.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:22.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj198 /dev/null 2020-11-16T21:55:22.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:22.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj199 /dev/null 2020-11-16T21:55:23.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2020-11-16T21:55:23.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj200 /dev/null 2020-11-16T21:55:23.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:351: TEST_backfill_out2: get_pg test obj1 2020-11-16T21:55:23.173 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T21:55:23.173 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T21:55:23.173 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T21:55:23.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T21:55:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:351: TEST_backfill_out2: local PG=1.0 2020-11-16T21:55:23.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:353: TEST_backfill_out2: get_primary test obj1 2020-11-16T21:55:23.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:55:23.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:55:23.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:55:23.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:55:23.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:353: TEST_backfill_out2: local primary=1 2020-11-16T21:55:23.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:354: TEST_backfill_out2: get_not_primary test obj1 2020-11-16T21:55:23.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2020-11-16T21:55:23.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2020-11-16T21:55:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2020-11-16T21:55:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:55:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:55:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:55:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:55:24.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T21:55:24.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2020-11-16T21:55:24.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T21:55:24.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:354: TEST_backfill_out2: local otherosd=0 2020-11-16T21:55:24.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:356: TEST_backfill_out2: ceph osd set nobackfill 2020-11-16T21:55:25.528 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T21:55:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:357: TEST_backfill_out2: ceph osd pool set test size 3 2020-11-16T21:55:26.373 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 3 2020-11-16T21:55:26.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:358: TEST_backfill_out2: ceph osd out osd.0 2020-11-16T21:55:27.245 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T21:55:27.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:359: TEST_backfill_out2: ceph osd out osd.1 2020-11-16T21:55:27.904 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. 2020-11-16T21:55:27.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:361: TEST_backfill_out2: sleep 2 2020-11-16T21:55:29.917 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:362: TEST_backfill_out2: get_primary test obj1 2020-11-16T21:55:29.917 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:55:29.917 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:55:29.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:55:29.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:55:30.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:362: TEST_backfill_out2: primary=1 2020-11-16T21:55:30.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:363: TEST_backfill_out2: ceph osd unset nobackfill 2020-11-16T21:55:30.955 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T21:55:30.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:364: TEST_backfill_out2: ceph tell osd.1 get_latest_osdmap 2020-11-16T21:55:31.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:365: TEST_backfill_out2: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T21:55:31.287 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T21:55:31.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:366: TEST_backfill_out2: sleep 2 2020-11-16T21:55:33.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:368: TEST_backfill_out2: wait_for_clean 2020-11-16T21:55:33.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:55:33.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:55:33.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:55:33.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:55:33.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:55:33.301 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:55:33.301 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:55:33.301 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:55:33.301 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:55:33.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:55:33.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:55:33.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:55:33.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:55:33.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:55:33.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:55:33.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:55:33.907 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:55:33.907 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:55:33.907 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:55:33.907 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:55:33.907 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:55:33.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:55:33.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:55:33.908 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:55:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836502 2020-11-16T21:55:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836502 2020-11-16T21:55:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502' 2020-11-16T21:55:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:55:34.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:55:34.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672980 2020-11-16T21:55:34.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672980 2020-11-16T21:55:34.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672980' 2020-11-16T21:55:34.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:55:34.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:55:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509460 2020-11-16T21:55:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509460 2020-11-16T21:55:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-64424509460' 2020-11-16T21:55:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:55:34.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:55:34.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345937 2020-11-16T21:55:34.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345937 2020-11-16T21:55:34.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-64424509460 3-85899345937' 2020-11-16T21:55:34.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:55:34.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:55:34.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182417 2020-11-16T21:55:34.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182417 2020-11-16T21:55:34.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-64424509460 3-85899345937 4-107374182417' 2020-11-16T21:55:34.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:55:34.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:55:34.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018894 2020-11-16T21:55:34.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018894 2020-11-16T21:55:34.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-64424509460 3-85899345937 4-107374182417 5-128849018894' 2020-11-16T21:55:34.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:55:34.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836502 2020-11-16T21:55:34.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:55:34.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:55:34.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836502 2020-11-16T21:55:34.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:55:34.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836502 2020-11-16T21:55:34.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2020-11-16T21:55:34.905 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836502 2020-11-16T21:55:34.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:55:35.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836502 -lt 21474836502 2020-11-16T21:55:35.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:55:35.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672980 2020-11-16T21:55:35.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:55:35.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:55:35.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672980 2020-11-16T21:55:35.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:55:35.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672980 2020-11-16T21:55:35.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2020-11-16T21:55:35.308 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672980 2020-11-16T21:55:35.308 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:55:35.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672981 -lt 42949672980 2020-11-16T21:55:35.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:55:35.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509460 2020-11-16T21:55:35.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:55:35.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:55:35.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509460 2020-11-16T21:55:35.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:55:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509460 2020-11-16T21:55:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2020-11-16T21:55:35.715 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509460 2020-11-16T21:55:35.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:55:36.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509460 -lt 64424509460 2020-11-16T21:55:36.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:55:36.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345937 2020-11-16T21:55:36.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:55:36.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:55:36.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345937 2020-11-16T21:55:36.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:55:36.134 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 85899345937 2020-11-16T21:55:36.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345937 2020-11-16T21:55:36.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2020-11-16T21:55:36.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:55:36.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345937 -lt 85899345937 2020-11-16T21:55:36.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:55:36.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182417 2020-11-16T21:55:36.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:55:36.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:55:36.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182417 2020-11-16T21:55:36.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:55:36.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182417 2020-11-16T21:55:36.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2020-11-16T21:55:36.507 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 107374182417 2020-11-16T21:55:36.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:55:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182417 -lt 107374182417 2020-11-16T21:55:36.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:55:36.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018894 2020-11-16T21:55:36.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:55:36.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:55:36.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018894 2020-11-16T21:55:36.899 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:55:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018894 2020-11-16T21:55:36.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018894' 2020-11-16T21:55:36.901 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 128849018894 2020-11-16T21:55:36.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:55:37.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018894 -lt 128849018894 2020-11-16T21:55:37.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:55:37.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:37.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:37.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:55:37.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:37.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:37.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:37.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:37.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:37.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:37.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:38.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:38.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:38.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:38.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:38.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:38.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T21:55:38.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T21:55:38.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T21:55:38.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:38.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:38.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:38.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:38.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:38.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:38.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:38.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:38.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:38.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:39.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:39.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:39.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:39.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:39.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:39.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:39.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:39.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:39.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:39.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:39.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:39.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:39.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:40.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=1 2020-11-16T21:55:40.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 1 '!=' null 2020-11-16T21:55:40.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:40.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:40.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:40.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:40.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:40.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:40.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:40.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:40.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:40.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:40.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:40.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:40.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:40.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:40.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:41.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:41.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:41.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:41.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:41.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:41.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:41.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:41.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:41.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:41.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:55:41.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:55:41.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:41.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:41.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:41.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:41.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:41.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:41.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:41.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:41.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:41.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:41.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:42.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:42.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:42.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:42.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:42.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:42.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:42.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:42.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:42.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:42.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:42.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:42.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:42.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:43.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:55:43.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:55:43.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:43.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:43.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:43.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:43.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:43.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:43.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:43.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:43.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:43.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:43.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:43.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:43.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:43.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:43.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:43.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:43.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:43.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:43.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:43.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:43.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:43.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:43.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:44.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:55:44.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:55:44.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:44.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:44.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:44.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:44.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:44.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:44.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:44.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:44.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:44.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:44.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:44.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:44.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:44.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:44.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:45.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:45.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:45.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:45.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:45.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:45.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:45.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:45.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:45.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:55:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:55:45.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:45.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:46.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:46.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:46.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:46.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:46.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:46.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:46.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:46.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:46.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:46.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:46.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:46.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:46.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:46.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:46.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:46.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:46.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:46.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:46.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:46.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:46.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:47.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:55:47.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:55:47.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:47.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:47.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:47.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:47.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:47.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:47.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:47.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:47.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:47.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:47.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:47.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:47.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:47.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:47.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:48.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:48.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:48.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:48.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:48.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:48.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:48.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:48.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:48.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:48.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:55:48.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:55:48.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:48.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:48.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:49.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:49.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:49.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:49.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:49.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:49.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:49.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:49.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:49.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:49.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:49.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:49.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:49.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:49.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:49.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:49.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:49.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:49.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:49.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:49.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:49.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:50.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:55:50.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:55:50.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:50.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:50.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:50.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:50.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:50.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:50.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:50.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:50.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:50.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:50.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:50.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:50.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:50.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:50.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:51.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:51.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:51.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:51.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:51.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:51.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:51.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:51.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:51.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:51.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:55:51.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:55:51.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:51.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:51.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:51.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:51.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:51.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:51.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:51.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:51.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:51.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:51.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:52.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:52.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:52.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:52.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:52.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:52.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:52.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:52.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:52.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:52.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:52.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:52.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:52.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:53.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:55:53.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:55:53.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:53.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:53.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:53.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:53.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:53.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:53.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:53.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:53.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:53.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:53.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:53.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:53.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:53.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:53.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:54.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:54.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:54.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:54.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:54.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:54.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:54.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:54.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:54.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:54.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:55:54.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:55:54.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:54.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:54.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:54.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:54.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:54.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:54.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:54.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:54.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:54.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:54.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:55.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:55.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:55.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:55.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:55.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:55.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:55.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:56.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:55:56.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:55:56.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:56.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:56.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:56.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:56.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:56.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:56.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:56.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:56.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:56.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:56.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:56.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:56.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:56.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:56.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:57.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:57.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:55:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:55:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:57.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:57.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:57.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:57.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:57.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:57.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:57.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:57.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:57.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:57.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:57.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:57.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:55:58.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:55:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:55:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:55:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:55:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:55:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:55:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:55:58.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:55:58.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:55:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:55:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:55:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:55:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:55:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:55:59.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:55:59.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:55:59.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:55:59.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:55:59.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:55:59.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:55:59.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:55:59.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:55:59.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:55:59.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:55:59.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:55:59.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:00.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:00.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:00.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:00.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:00.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:00.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:00.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:00.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:00.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:00.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:56:00.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:56:00.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:00.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:00.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:00.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:00.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:00.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:00.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:00.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:00.599 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:00.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:00.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:00.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:00.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:00.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:00.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:01.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:01.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:01.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:56:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:56:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:02.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:02.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:02.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:02.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:02.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:02.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:02.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:02.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:02.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:02.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:02.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:02.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:02.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:02.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:02.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:02.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:02.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:02.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:02.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:02.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:02.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:03.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:56:03.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:56:03.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:03.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:03.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:03.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:03.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:03.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:03.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:03.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:03.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:03.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:03.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:03.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:03.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:04.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:04.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:04.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:04.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:04.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:04.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:04.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:04.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:04.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:04.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:56:04.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:56:04.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:04.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:04.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:04.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:04.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:04.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:04.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:04.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:04.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:04.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:05.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:05.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:05.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:05.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:05.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:05.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:05.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:05.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:05.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:05.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:05.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:05.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:05.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:06.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:56:06.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:56:06.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:06.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:06.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:06.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:06.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:06.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:06.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:06.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:06.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:06.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:06.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:06.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:06.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:06.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:07.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:07.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:07.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:07.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:07.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:07.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:07.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:07.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:07.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:07.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:56:07.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:56:07.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:07.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:07.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:07.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:07.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:07.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:07.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:07.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:07.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:07.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:07.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:08.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:08.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:08.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:08.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:08.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:08.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:08.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:08.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:08.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:08.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:08.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:08.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:08.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:09.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:56:09.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:56:09.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:09.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:09.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:09.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:09.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:09.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:09.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:09.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:09.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:09.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:09.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:09.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:09.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:09.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:09.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:10.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:10.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:10.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:10.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:10.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:10.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:10.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:10.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:10.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:10.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:56:10.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:56:10.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:10.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:10.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:10.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:10.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:10.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:10.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:10.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:10.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:10.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:10.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:11.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:11.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:11.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:11.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:11.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:11.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:11.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:11.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:11.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:11.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:11.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:11.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:11.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:56:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:56:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:12.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:12.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:12.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:12.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:12.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:12.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:12.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:12.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:12.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:12.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:12.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:12.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:13.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:13.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:13.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:13.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:13.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:13.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:13.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:13.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:13.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:56:13.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:56:13.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:13.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:13.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:13.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:13.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:13.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:13.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:13.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:13.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:13.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:13.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:14.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:14.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:14.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:14.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:14.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:14.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:14.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:14.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:14.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:14.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:14.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:14.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:14.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:14.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:56:14.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:56:14.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:14.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:14.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:15.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:15.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:15.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:15.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:15.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:15.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:15.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:15.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:15.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:15.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:15.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:15.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:15.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:15.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:15.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:15.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:15.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:15.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:15.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:15.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:16.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T21:56:16.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T21:56:16.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:16.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:16.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:16.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:16.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:16.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:16.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:16.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:16.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:16.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:16.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:16.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:16.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:16.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:16.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:17.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:17.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:17.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T21:56:17.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T21:56:17.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:56:17.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:17.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:56:17.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:17.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:17.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:17.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:17.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:17.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:17.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:17.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:18.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:56:18.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:18.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:18.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:56:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:56:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:56:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:56:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:56:18.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:56:18.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:56:18.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:56:18.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:56:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T21:56:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T21:56:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2020-11-16T21:56:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:56:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-11-16T21:56:19.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:56:19.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:56:19.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:56:19.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:56:19.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:56:19.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:56:19.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:56:19.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:56:19.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:56:19.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:56:19.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:56:19.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:56:20.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:56:20.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:56:20.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:56:20.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:370: TEST_backfill_out2: expr 200 '*' 2 2020-11-16T21:56:20.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:370: TEST_backfill_out2: local misplaced=400 2020-11-16T21:56:20.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:372: TEST_backfill_out2: check td/osd-backfill-stats 1.0 1 replicated 200 0 400 0 2020-11-16T21:56:20.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2020-11-16T21:56:20.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2020-11-16T21:56:20.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2020-11-16T21:56:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2020-11-16T21:56:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=200 2020-11-16T21:56:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2020-11-16T21:56:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=400 2020-11-16T21:56:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2020-11-16T21:56:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2020-11-16T21:56:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2020-11-16T21:56:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2020-11-16T21:56:20.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2020-11-16T21:56:20.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2020-11-16T21:56:20.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2020-11-16T21:56:20.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2020-11-16T21:56:20.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2020-11-16T21:56:20.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2020-11-16T21:56:20.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2020-11-16T21:56:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2020-11-16T21:56:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2020-11-16T21:56:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2020-11-16T21:56:20.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2020-11-16T21:56:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:56:20.339 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2020-11-16T21:56:20.339 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2020-11-16T21:56:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[2,4,3]/[1,0]' 2020-11-16T21:56:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[2,4,3]/[1,0]' ']' 2020-11-16T21:56:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2020-11-16T21:56:20.364 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:56:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:56:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [2,4,3]/[1,0] ' 2020-11-16T21:56:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2020-11-16T21:56:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2020-11-16T21:56:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:56:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=200 2020-11-16T21:56:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 200 ']' 2020-11-16T21:56:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 200 200 2020-11-16T21:56:20.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2020-11-16T21:56:20.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:56:20.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2020-11-16T21:56:20.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:56:20.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:56:20.376 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:56:20.377 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2020-11-16T21:56:20.377 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:56:20.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=1 2020-11-16T21:56:20.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 1 ']' 2020-11-16T21:56:20.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 1 0 2020-11-16T21:56:20.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2020-11-16T21:56:20.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:56:20.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:56:20.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:56:20.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2020-11-16T21:56:20.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:56:20.392 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:56:20.393 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [2,4,3]/[1,0] ' 2020-11-16T21:56:20.393 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2020-11-16T21:56:20.393 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2020-11-16T21:56:20.393 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:56:20.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=400 2020-11-16T21:56:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 400 ']' 2020-11-16T21:56:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 400 400 2020-11-16T21:56:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2020-11-16T21:56:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:56:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2020-11-16T21:56:20.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:56:20.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:56:20.402 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T21:56:20.402 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2020-11-16T21:56:20.402 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:56:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=2 2020-11-16T21:56:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 2 ']' 2020-11-16T21:56:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 2 0 2020-11-16T21:56:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=2 2020-11-16T21:56:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:56:20.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:56:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:56:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2020-11-16T21:56:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:374: TEST_backfill_out2: delete_pool test 2020-11-16T21:56:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T21:56:20.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T21:56:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T21:56:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:375: TEST_backfill_out2: kill_daemons td/osd-backfill-stats 2020-11-16T21:56:20.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:56:20.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:56:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:56:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:56:20.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:56:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:56:21.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2020-11-16T21:56:21.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:56:21.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:56:21.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:56:21.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:56:21.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:56:21.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:56:21.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:56:21.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:56:21.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:56:21.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:56:21.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:56:21.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:56:21.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:56:21.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:56:21.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:56:21.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:56:21.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:56:21.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:56:21.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:56:21.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:56:21.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:56:21.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:56:21.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:56:21.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:56:21.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:56:21.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:56:21.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:56:21.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:56:21.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2020-11-16T21:56:21.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2020-11-16T21:56:21.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2020-11-16T21:56:21.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2020-11-16T21:56:21.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:56:21.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:56:21.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:56:21.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:56:21.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:56:21.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:56:21.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:56:21.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:56:21.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:56:21.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:56:21.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:56:21.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:56:21.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:56:21.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:56:21.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:56:21.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:56:21.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:56:21.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:56:21.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:56:21.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:56:21.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:56:21.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:56:21.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:56:21.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:56:21.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:56:21.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:56:21.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:56:21.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:56:21.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2020-11-16T21:56:21.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:56:21.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:56:21.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:56:21.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.465868 2020-11-16T21:56:21.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:56:21.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:56:21.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:56:21.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2020-11-16T21:56:21.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_remapped td/osd-backfill-stats 2020-11-16T21:56:21.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:442: TEST_backfill_remapped: local dir=td/osd-backfill-stats 2020-11-16T21:56:21.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:444: TEST_backfill_remapped: run_mon td/osd-backfill-stats a 2020-11-16T21:56:21.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2020-11-16T21:56:21.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:56:21.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:56:21.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:56:21.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2020-11-16T21:56:21.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2020-11-16T21:56:21.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:56:21.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:56:21.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:56:21.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:56:21.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:56:21.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:56:21.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:56:21.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:56:21.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:56:21.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:56:21.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:56:21.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:56:21.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:56:21.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:56:21.612 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:56:21.612 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:56:21.612 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:56:21.613 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:56:21.613 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:56:21.613 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:56:21.613 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:56:21.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:56:21.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get fsid 2020-11-16T21:56:21.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:56:21.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:56:21.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:56:21.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:56:21.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:56:21.717 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:56:21.718 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:56:21.718 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:56:21.718 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:56:21.718 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:56:21.718 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:56:21.719 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:56:21.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:56:21.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get mon_host 2020-11-16T21:56:21.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:445: TEST_backfill_remapped: run_mgr td/osd-backfill-stats x 2020-11-16T21:56:21.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2020-11-16T21:56:21.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:56:21.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:56:21.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:56:21.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2020-11-16T21:56:21.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:56:22.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:56:22.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:56:22.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:56:22.351 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:56:22.352 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:56:22.352 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:56:22.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:56:22.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:56:22.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:56:22.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:446: TEST_backfill_remapped: run_osd td/osd-backfill-stats 0 2020-11-16T21:56:22.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:56:22.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:56:22.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:56:22.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:56:22.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2020-11-16T21:56:22.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:56:22.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:56:22.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:56:22.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:56:22.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2020-11-16T21:56:22.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2020-11-16T21:56:22.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:56:22.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:56:22.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:56:22.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:56:22.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:56:22.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:56:22.392 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:56:22.392 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:56:22.392 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:56:22.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:56:22.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:56:22.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:56:22.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:56:22.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:56:22.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:56:22.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:56:22.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:56:22.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:56:22.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:56:22.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:56:22.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:56:22.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2020-11-16T21:56:22.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:56:22.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=be532e82-d731-4218-b9ef-1e7cc356a03f 2020-11-16T21:56:22.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 be532e82-d731-4218-b9ef-1e7cc356a03f' 2020-11-16T21:56:22.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:56:22.396 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 be532e82-d731-4218-b9ef-1e7cc356a03f 2020-11-16T21:56:22.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCG9bJf77kvGRAANUOSdTd+yZ2Lbi8Gpr4Lvw== 2020-11-16T21:56:22.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCG9bJf77kvGRAANUOSdTd+yZ2Lbi8Gpr4Lvw=="}' 2020-11-16T21:56:22.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new be532e82-d731-4218-b9ef-1e7cc356a03f -i td/osd-backfill-stats/0/new.json 2020-11-16T21:56:22.757 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:56:22.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2020-11-16T21:56:22.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCG9bJf77kvGRAANUOSdTd+yZ2Lbi8Gpr4Lvw== --osd-uuid be532e82-d731-4218-b9ef-1e7cc356a03f 2020-11-16T21:56:22.801 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:22.806+0000 7fcaf3f92d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:22.804 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:22.810+0000 7fcaf3f92d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:22.805 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:22.814+0000 7fcaf3f92d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:22.806 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:22.814+0000 7fcaf3f92d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:56:22.806 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:22.814+0000 7fcaf3f92d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:56:22.807 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:22.814+0000 7fcaf3f92d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:56:22.843 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:22.850+0000 7fcaf3f92d80 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2020-11-16T21:56:27.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2020-11-16T21:56:27.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:56:27.152 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:56:27.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:56:27.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:56:27.644 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:56:27.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:56:27.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:56:27.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:56:27.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:56:27.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:56:27.673 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:27.682+0000 7f7cb4810d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:27.676 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:27.682+0000 7f7cb4810d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:27.678 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:27.686+0000 7f7cb4810d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:28.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:56:28.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:56:28.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:56:28.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:56:28.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:56:28.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:56:28.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:56:28.041 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:56:28.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:56:28.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:56:28.211 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:28.218+0000 7f7cb4810d80 -1 Falling back to public interface 2020-11-16T21:56:28.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:56:29.047 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:29.050+0000 7f7cb4810d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:56:29.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:56:29.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:56:29.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:56:29.405 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:56:29.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:56:29.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:56:29.791 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/559938,v1:127.0.0.1:6803/559938] [v2:127.0.0.1:6804/559938,v1:127.0.0.1:6805/559938] exists,up be532e82-d731-4218-b9ef-1e7cc356a03f 2020-11-16T21:56:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:56:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:56:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:56:29.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:447: TEST_backfill_remapped: run_osd td/osd-backfill-stats 1 2020-11-16T21:56:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:56:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:56:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:56:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:56:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2020-11-16T21:56:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:56:29.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:56:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:56:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:56:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2020-11-16T21:56:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2020-11-16T21:56:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:56:29.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:56:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:56:29.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:56:29.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:56:29.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:56:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:56:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:56:29.796 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:56:29.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:56:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:56:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:56:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:56:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:56:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:56:29.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:56:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:56:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:56:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:56:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:56:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:56:29.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2020-11-16T21:56:29.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:56:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e817093b-e287-42aa-ad1d-ffaadd8800eb 2020-11-16T21:56:29.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 e817093b-e287-42aa-ad1d-ffaadd8800eb' 2020-11-16T21:56:29.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:56:29.800 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 e817093b-e287-42aa-ad1d-ffaadd8800eb 2020-11-16T21:56:29.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCN9bJfJ2H9MBAAM5aV6m7e164WevMLdXeVWw== 2020-11-16T21:56:29.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCN9bJfJ2H9MBAAM5aV6m7e164WevMLdXeVWw=="}' 2020-11-16T21:56:29.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e817093b-e287-42aa-ad1d-ffaadd8800eb -i td/osd-backfill-stats/1/new.json 2020-11-16T21:56:30.371 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:56:30.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2020-11-16T21:56:30.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCN9bJfJ2H9MBAAM5aV6m7e164WevMLdXeVWw== --osd-uuid e817093b-e287-42aa-ad1d-ffaadd8800eb 2020-11-16T21:56:30.415 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:30.421+0000 7fcca4f8dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:30.417 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:30.425+0000 7fcca4f8dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:30.418 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:30.425+0000 7fcca4f8dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:30.420 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:30.425+0000 7fcca4f8dd80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:56:34.602 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:30.425+0000 7fcca4f8dd80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:56:34.602 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:30.425+0000 7fcca4f8dd80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:56:34.603 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:30.469+0000 7fcca4f8dd80 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2020-11-16T21:56:34.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2020-11-16T21:56:34.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:56:34.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:56:34.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:56:34.604 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:56:34.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:56:34.995 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:56:34.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:56:34.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:56:34.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:56:34.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:56:35.026 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:35.033+0000 7f1ce43add80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:35.028 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:35.033+0000 7f1ce43add80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:35.030 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:35.037+0000 7f1ce43add80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:56:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:56:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:56:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:56:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:56:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:56:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:56:35.378 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:56:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:56:35.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:56:35.574 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:35.581+0000 7f1ce43add80 -1 Falling back to public interface 2020-11-16T21:56:35.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:56:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:36.409+0000 7f1ce43add80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:56:36.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:56:36.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:56:36.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:56:36.765 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:56:36.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:56:36.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:56:37.171 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/560932,v1:127.0.0.1:6811/560932] [v2:127.0.0.1:6812/560932,v1:127.0.0.1:6813/560932] exists,up e817093b-e287-42aa-ad1d-ffaadd8800eb 2020-11-16T21:56:37.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:56:37.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:56:37.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:56:37.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:448: TEST_backfill_remapped: run_osd td/osd-backfill-stats 2 2020-11-16T21:56:37.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:56:37.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:56:37.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:56:37.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:56:37.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2020-11-16T21:56:37.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:56:37.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:56:37.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:56:37.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:56:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2020-11-16T21:56:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2020-11-16T21:56:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:56:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:56:37.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:56:37.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:56:37.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:56:37.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:56:37.177 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:56:37.177 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:56:37.178 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:56:37.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:56:37.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:56:37.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:56:37.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:56:37.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:56:37.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:56:37.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:56:37.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:56:37.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:56:37.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:56:37.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:56:37.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:56:37.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2020-11-16T21:56:37.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:56:37.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8c85d70c-05cc-4ba5-9f91-7ba0a9d17e58 2020-11-16T21:56:37.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 8c85d70c-05cc-4ba5-9f91-7ba0a9d17e58' 2020-11-16T21:56:37.180 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 8c85d70c-05cc-4ba5-9f91-7ba0a9d17e58 2020-11-16T21:56:37.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:56:37.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCV9bJf9chQDBAAVFwZZWPWF1WGzZIf4gtE4Q== 2020-11-16T21:56:37.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCV9bJf9chQDBAAVFwZZWPWF1WGzZIf4gtE4Q=="}' 2020-11-16T21:56:37.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8c85d70c-05cc-4ba5-9f91-7ba0a9d17e58 -i td/osd-backfill-stats/2/new.json 2020-11-16T21:56:37.722 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:56:37.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2020-11-16T21:56:37.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCV9bJf9chQDBAAVFwZZWPWF1WGzZIf4gtE4Q== --osd-uuid 8c85d70c-05cc-4ba5-9f91-7ba0a9d17e58 2020-11-16T21:56:37.764 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:37.769+0000 7f8682af9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:37.766 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:37.773+0000 7f8682af9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:37.768 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:37.773+0000 7f8682af9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:37.768 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:37.773+0000 7f8682af9d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:56:37.768 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:37.773+0000 7f8682af9d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:56:37.769 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:37.773+0000 7f8682af9d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:56:37.810 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:37.817+0000 7f8682af9d80 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2020-11-16T21:56:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2020-11-16T21:56:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:56:42.078 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:56:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:56:42.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:56:42.582 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:56:42.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:56:42.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:56:42.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:56:42.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:56:42.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:56:42.613 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:42.621+0000 7f6b3e77ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:42.621+0000 7f6b3e77ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:42.625+0000 7f6b3e77ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:42.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:56:42.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:56:42.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:56:42.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:56:42.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:56:42.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:56:42.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:56:42.969 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:56:42.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:56:42.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:56:43.163 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:43.169+0000 7f6b3e77ad80 -1 Falling back to public interface 2020-11-16T21:56:43.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:56:44.013 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:44.013+0000 7f6b3e77ad80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:56:44.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:56:44.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:56:44.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:56:44.365 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:56:44.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:56:44.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:56:44.755 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/561926,v1:127.0.0.1:6819/561926] [v2:127.0.0.1:6820/561926,v1:127.0.0.1:6821/561926] exists,up 8c85d70c-05cc-4ba5-9f91-7ba0a9d17e58 2020-11-16T21:56:44.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:56:44.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:56:44.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:56:44.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:449: TEST_backfill_remapped: run_osd td/osd-backfill-stats 3 2020-11-16T21:56:44.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:56:44.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:56:44.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:56:44.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:56:44.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2020-11-16T21:56:44.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:56:44.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:56:44.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:56:44.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:56:44.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2020-11-16T21:56:44.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2020-11-16T21:56:44.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:56:44.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:56:44.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:56:44.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:56:44.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:56:44.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:56:44.760 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:56:44.760 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:56:44.760 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:56:44.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:56:44.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:56:44.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:56:44.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:56:44.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:56:44.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:56:44.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:56:44.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:56:44.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:56:44.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:56:44.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:56:44.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:56:44.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2020-11-16T21:56:44.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:56:44.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9a8371b3-ebce-456f-a23e-2f5c54c33903 2020-11-16T21:56:44.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 9a8371b3-ebce-456f-a23e-2f5c54c33903' 2020-11-16T21:56:44.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:56:44.764 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 9a8371b3-ebce-456f-a23e-2f5c54c33903 2020-11-16T21:56:44.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCc9bJfHaUxLxAAAvOMwHJ6197KjnX5jvTb8w== 2020-11-16T21:56:44.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCc9bJfHaUxLxAAAvOMwHJ6197KjnX5jvTb8w=="}' 2020-11-16T21:56:44.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9a8371b3-ebce-456f-a23e-2f5c54c33903 -i td/osd-backfill-stats/3/new.json 2020-11-16T21:56:45.281 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:56:45.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2020-11-16T21:56:45.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCc9bJfHaUxLxAAAvOMwHJ6197KjnX5jvTb8w== --osd-uuid 9a8371b3-ebce-456f-a23e-2f5c54c33903 2020-11-16T21:56:45.326 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:45.333+0000 7f2c91659d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:45.329 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:45.337+0000 7f2c91659d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:45.331 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:45.337+0000 7f2c91659d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:45.331 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:45.337+0000 7f2c91659d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:56:45.332 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:45.337+0000 7f2c91659d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:56:45.332 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:45.337+0000 7f2c91659d80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:56:45.391 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:45.397+0000 7f2c91659d80 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2020-11-16T21:56:49.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2020-11-16T21:56:49.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:56:49.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:56:49.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:56:49.689 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:56:50.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:56:50.197 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:56:50.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:56:50.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:56:50.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:56:50.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:56:50.227 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:50.233+0000 7f741736ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:50.230 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:50.237+0000 7f741736ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:50.232 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:50.237+0000 7f741736ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:56:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:56:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:56:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:56:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:56:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:56:50.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:56:50.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:56:50.585 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:56:50.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:56:50.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:56:50.775 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:50.781+0000 7f741736ed80 -1 Falling back to public interface 2020-11-16T21:56:50.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:56:51.620 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:56:51.625+0000 7f741736ed80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:56:51.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:56:51.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:56:51.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:56:51.978 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:56:51.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:56:51.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:56:52.398 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/562920,v1:127.0.0.1:6827/562920] [v2:127.0.0.1:6828/562920,v1:127.0.0.1:6829/562920] exists,up 9a8371b3-ebce-456f-a23e-2f5c54c33903 2020-11-16T21:56:52.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:56:52.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:56:52.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:56:52.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:451: TEST_backfill_remapped: create_pool test 1 1 2020-11-16T21:56:52.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T21:56:53.061 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T21:56:53.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:56:54.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:452: TEST_backfill_remapped: ceph osd pool set test size 3 2020-11-16T21:56:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 3 2020-11-16T21:56:54.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:453: TEST_backfill_remapped: sleep 5 2020-11-16T21:56:59.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:455: TEST_backfill_remapped: wait_for_clean 2020-11-16T21:56:59.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:56:59.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:56:59.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:56:59.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:56:59.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:56:59.841 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:56:59.841 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:56:59.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:56:59.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:56:59.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:57:00.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:57:00.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:57:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:57:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:57:00.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:57:00.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:57:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:57:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:57:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:3' 2020-11-16T21:57:00.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:57:00.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:57:00.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:57:00.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836489 2020-11-16T21:57:00.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836489 2020-11-16T21:57:00.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489' 2020-11-16T21:57:00.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:57:00.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:57:00.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672967 2020-11-16T21:57:00.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672967 2020-11-16T21:57:00.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2020-11-16T21:57:00.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:57:00.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:57:00.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509446 2020-11-16T21:57:00.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509446 2020-11-16T21:57:00.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509446' 2020-11-16T21:57:00.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:57:00.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:57:01.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378628 2020-11-16T21:57:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378628 2020-11-16T21:57:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509446 3-81604378628' 2020-11-16T21:57:01.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:57:01.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2020-11-16T21:57:01.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:57:01.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:57:01.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836489 2020-11-16T21:57:01.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:57:01.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836489 2020-11-16T21:57:01.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-11-16T21:57:01.118 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836489 2020-11-16T21:57:01.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:57:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836486 -lt 21474836489 2020-11-16T21:57:01.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:57:02.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:57:02.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:57:02.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836489 -lt 21474836489 2020-11-16T21:57:02.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:57:02.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672967 2020-11-16T21:57:02.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:57:02.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:57:02.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672967 2020-11-16T21:57:02.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:57:02.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672967 2020-11-16T21:57:02.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-11-16T21:57:02.924 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672967 2020-11-16T21:57:02.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:57:03.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672967 -lt 42949672967 2020-11-16T21:57:03.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:57:03.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2020-11-16T21:57:03.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:57:03.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:57:03.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2020-11-16T21:57:03.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:57:03.294 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509446 2020-11-16T21:57:03.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509446 2020-11-16T21:57:03.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-11-16T21:57:03.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:57:03.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509446 -lt 64424509446 2020-11-16T21:57:03.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:57:03.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378628 2020-11-16T21:57:03.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:57:03.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:57:03.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378628 2020-11-16T21:57:03.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:57:03.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378628 2020-11-16T21:57:03.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378628' 2020-11-16T21:57:03.678 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378628 2020-11-16T21:57:03.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:57:04.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378628 -lt 81604378628 2020-11-16T21:57:04.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:57:04.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:04.069 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:57:04.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:57:04.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:57:04.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:57:04.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:57:04.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:57:04.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:57:04.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:57:04.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:57:04.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:57:04.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:04.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:57:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:57:05.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:57:05.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: seq 1 200 2020-11-16T21:57:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj1 /dev/null 2020-11-16T21:57:05.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:05.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj2 /dev/null 2020-11-16T21:57:05.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:05.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj3 /dev/null 2020-11-16T21:57:05.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:05.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj4 /dev/null 2020-11-16T21:57:05.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:05.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj5 /dev/null 2020-11-16T21:57:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:06.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj6 /dev/null 2020-11-16T21:57:06.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:06.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj7 /dev/null 2020-11-16T21:57:06.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:06.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj8 /dev/null 2020-11-16T21:57:06.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:06.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj9 /dev/null 2020-11-16T21:57:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj10 /dev/null 2020-11-16T21:57:06.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:06.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj11 /dev/null 2020-11-16T21:57:06.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:06.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj12 /dev/null 2020-11-16T21:57:06.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:06.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj13 /dev/null 2020-11-16T21:57:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj14 /dev/null 2020-11-16T21:57:07.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:07.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj15 /dev/null 2020-11-16T21:57:07.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:07.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj16 /dev/null 2020-11-16T21:57:07.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:07.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj17 /dev/null 2020-11-16T21:57:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj18 /dev/null 2020-11-16T21:57:07.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:07.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj19 /dev/null 2020-11-16T21:57:07.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:07.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj20 /dev/null 2020-11-16T21:57:07.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:07.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj21 /dev/null 2020-11-16T21:57:08.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:08.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj22 /dev/null 2020-11-16T21:57:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj23 /dev/null 2020-11-16T21:57:08.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:08.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj24 /dev/null 2020-11-16T21:57:08.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:08.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj25 /dev/null 2020-11-16T21:57:08.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:08.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj26 /dev/null 2020-11-16T21:57:08.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:08.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj27 /dev/null 2020-11-16T21:57:08.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:08.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj28 /dev/null 2020-11-16T21:57:08.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:08.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj29 /dev/null 2020-11-16T21:57:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj30 /dev/null 2020-11-16T21:57:09.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:09.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj31 /dev/null 2020-11-16T21:57:09.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:09.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj32 /dev/null 2020-11-16T21:57:09.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:09.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj33 /dev/null 2020-11-16T21:57:09.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:09.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj34 /dev/null 2020-11-16T21:57:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:09.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj35 /dev/null 2020-11-16T21:57:09.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:09.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj36 /dev/null 2020-11-16T21:57:09.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:09.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj37 /dev/null 2020-11-16T21:57:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj38 /dev/null 2020-11-16T21:57:10.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:10.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj39 /dev/null 2020-11-16T21:57:10.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:10.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj40 /dev/null 2020-11-16T21:57:10.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:10.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj41 /dev/null 2020-11-16T21:57:10.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:10.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj42 /dev/null 2020-11-16T21:57:10.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:10.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj43 /dev/null 2020-11-16T21:57:10.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:10.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj44 /dev/null 2020-11-16T21:57:10.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:10.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj45 /dev/null 2020-11-16T21:57:11.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:11.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj46 /dev/null 2020-11-16T21:57:11.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:11.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj47 /dev/null 2020-11-16T21:57:11.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:11.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj48 /dev/null 2020-11-16T21:57:11.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:11.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj49 /dev/null 2020-11-16T21:57:11.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:11.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj50 /dev/null 2020-11-16T21:57:11.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:11.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj51 /dev/null 2020-11-16T21:57:11.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:11.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj52 /dev/null 2020-11-16T21:57:11.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:11.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj53 /dev/null 2020-11-16T21:57:12.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:12.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj54 /dev/null 2020-11-16T21:57:12.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:12.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj55 /dev/null 2020-11-16T21:57:12.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:12.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj56 /dev/null 2020-11-16T21:57:12.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:12.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj57 /dev/null 2020-11-16T21:57:12.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:12.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj58 /dev/null 2020-11-16T21:57:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj59 /dev/null 2020-11-16T21:57:12.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:12.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj60 /dev/null 2020-11-16T21:57:12.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:12.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj61 /dev/null 2020-11-16T21:57:13.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:13.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj62 /dev/null 2020-11-16T21:57:13.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:13.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj63 /dev/null 2020-11-16T21:57:13.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:13.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj64 /dev/null 2020-11-16T21:57:13.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:13.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj65 /dev/null 2020-11-16T21:57:13.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:13.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj66 /dev/null 2020-11-16T21:57:13.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:13.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj67 /dev/null 2020-11-16T21:57:13.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:13.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj68 /dev/null 2020-11-16T21:57:13.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:13.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj69 /dev/null 2020-11-16T21:57:13.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:13.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj70 /dev/null 2020-11-16T21:57:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj71 /dev/null 2020-11-16T21:57:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj72 /dev/null 2020-11-16T21:57:14.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:14.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj73 /dev/null 2020-11-16T21:57:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:14.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj74 /dev/null 2020-11-16T21:57:14.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:14.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj75 /dev/null 2020-11-16T21:57:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj76 /dev/null 2020-11-16T21:57:14.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:14.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj77 /dev/null 2020-11-16T21:57:14.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:14.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj78 /dev/null 2020-11-16T21:57:15.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:15.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj79 /dev/null 2020-11-16T21:57:15.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:15.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj80 /dev/null 2020-11-16T21:57:15.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:15.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj81 /dev/null 2020-11-16T21:57:15.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:15.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj82 /dev/null 2020-11-16T21:57:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj83 /dev/null 2020-11-16T21:57:15.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:15.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj84 /dev/null 2020-11-16T21:57:15.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:15.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj85 /dev/null 2020-11-16T21:57:15.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:15.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj86 /dev/null 2020-11-16T21:57:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj87 /dev/null 2020-11-16T21:57:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:16.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj88 /dev/null 2020-11-16T21:57:16.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:16.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj89 /dev/null 2020-11-16T21:57:16.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:16.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj90 /dev/null 2020-11-16T21:57:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj91 /dev/null 2020-11-16T21:57:16.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:16.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj92 /dev/null 2020-11-16T21:57:16.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:16.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj93 /dev/null 2020-11-16T21:57:16.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:16.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj94 /dev/null 2020-11-16T21:57:17.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:17.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj95 /dev/null 2020-11-16T21:57:17.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:17.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj96 /dev/null 2020-11-16T21:57:17.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:17.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj97 /dev/null 2020-11-16T21:57:17.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:17.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj98 /dev/null 2020-11-16T21:57:17.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:17.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj99 /dev/null 2020-11-16T21:57:17.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:17.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj100 /dev/null 2020-11-16T21:57:17.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:17.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj101 /dev/null 2020-11-16T21:57:17.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:17.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj102 /dev/null 2020-11-16T21:57:18.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:18.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj103 /dev/null 2020-11-16T21:57:18.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:18.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj104 /dev/null 2020-11-16T21:57:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj105 /dev/null 2020-11-16T21:57:18.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:18.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj106 /dev/null 2020-11-16T21:57:18.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:18.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj107 /dev/null 2020-11-16T21:57:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj108 /dev/null 2020-11-16T21:57:18.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:18.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj109 /dev/null 2020-11-16T21:57:18.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:18.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj110 /dev/null 2020-11-16T21:57:19.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:19.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj111 /dev/null 2020-11-16T21:57:19.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:19.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj112 /dev/null 2020-11-16T21:57:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj113 /dev/null 2020-11-16T21:57:19.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:19.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj114 /dev/null 2020-11-16T21:57:19.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:19.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj115 /dev/null 2020-11-16T21:57:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:19.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj116 /dev/null 2020-11-16T21:57:19.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:19.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj117 /dev/null 2020-11-16T21:57:19.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:19.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj118 /dev/null 2020-11-16T21:57:20.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:20.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj119 /dev/null 2020-11-16T21:57:20.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:20.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj120 /dev/null 2020-11-16T21:57:20.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:20.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj121 /dev/null 2020-11-16T21:57:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:20.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj122 /dev/null 2020-11-16T21:57:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:20.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj123 /dev/null 2020-11-16T21:57:20.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:20.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj124 /dev/null 2020-11-16T21:57:20.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:20.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj125 /dev/null 2020-11-16T21:57:20.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:20.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj126 /dev/null 2020-11-16T21:57:21.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:21.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj127 /dev/null 2020-11-16T21:57:21.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:21.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj128 /dev/null 2020-11-16T21:57:21.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:21.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj129 /dev/null 2020-11-16T21:57:21.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:21.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj130 /dev/null 2020-11-16T21:57:21.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:21.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj131 /dev/null 2020-11-16T21:57:21.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:21.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj132 /dev/null 2020-11-16T21:57:21.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:21.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj133 /dev/null 2020-11-16T21:57:21.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:21.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj134 /dev/null 2020-11-16T21:57:22.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:22.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj135 /dev/null 2020-11-16T21:57:22.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:22.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj136 /dev/null 2020-11-16T21:57:22.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:22.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj137 /dev/null 2020-11-16T21:57:22.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:22.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj138 /dev/null 2020-11-16T21:57:22.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:22.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj139 /dev/null 2020-11-16T21:57:22.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:22.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj140 /dev/null 2020-11-16T21:57:22.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:22.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj141 /dev/null 2020-11-16T21:57:22.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:22.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj142 /dev/null 2020-11-16T21:57:23.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:23.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj143 /dev/null 2020-11-16T21:57:23.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:23.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj144 /dev/null 2020-11-16T21:57:23.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:23.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj145 /dev/null 2020-11-16T21:57:23.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:23.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj146 /dev/null 2020-11-16T21:57:23.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:23.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj147 /dev/null 2020-11-16T21:57:23.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:23.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj148 /dev/null 2020-11-16T21:57:23.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:23.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj149 /dev/null 2020-11-16T21:57:23.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:23.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj150 /dev/null 2020-11-16T21:57:24.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:24.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj151 /dev/null 2020-11-16T21:57:24.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:24.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj152 /dev/null 2020-11-16T21:57:24.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:24.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj153 /dev/null 2020-11-16T21:57:24.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:24.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj154 /dev/null 2020-11-16T21:57:24.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:24.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj155 /dev/null 2020-11-16T21:57:24.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:24.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj156 /dev/null 2020-11-16T21:57:24.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:24.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj157 /dev/null 2020-11-16T21:57:25.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:25.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj158 /dev/null 2020-11-16T21:57:25.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:25.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj159 /dev/null 2020-11-16T21:57:25.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:25.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj160 /dev/null 2020-11-16T21:57:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj161 /dev/null 2020-11-16T21:57:25.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:25.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj162 /dev/null 2020-11-16T21:57:25.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:25.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj163 /dev/null 2020-11-16T21:57:25.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:25.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj164 /dev/null 2020-11-16T21:57:25.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:25.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj165 /dev/null 2020-11-16T21:57:26.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:26.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj166 /dev/null 2020-11-16T21:57:26.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:26.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj167 /dev/null 2020-11-16T21:57:26.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:26.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj168 /dev/null 2020-11-16T21:57:26.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:26.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj169 /dev/null 2020-11-16T21:57:26.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:26.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj170 /dev/null 2020-11-16T21:57:26.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:26.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj171 /dev/null 2020-11-16T21:57:26.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:26.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj172 /dev/null 2020-11-16T21:57:26.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:26.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj173 /dev/null 2020-11-16T21:57:27.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:27.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj174 /dev/null 2020-11-16T21:57:27.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:27.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj175 /dev/null 2020-11-16T21:57:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj176 /dev/null 2020-11-16T21:57:27.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:27.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj177 /dev/null 2020-11-16T21:57:27.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:27.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj178 /dev/null 2020-11-16T21:57:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj179 /dev/null 2020-11-16T21:57:27.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:27.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj180 /dev/null 2020-11-16T21:57:27.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:27.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj181 /dev/null 2020-11-16T21:57:28.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:28.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj182 /dev/null 2020-11-16T21:57:28.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:28.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj183 /dev/null 2020-11-16T21:57:28.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:28.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj184 /dev/null 2020-11-16T21:57:28.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:28.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj185 /dev/null 2020-11-16T21:57:28.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:28.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj186 /dev/null 2020-11-16T21:57:28.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:28.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj187 /dev/null 2020-11-16T21:57:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj188 /dev/null 2020-11-16T21:57:28.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:28.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj189 /dev/null 2020-11-16T21:57:29.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:29.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj190 /dev/null 2020-11-16T21:57:29.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:29.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj191 /dev/null 2020-11-16T21:57:29.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:29.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj192 /dev/null 2020-11-16T21:57:29.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:29.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj193 /dev/null 2020-11-16T21:57:29.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:29.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj194 /dev/null 2020-11-16T21:57:29.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:29.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj195 /dev/null 2020-11-16T21:57:29.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:29.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj196 /dev/null 2020-11-16T21:57:29.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:29.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj197 /dev/null 2020-11-16T21:57:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:29.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj198 /dev/null 2020-11-16T21:57:30.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:30.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj199 /dev/null 2020-11-16T21:57:30.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2020-11-16T21:57:30.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj200 /dev/null 2020-11-16T21:57:30.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:462: TEST_backfill_remapped: get_pg test obj1 2020-11-16T21:57:30.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T21:57:30.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T21:57:30.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T21:57:30.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T21:57:30.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:462: TEST_backfill_remapped: local PG=1.0 2020-11-16T21:57:30.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:464: TEST_backfill_remapped: get_primary test obj1 2020-11-16T21:57:30.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:57:30.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:57:30.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:57:30.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:57:31.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:464: TEST_backfill_remapped: local primary=1 2020-11-16T21:57:31.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:465: TEST_backfill_remapped: get_not_primary test obj1 2020-11-16T21:57:31.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2020-11-16T21:57:31.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2020-11-16T21:57:31.161 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2020-11-16T21:57:31.161 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:57:31.161 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:57:31.162 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:57:31.162 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:57:31.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T21:57:31.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T21:57:31.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2020-11-16T21:57:31.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:465: TEST_backfill_remapped: local otherosd=0 2020-11-16T21:57:31.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:467: TEST_backfill_remapped: ceph osd set nobackfill 2020-11-16T21:57:32.672 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T21:57:32.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:468: TEST_backfill_remapped: ceph osd out osd.0 2020-11-16T21:57:33.215 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T21:57:33.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:469: TEST_backfill_remapped: get_osds test obj1 2020-11-16T21:57:33.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test 2020-11-16T21:57:33.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj1 2020-11-16T21:57:33.226 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test obj1 2020-11-16T21:57:33.226 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-11-16T21:57:33.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=1 2020-11-16T21:57:33.787 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:57:33.787 INFO:tasks.workunit.client.0.smithi189.stderr:0' 2020-11-16T21:57:33.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 1 2 0 2020-11-16T21:57:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:469: TEST_backfill_remapped: for i in $(get_osds $poolname obj1) 2020-11-16T21:57:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:471: TEST_backfill_remapped: '[' 1 = 1 -o 1 = 0 ']' 2020-11-16T21:57:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:473: TEST_backfill_remapped: continue 2020-11-16T21:57:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:469: TEST_backfill_remapped: for i in $(get_osds $poolname obj1) 2020-11-16T21:57:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:471: TEST_backfill_remapped: '[' 2 = 1 -o 2 = 0 ']' 2020-11-16T21:57:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:475: TEST_backfill_remapped: ceph osd out osd.2 2020-11-16T21:57:34.424 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already out. 2020-11-16T21:57:34.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:476: TEST_backfill_remapped: break 2020-11-16T21:57:34.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:478: TEST_backfill_remapped: ceph osd out osd.1 2020-11-16T21:57:35.241 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. 2020-11-16T21:57:35.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:479: TEST_backfill_remapped: ceph osd pool set test size 2 2020-11-16T21:57:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T21:57:36.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:480: TEST_backfill_remapped: sleep 2 2020-11-16T21:57:38.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:484: TEST_backfill_remapped: get_primary test obj1 2020-11-16T21:57:38.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T21:57:38.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T21:57:38.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T21:57:38.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T21:57:38.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:484: TEST_backfill_remapped: primary=0 2020-11-16T21:57:38.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:486: TEST_backfill_remapped: ceph osd unset nobackfill 2020-11-16T21:57:39.502 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T21:57:39.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:487: TEST_backfill_remapped: ceph tell osd.0 get_latest_osdmap 2020-11-16T21:57:39.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:488: TEST_backfill_remapped: ceph tell osd.0 debug kick_recovery_wq 0 2020-11-16T21:57:39.847 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T21:57:39.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:490: TEST_backfill_remapped: sleep 2 2020-11-16T21:57:41.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:492: TEST_backfill_remapped: wait_for_clean 2020-11-16T21:57:41.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:57:41.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:57:41.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:57:41.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:57:41.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:57:41.858 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:57:41.858 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:57:41.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:57:41.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:57:41.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:57:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:57:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:57:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:57:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:57:42.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:57:42.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:57:42.503 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:57:42.503 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:57:42.503 INFO:tasks.workunit.client.0.smithi189.stderr:3' 2020-11-16T21:57:42.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:57:42.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:57:42.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:57:42.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836499 2020-11-16T21:57:42.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836499 2020-11-16T21:57:42.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499' 2020-11-16T21:57:42.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:57:42.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:57:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672978 2020-11-16T21:57:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672978 2020-11-16T21:57:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672978' 2020-11-16T21:57:42.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:57:42.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:57:43.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509456 2020-11-16T21:57:43.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509456 2020-11-16T21:57:43.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509456' 2020-11-16T21:57:43.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:57:43.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:57:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378639 2020-11-16T21:57:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378639 2020-11-16T21:57:43.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509456 3-81604378639' 2020-11-16T21:57:43.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:57:43.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836499 2020-11-16T21:57:43.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:57:43.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:57:43.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836499 2020-11-16T21:57:43.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:57:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836499 2020-11-16T21:57:43.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2020-11-16T21:57:43.185 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836499 2020-11-16T21:57:43.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:57:43.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836497 -lt 21474836499 2020-11-16T21:57:43.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:57:44.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:57:44.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:57:44.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836499 -lt 21474836499 2020-11-16T21:57:44.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:57:44.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672978 2020-11-16T21:57:44.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:57:44.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:57:44.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672978 2020-11-16T21:57:44.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:57:44.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672978 2020-11-16T21:57:44.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2020-11-16T21:57:44.992 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672978 2020-11-16T21:57:44.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:57:45.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672978 -lt 42949672978 2020-11-16T21:57:45.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:57:45.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509456 2020-11-16T21:57:45.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:57:45.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:57:45.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509456 2020-11-16T21:57:45.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:57:45.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509456 2020-11-16T21:57:45.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2020-11-16T21:57:45.400 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509456 2020-11-16T21:57:45.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:57:45.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509456 -lt 64424509456 2020-11-16T21:57:45.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:57:45.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378639 2020-11-16T21:57:45.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:57:45.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:57:45.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378639 2020-11-16T21:57:45.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:57:45.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378639 2020-11-16T21:57:45.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378639' 2020-11-16T21:57:45.795 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378639 2020-11-16T21:57:45.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:57:46.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378639 -lt 81604378639 2020-11-16T21:57:46.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:57:46.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:46.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:46.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:57:46.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:57:46.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:57:46.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:57:46.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:57:46.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:57:46.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:57:46.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:57:47.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:57:47.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:57:47.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:47.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:47.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:57:47.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T21:57:47.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T21:57:47.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T21:57:47.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:57:47.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:57:47.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:57:47.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:57:47.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:57:47.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:57:47.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:57:47.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:57:47.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:57:47.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:57:48.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:57:48.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:57:48.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:48.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:48.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:57:48.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:57:48.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:57:48.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:57:48.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:57:48.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:57:48.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:57:48.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:57:48.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:57:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:57:49.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:57:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:57:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:57:49.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:57:49.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:57:49.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:57:49.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:57:49.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:57:49.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:57:49.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:57:49.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:57:49.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:57:49.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:57:49.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:57:49.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:49.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:49.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:57:49.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:57:49.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:57:49.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:57:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:57:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:57:49.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:57:49.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:57:49.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:57:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:57:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:57:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:57:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:57:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:57:50.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:57:50.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:57:50.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:57:50.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:57:50.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:57:50.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:57:50.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:57:50.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:57:50.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:57:50.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:57:50.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:50.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:51.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:57:51.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:57:51.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:57:51.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:57:51.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:57:51.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:57:51.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:57:51.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:57:51.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:57:51.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:57:51.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:57:51.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:57:51.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:57:51.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:57:51.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:57:51.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:57:51.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:57:51.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:57:51.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:57:51.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:57:51.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:57:51.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:57:52.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:57:52.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:57:52.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:52.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:52.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:57:52.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:57:52.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:57:52.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:57:52.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:57:52.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:57:52.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:57:52.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:57:52.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:57:53.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:57:53.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:57:53.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:57:53.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:57:53.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:57:53.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:57:53.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:57:53.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:57:53.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:57:53.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:57:53.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:57:53.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:57:53.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:57:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:57:53.698 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:57:53.698 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:53.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:57:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:57:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:57:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:57:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:57:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:57:54.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:57:54.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:57:54.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:57:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:57:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:57:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:57:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:57:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:57:54.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:57:54.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:57:54.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:57:54.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:57:54.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:57:54.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:57:54.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:57:54.926 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:57:55.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:57:55.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:57:55.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:55.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:55.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:57:55.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:57:55.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:57:55.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:57:55.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:57:55.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:57:55.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:57:55.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:57:55.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:57:56.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7 2020-11-16T21:57:56.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7 '!=' null 2020-11-16T21:57:56.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:57:56.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:57:56.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:57:56.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:57:56.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:57:56.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:57:56.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:57:56.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:57:56.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:57:56.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:57:56.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:57:56.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:57:56.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:57:56.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:56.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:57.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:57:57.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:57:57.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:57:57.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:57:57.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:57:57.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:57:57.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:57:57.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:57:57.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:57:57.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7 2020-11-16T21:57:57.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7 '!=' null 2020-11-16T21:57:57.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:57:57.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:57:57.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:57:57.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:57:57.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:57:57.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:57:57.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:57:57.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:57:57.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:57:57.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:57:57.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:57:58.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:57:58.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:57:58.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:58.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:57:58.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:57:58.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:57:58.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:57:58.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:57:58.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:57:58.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:57:58.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:57:58.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:57:58.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:57:59.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:57:59.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:57:59.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:57:59.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:57:59.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:57:59.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:57:59.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:57:59.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:57:59.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:57:59.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:57:59.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:57:59.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:57:59.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:57:59.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:57:59.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:57:59.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:57:59.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:00.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:58:00.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:58:00.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:58:00.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:58:00.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:58:00.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:58:00.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:58:00.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:58:00.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:58:00.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:58:00.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:58:00.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:58:00.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:58:00.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:58:00.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:58:00.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:58:00.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:58:00.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:58:00.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:58:00.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:58:00.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:58:00.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:58:00.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:58:00.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:58:00.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:58:00.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:01.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:58:01.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:58:01.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:58:01.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:58:01.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:58:01.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:58:01.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:58:01.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:58:01.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:58:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:58:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:58:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:58:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:58:01.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:58:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:58:02.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:58:02.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:58:02.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:58:02.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:58:02.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:58:02.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:58:02.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:58:02.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:58:02.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:58:02.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:58:02.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:02.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:58:02.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:58:02.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:58:02.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:58:02.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:58:02.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:58:02.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:58:02.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:58:02.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:58:03.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2020-11-16T21:58:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2020-11-16T21:58:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:58:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:58:03.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:58:03.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:58:03.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:58:03.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:58:03.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:58:03.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:58:03.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:58:03.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:58:03.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:58:03.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:58:03.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:58:03.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:58:03.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:58:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:58:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:58:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:58:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:58:04.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:58:04.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:58:04.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:58:04.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:58:04.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:58:04.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:58:04.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:58:04.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:58:04.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:58:04.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:58:04.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:58:04.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:58:04.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:58:04.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:58:04.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:58:04.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:58:04.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:58:05.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:58:05.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:58:05.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:58:05.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:05.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:58:05.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:58:05.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:58:05.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:58:05.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:58:05.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:58:05.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:58:05.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:58:05.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:58:06.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2020-11-16T21:58:06.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2020-11-16T21:58:06.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:58:06.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:58:06.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:58:06.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:58:06.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:58:06.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:58:06.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:58:06.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:58:06.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:58:06.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:58:06.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:58:06.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:58:06.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:58:06.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:58:06.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:07.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:58:07.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:58:07.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:58:07.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:58:07.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:58:07.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:58:07.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:58:07.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:58:07.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:58:07.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2020-11-16T21:58:07.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2020-11-16T21:58:07.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:58:07.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:58:07.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:58:07.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:58:07.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:58:07.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:58:07.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:58:07.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:58:07.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:58:07.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:58:07.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:58:08.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:58:08.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:58:08.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:58:08.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:08.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:58:08.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:58:08.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:58:08.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:58:08.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:58:08.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:58:08.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:58:08.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:58:08.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:58:09.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:58:09.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:58:09.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:58:09.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:58:09.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:58:09.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:58:09.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:58:09.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:58:09.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:58:09.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:58:09.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:58:09.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:58:09.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:58:09.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:58:09.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:58:09.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:58:09.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:09.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:58:09.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:58:09.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:58:09.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:58:09.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:58:09.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:58:09.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:58:09.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:58:09.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:58:10.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:58:10.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:58:10.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:58:10.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:58:10.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:58:10.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:58:10.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:58:10.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:58:10.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:58:10.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:58:10.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:58:10.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:58:10.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:58:11.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:58:11.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:58:11.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:58:11.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:58:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:58:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:58:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:58:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:58:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:58:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:58:11.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:58:11.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:58:11.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:58:11.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:58:11.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:58:11.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:58:11.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:58:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:58:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:58:12.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:58:12.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:58:12.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:58:12.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:58:12.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:58:12.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:58:12.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:58:12.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:58:12.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:58:12.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:12.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:58:12.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:58:12.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:58:12.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:58:12.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:58:12.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:58:12.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:58:12.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:58:12.945 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:58:13.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:58:13.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:58:13.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:58:13.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:58:13.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:58:13.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:58:13.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:58:13.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:58:13.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:58:13.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:58:13.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:58:13.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:58:13.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:58:13.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:58:13.882 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:58:13.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:58:13.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:14.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:58:14.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:58:14.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:58:14.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:58:14.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:58:14.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:58:14.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:58:14.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:58:14.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:58:14.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:58:14.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:58:14.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:58:14.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:58:14.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:58:14.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:58:14.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:58:14.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:58:14.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:58:14.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:58:14.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:58:14.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:58:14.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:58:15.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:58:15.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:58:15.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:58:15.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:58:15.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:58:15.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:58:15.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:58:15.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:494: TEST_backfill_remapped: expr 200 '*' 2 2020-11-16T21:58:15.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:494: TEST_backfill_remapped: local misplaced=400 2020-11-16T21:58:15.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:496: TEST_backfill_remapped: check td/osd-backfill-stats 1.0 0 replicated 0 0 400 200 '' '' false 2020-11-16T21:58:15.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2020-11-16T21:58:15.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2020-11-16T21:58:15.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=0 2020-11-16T21:58:15.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2020-11-16T21:58:15.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=0 2020-11-16T21:58:15.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2020-11-16T21:58:15.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=400 2020-11-16T21:58:15.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=200 2020-11-16T21:58:15.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2020-11-16T21:58:15.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2020-11-16T21:58:15.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=false 2020-11-16T21:58:15.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log 2020-11-16T21:58:15.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.0.log 2020-11-16T21:58:15.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' false = true ']' 2020-11-16T21:58:15.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2020-11-16T21:58:15.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2020-11-16T21:58:15.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2020-11-16T21:58:15.760 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.0.log 2020-11-16T21:58:15.760 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2020-11-16T21:58:15.760 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2020-11-16T21:58:15.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[3]/[0,1]' 2020-11-16T21:58:15.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[3]/[0,1]' ']' 2020-11-16T21:58:15.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2020-11-16T21:58:15.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:58:15.781 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.0.log 2020-11-16T21:58:15.782 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [3]/[0,1] ' 2020-11-16T21:58:15.782 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2020-11-16T21:58:15.782 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2020-11-16T21:58:15.782 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:58:15.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=0 2020-11-16T21:58:15.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 0 ']' 2020-11-16T21:58:15.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 0 0 2020-11-16T21:58:15.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2020-11-16T21:58:15.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:58:15.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2020-11-16T21:58:15.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:58:15.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:58:15.787 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.0.log 2020-11-16T21:58:15.787 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2020-11-16T21:58:15.788 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:58:15.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=0 2020-11-16T21:58:15.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 0 ']' 2020-11-16T21:58:15.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 0 0 2020-11-16T21:58:15.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2020-11-16T21:58:15.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:58:15.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T21:58:15.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:58:15.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2020-11-16T21:58:15.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:58:15.795 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.0.log 2020-11-16T21:58:15.795 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [3]/[0,1] ' 2020-11-16T21:58:15.795 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2020-11-16T21:58:15.795 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2020-11-16T21:58:15.796 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:58:15.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=400 2020-11-16T21:58:15.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 400 ']' 2020-11-16T21:58:15.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 400 400 2020-11-16T21:58:15.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2020-11-16T21:58:15.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T21:58:15.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2020-11-16T21:58:15.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T21:58:15.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T21:58:15.799 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.0.log 2020-11-16T21:58:15.800 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2020-11-16T21:58:15.800 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T21:58:15.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=201 2020-11-16T21:58:15.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 201 ']' 2020-11-16T21:58:15.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 201 200 2020-11-16T21:58:15.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=201 2020-11-16T21:58:15.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T21:58:15.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=200 2020-11-16T21:58:15.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T21:58:15.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2020-11-16T21:58:15.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:498: TEST_backfill_remapped: delete_pool test 2020-11-16T21:58:15.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T21:58:15.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T21:58:16.372 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T21:58:16.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:499: TEST_backfill_remapped: kill_daemons td/osd-backfill-stats 2020-11-16T21:58:16.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:58:16.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:58:16.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:58:16.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:58:16.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:58:16.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:58:16.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2020-11-16T21:58:16.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:58:16.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:58:16.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:58:16.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:58:16.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:58:16.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:58:16.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:58:16.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:58:16.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:58:16.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:58:16.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:58:16.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:58:16.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:58:16.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:58:16.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:58:16.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:58:16.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:58:16.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:58:16.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:58:16.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:58:16.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:58:16.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:58:16.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:58:16.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:16.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:16.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:58:16.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:58:16.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:58:16.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2020-11-16T21:58:16.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2020-11-16T21:58:16.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2020-11-16T21:58:16.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2020-11-16T21:58:16.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T21:58:16.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T21:58:16.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T21:58:16.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T21:58:16.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T21:58:16.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T21:58:16.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T21:58:16.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T21:58:16.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T21:58:16.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T21:58:16.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T21:58:16.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T21:58:16.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T21:58:16.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T21:58:16.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T21:58:16.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:58:16.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T21:58:16.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T21:58:16.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T21:58:16.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T21:58:16.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T21:58:16.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T21:58:16.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T21:58:16.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:16.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:16.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T21:58:16.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T21:58:16.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T21:58:16.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2020-11-16T21:58:16.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T21:58:16.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:16.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:16.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.465868 2020-11-16T21:58:16.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T21:58:16.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T21:58:16.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T21:58:16.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2020-11-16T21:58:16.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_sizeup td/osd-backfill-stats 2020-11-16T21:58:16.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:133: TEST_backfill_sizeup: local dir=td/osd-backfill-stats 2020-11-16T21:58:16.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:135: TEST_backfill_sizeup: run_mon td/osd-backfill-stats a 2020-11-16T21:58:16.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2020-11-16T21:58:16.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T21:58:16.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T21:58:16.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T21:58:16.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2020-11-16T21:58:16.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2020-11-16T21:58:17.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T21:58:17.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:58:17.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:58:17.062 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:58:17.062 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:17.062 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:17.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:17.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T21:58:17.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T21:58:17.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T21:58:17.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:58:17.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:58:17.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T21:58:17.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T21:58:17.276 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:58:17.276 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:58:17.276 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:58:17.277 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:58:17.277 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:17.277 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:17.278 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:58:17.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:58:17.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get fsid 2020-11-16T21:58:17.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T21:58:17.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T21:58:17.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T21:58:17.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T21:58:17.391 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T21:58:17.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T21:58:17.391 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T21:58:17.391 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T21:58:17.392 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T21:58:17.392 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:17.392 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:17.392 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T21:58:17.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T21:58:17.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get mon_host 2020-11-16T21:58:17.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:136: TEST_backfill_sizeup: run_mgr td/osd-backfill-stats x 2020-11-16T21:58:17.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2020-11-16T21:58:17.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T21:58:17.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T21:58:17.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T21:58:17.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2020-11-16T21:58:17.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T21:58:18.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T21:58:18.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:58:18.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:58:18.039 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:58:18.039 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:18.039 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:18.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:18.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:58:18.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T21:58:18.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:137: TEST_backfill_sizeup: export CEPH_ARGS 2020-11-16T21:58:18.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:138: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 0 2020-11-16T21:58:18.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:58:18.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:58:18.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T21:58:18.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:58:18.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2020-11-16T21:58:18.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:58:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:58:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:58:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:58:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2020-11-16T21:58:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2020-11-16T21:58:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:58:18.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:58:18.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:58:18.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:58:18.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:58:18.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:58:18.073 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:58:18.073 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:18.073 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:18.074 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:18.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:18.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:58:18.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:58:18.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:58:18.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:58:18.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:58:18.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:58:18.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:58:18.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:58:18.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:58:18.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:58:18.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2020-11-16T21:58:18.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:58:18.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=842c0b54-0f04-4ca3-a66f-8d6b151f2676 2020-11-16T21:58:18.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 842c0b54-0f04-4ca3-a66f-8d6b151f2676' 2020-11-16T21:58:18.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:58:18.077 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 842c0b54-0f04-4ca3-a66f-8d6b151f2676 2020-11-16T21:58:18.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD69bJfyitPBhAAYalYc2uFjfGCYqxe4ppc1g== 2020-11-16T21:58:18.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD69bJfyitPBhAAYalYc2uFjfGCYqxe4ppc1g=="}' 2020-11-16T21:58:18.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 842c0b54-0f04-4ca3-a66f-8d6b151f2676 -i td/osd-backfill-stats/0/new.json 2020-11-16T21:58:18.446 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:58:18.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2020-11-16T21:58:18.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD69bJfyitPBhAAYalYc2uFjfGCYqxe4ppc1g== --osd-uuid 842c0b54-0f04-4ca3-a66f-8d6b151f2676 2020-11-16T21:58:18.491 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:18.499+0000 7fc632c0ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:18.493 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:18.503+0000 7fc632c0ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:18.495 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:18.503+0000 7fc632c0ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:18.496 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:18.503+0000 7fc632c0ad80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:58:18.496 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:18.503+0000 7fc632c0ad80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:58:18.496 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:18.503+0000 7fc632c0ad80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T21:58:18.540 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:18.547+0000 7fc632c0ad80 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2020-11-16T21:58:22.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2020-11-16T21:58:22.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:58:22.828 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T21:58:22.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T21:58:22.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:58:23.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T21:58:23.333 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T21:58:23.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:58:23.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:58:23.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:58:23.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:58:23.363 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:23.371+0000 7f680223dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:23.364 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:23.371+0000 7f680223dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:23.366 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:23.375+0000 7f680223dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:23.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T21:58:23.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:58:23.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T21:58:23.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:58:23.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:58:23.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:58:23.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:58:23.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:58:23.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:58:23.723 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:58:23.903 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:23.911+0000 7f680223dd80 -1 Falling back to public interface 2020-11-16T21:58:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:58:24.753 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:24.763+0000 7f680223dd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T21:58:25.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:58:25.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:58:25.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:58:25.111 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:58:25.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:58:25.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T21:58:25.496 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/571115,v1:127.0.0.1:6803/571115] [v2:127.0.0.1:6804/571115,v1:127.0.0.1:6805/571115] exists,up 842c0b54-0f04-4ca3-a66f-8d6b151f2676 2020-11-16T21:58:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:58:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:58:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:58:25.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:139: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 1 2020-11-16T21:58:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:58:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:58:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T21:58:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:58:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2020-11-16T21:58:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:58:25.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:58:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:58:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:58:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2020-11-16T21:58:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2020-11-16T21:58:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:58:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:58:25.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:58:25.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:58:25.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:58:25.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:58:25.500 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:58:25.500 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:25.500 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:25.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:58:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:58:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:58:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:58:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:58:25.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:58:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:58:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:58:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:58:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:58:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2020-11-16T21:58:25.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:58:25.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=241d8256-bf43-458f-919c-7080e2a06f36 2020-11-16T21:58:25.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 241d8256-bf43-458f-919c-7080e2a06f36' 2020-11-16T21:58:25.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:58:25.503 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 241d8256-bf43-458f-919c-7080e2a06f36 2020-11-16T21:58:25.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAB9rJfWN7jHxAA6zRu9XG9CW/lVe5+C4IAHg== 2020-11-16T21:58:25.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAB9rJfWN7jHxAA6zRu9XG9CW/lVe5+C4IAHg=="}' 2020-11-16T21:58:25.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 241d8256-bf43-458f-919c-7080e2a06f36 -i td/osd-backfill-stats/1/new.json 2020-11-16T21:58:25.983 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:58:25.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2020-11-16T21:58:25.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAB9rJfWN7jHxAA6zRu9XG9CW/lVe5+C4IAHg== --osd-uuid 241d8256-bf43-458f-919c-7080e2a06f36 2020-11-16T21:58:26.025 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:26.035+0000 7fe80f442d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:26.027 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:26.035+0000 7fe80f442d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:26.029 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:26.039+0000 7fe80f442d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:26.030 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:26.039+0000 7fe80f442d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:58:26.030 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:26.039+0000 7fe80f442d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:58:26.030 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:26.039+0000 7fe80f442d80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T21:58:26.069 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:26.075+0000 7fe80f442d80 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2020-11-16T21:58:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2020-11-16T21:58:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:58:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T21:58:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:58:30.333 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T21:58:30.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T21:58:30.840 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T21:58:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:58:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:58:30.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:58:30.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:58:30.870 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:30.879+0000 7fe2980b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:30.872 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:30.879+0000 7fe2980b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:30.873 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:30.883+0000 7fe2980b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:31.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T21:58:31.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:58:31.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T21:58:31.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:58:31.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:58:31.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:58:31.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:58:31.228 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:58:31.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:58:31.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:58:31.411 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:31.419+0000 7fe2980b1d80 -1 Falling back to public interface 2020-11-16T21:58:31.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:58:32.329 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:32.339+0000 7fe2980b1d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T21:58:32.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:58:32.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:58:32.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:58:32.604 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:58:32.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:58:32.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T21:58:33.017 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/572109,v1:127.0.0.1:6811/572109] [v2:127.0.0.1:6812/572109,v1:127.0.0.1:6813/572109] exists,up 241d8256-bf43-458f-919c-7080e2a06f36 2020-11-16T21:58:33.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:58:33.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:58:33.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:58:33.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:140: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 2 2020-11-16T21:58:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:58:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:58:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T21:58:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:58:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2020-11-16T21:58:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:58:33.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:58:33.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:58:33.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:58:33.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2020-11-16T21:58:33.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2020-11-16T21:58:33.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:58:33.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:58:33.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:58:33.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:58:33.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:58:33.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:58:33.022 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:58:33.022 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:33.023 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:33.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:33.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:33.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:58:33.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:58:33.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:58:33.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:58:33.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:58:33.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:58:33.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:58:33.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:58:33.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:58:33.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:58:33.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2020-11-16T21:58:33.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:58:33.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=133d8622-0464-4d2c-965d-28b0ff709e7e 2020-11-16T21:58:33.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 133d8622-0464-4d2c-965d-28b0ff709e7e' 2020-11-16T21:58:33.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:58:33.026 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 133d8622-0464-4d2c-965d-28b0ff709e7e 2020-11-16T21:58:33.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAJ9rJfP9VWAxAArMA0v7bNXfltlnMt0KsV5w== 2020-11-16T21:58:33.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAJ9rJfP9VWAxAArMA0v7bNXfltlnMt0KsV5w=="}' 2020-11-16T21:58:33.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 133d8622-0464-4d2c-965d-28b0ff709e7e -i td/osd-backfill-stats/2/new.json 2020-11-16T21:58:33.554 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T21:58:33.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2020-11-16T21:58:33.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAJ9rJfP9VWAxAArMA0v7bNXfltlnMt0KsV5w== --osd-uuid 133d8622-0464-4d2c-965d-28b0ff709e7e 2020-11-16T21:58:33.595 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:33.603+0000 7fe174803d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:33.598 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:33.607+0000 7fe174803d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:33.600 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:33.607+0000 7fe174803d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:33.600 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:33.607+0000 7fe174803d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:58:33.600 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:33.607+0000 7fe174803d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:58:33.600 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:33.607+0000 7fe174803d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T21:58:33.648 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:33.655+0000 7fe174803d80 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2020-11-16T21:58:37.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2020-11-16T21:58:37.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:58:37.961 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T21:58:37.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T21:58:37.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:58:38.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T21:58:38.468 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T21:58:38.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:58:38.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:58:38.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:58:38.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:58:38.497 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:38.507+0000 7fe8c8a30d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:38.500 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:38.507+0000 7fe8c8a30d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:38.501 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:38.511+0000 7fe8c8a30d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:38.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T21:58:38.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:58:38.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T21:58:38.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:58:38.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:58:38.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:58:38.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:58:38.857 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:58:38.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:58:38.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:58:39.038 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:39.047+0000 7fe8c8a30d80 -1 Falling back to public interface 2020-11-16T21:58:39.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:58:39.892 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:39.899+0000 7fe8c8a30d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T21:58:40.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:58:40.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:58:40.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:58:40.260 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:58:40.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:58:40.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T21:58:40.688 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/573103,v1:127.0.0.1:6819/573103] [v2:127.0.0.1:6820/573103,v1:127.0.0.1:6821/573103] exists,up 133d8622-0464-4d2c-965d-28b0ff709e7e 2020-11-16T21:58:40.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:58:40.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:58:40.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:58:40.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:141: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 3 2020-11-16T21:58:40.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:58:40.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:58:40.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T21:58:40.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:58:40.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2020-11-16T21:58:40.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:58:40.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:58:40.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:58:40.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:58:40.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2020-11-16T21:58:40.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2020-11-16T21:58:40.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:58:40.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:58:40.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:58:40.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:58:40.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:58:40.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:58:40.693 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:58:40.693 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:40.693 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:40.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:40.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:40.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:58:40.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:58:40.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:58:40.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:58:40.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:58:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:58:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:58:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:58:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:58:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:58:40.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2020-11-16T21:58:40.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:58:40.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=43c2b34a-b38b-458e-8c1c-23738da79369 2020-11-16T21:58:40.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 43c2b34a-b38b-458e-8c1c-23738da79369' 2020-11-16T21:58:40.697 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 43c2b34a-b38b-458e-8c1c-23738da79369 2020-11-16T21:58:40.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:58:40.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAQ9rJfTqBoKxAACF+Q3ZyZqz3WK8bC0clqQA== 2020-11-16T21:58:40.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAQ9rJfTqBoKxAACF+Q3ZyZqz3WK8bC0clqQA=="}' 2020-11-16T21:58:40.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 43c2b34a-b38b-458e-8c1c-23738da79369 -i td/osd-backfill-stats/3/new.json 2020-11-16T21:58:41.278 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T21:58:41.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2020-11-16T21:58:41.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAQ9rJfTqBoKxAACF+Q3ZyZqz3WK8bC0clqQA== --osd-uuid 43c2b34a-b38b-458e-8c1c-23738da79369 2020-11-16T21:58:41.321 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:41.331+0000 7fbc74bebd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:41.325 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:41.335+0000 7fbc74bebd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:41.327 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:41.335+0000 7fbc74bebd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:41.327 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:41.335+0000 7fbc74bebd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:58:41.328 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:41.335+0000 7fbc74bebd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:58:41.328 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:41.335+0000 7fbc74bebd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T21:58:41.433 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:41.443+0000 7fbc74bebd80 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2020-11-16T21:58:45.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2020-11-16T21:58:45.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:58:45.485 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T21:58:45.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T21:58:45.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:58:46.007 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T21:58:46.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T21:58:46.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:58:46.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:58:46.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:58:46.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:58:46.033 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:46.039+0000 7f90cd545d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:46.035 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:46.043+0000 7f90cd545d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:46.043+0000 7f90cd545d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:46.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T21:58:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:58:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T21:58:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:58:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:58:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:58:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:58:46.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:58:46.376 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:58:46.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:58:46.575 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:46.583+0000 7f90cd545d80 -1 Falling back to public interface 2020-11-16T21:58:46.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:58:47.419 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:47.427+0000 7f90cd545d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T21:58:47.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:58:47.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:58:47.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:58:47.777 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:58:47.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:58:47.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T21:58:48.225 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/574097,v1:127.0.0.1:6827/574097] [v2:127.0.0.1:6828/574097,v1:127.0.0.1:6829/574097] exists,up 43c2b34a-b38b-458e-8c1c-23738da79369 2020-11-16T21:58:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:58:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:58:48.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:58:48.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:142: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 4 2020-11-16T21:58:48.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:58:48.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:58:48.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T21:58:48.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:58:48.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2020-11-16T21:58:48.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:58:48.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:58:48.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:58:48.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:58:48.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2020-11-16T21:58:48.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2020-11-16T21:58:48.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:58:48.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:58:48.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:58:48.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:58:48.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:58:48.230 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:58:48.230 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:58:48.230 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:48.230 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:48.230 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:48.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:48.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:58:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:58:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:58:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:58:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:58:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:58:48.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:58:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:58:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:58:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:58:48.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2020-11-16T21:58:48.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:58:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3fd9ff1d-f02b-485c-b195-0cbb9b23b73a 2020-11-16T21:58:48.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 3fd9ff1d-f02b-485c-b195-0cbb9b23b73a' 2020-11-16T21:58:48.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:58:48.233 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 3fd9ff1d-f02b-485c-b195-0cbb9b23b73a 2020-11-16T21:58:48.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAY9rJfmgV4DxAAFVzsqR0FSaoEziZbLWD2rg== 2020-11-16T21:58:48.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAY9rJfmgV4DxAAFVzsqR0FSaoEziZbLWD2rg=="}' 2020-11-16T21:58:48.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3fd9ff1d-f02b-485c-b195-0cbb9b23b73a -i td/osd-backfill-stats/4/new.json 2020-11-16T21:58:48.827 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T21:58:48.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2020-11-16T21:58:48.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAY9rJfmgV4DxAAFVzsqR0FSaoEziZbLWD2rg== --osd-uuid 3fd9ff1d-f02b-485c-b195-0cbb9b23b73a 2020-11-16T21:58:48.868 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:48.875+0000 7f526ebcad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:48.887 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:48.895+0000 7f526ebcad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:48.890 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:48.899+0000 7f526ebcad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:48.890 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:48.899+0000 7f526ebcad80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:58:48.891 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:48.899+0000 7f526ebcad80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:58:48.891 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:48.899+0000 7f526ebcad80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T21:58:48.971 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:48.979+0000 7f526ebcad80 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2020-11-16T21:58:53.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2020-11-16T21:58:53.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:58:53.368 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T21:58:53.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T21:58:53.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:58:53.870 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T21:58:53.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T21:58:53.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:58:53.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:58:53.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:58:53.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:58:53.899 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:53.907+0000 7f85cd366d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:53.901 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:53.911+0000 7f85cd366d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:53.902 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:53.911+0000 7f85cd366d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:54.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T21:58:54.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:58:54.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T21:58:54.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:58:54.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:58:54.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:58:54.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:58:54.265 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:58:54.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:58:54.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:58:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:54.443+0000 7f85cd366d80 -1 Falling back to public interface 2020-11-16T21:58:54.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:58:55.252 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:55.259+0000 7f85cd366d80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T21:58:55.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:58:55.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:58:55.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:58:55.629 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:58:55.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:58:55.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T21:58:56.044 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/575091,v1:127.0.0.1:6835/575091] [v2:127.0.0.1:6836/575091,v1:127.0.0.1:6837/575091] exists,up 3fd9ff1d-f02b-485c-b195-0cbb9b23b73a 2020-11-16T21:58:56.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:58:56.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:58:56.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:58:56.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:143: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 5 2020-11-16T21:58:56.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T21:58:56.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T21:58:56.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T21:58:56.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T21:58:56.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2020-11-16T21:58:56.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T21:58:56.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T21:58:56.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T21:58:56.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T21:58:56.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2020-11-16T21:58:56.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2020-11-16T21:58:56.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T21:58:56.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T21:58:56.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T21:58:56.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T21:58:56.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T21:58:56.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T21:58:56.050 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T21:58:56.050 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T21:58:56.050 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T21:58:56.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:56.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T21:58:56.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T21:58:56.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T21:58:56.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T21:58:56.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T21:58:56.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T21:58:56.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T21:58:56.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T21:58:56.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:58:56.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T21:58:56.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T21:58:56.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2020-11-16T21:58:56.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T21:58:56.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=abf1a77d-518b-4ccd-8082-968f9cc9f8ed 2020-11-16T21:58:56.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 abf1a77d-518b-4ccd-8082-968f9cc9f8ed' 2020-11-16T21:58:56.053 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 abf1a77d-518b-4ccd-8082-968f9cc9f8ed 2020-11-16T21:58:56.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T21:58:56.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAg9rJfkOPmBBAAwXTIrdgJD2i+PRDhgWsYqg== 2020-11-16T21:58:56.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAg9rJfkOPmBBAAwXTIrdgJD2i+PRDhgWsYqg=="}' 2020-11-16T21:58:56.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new abf1a77d-518b-4ccd-8082-968f9cc9f8ed -i td/osd-backfill-stats/5/new.json 2020-11-16T21:58:56.666 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T21:58:56.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2020-11-16T21:58:56.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAg9rJfkOPmBBAAwXTIrdgJD2i+PRDhgWsYqg== --osd-uuid abf1a77d-518b-4ccd-8082-968f9cc9f8ed 2020-11-16T21:58:56.710 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:56.719+0000 7fc1662d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:56.713 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:56.723+0000 7fc1662d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:56.715 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:56.723+0000 7fc1662d6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:58:56.715 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:56.723+0000 7fc1662d6d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:58:56.715 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:56.723+0000 7fc1662d6d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:58:56.716 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:56.723+0000 7fc1662d6d80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T21:58:56.762 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:58:56.771+0000 7fc1662d6d80 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2020-11-16T21:59:01.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2020-11-16T21:59:01.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T21:59:01.061 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T21:59:01.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T21:59:01.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T21:59:01.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T21:59:01.572 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T21:59:01.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T21:59:01.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T21:59:01.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T21:59:01.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T21:59:01.602 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:59:01.611+0000 7f30f72dcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:59:01.605 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:59:01.615+0000 7f30f72dcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:59:01.606 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:59:01.615+0000 7f30f72dcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T21:59:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T21:59:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T21:59:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T21:59:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T21:59:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T21:59:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:59:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T21:59:01.979 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T21:59:01.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:59:01.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:59:02.150 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:59:02.159+0000 7f30f72dcd80 -1 Falling back to public interface 2020-11-16T21:59:02.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T21:59:03.010 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T21:59:03.011+0000 7f30f72dcd80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T21:59:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T21:59:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T21:59:03.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T21:59:03.391 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T21:59:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T21:59:03.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T21:59:03.862 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/576085,v1:127.0.0.1:6843/576085] [v2:127.0.0.1:6844/576085,v1:127.0.0.1:6845/576085] exists,up abf1a77d-518b-4ccd-8082-968f9cc9f8ed 2020-11-16T21:59:03.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T21:59:03.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T21:59:03.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T21:59:03.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:145: TEST_backfill_sizeup: create_pool test 1 1 2020-11-16T21:59:03.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T21:59:04.422 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T21:59:04.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T21:59:05.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:146: TEST_backfill_sizeup: ceph osd pool set test size 1 --yes-i-really-mean-it 2020-11-16T21:59:06.426 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 1 2020-11-16T21:59:06.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:148: TEST_backfill_sizeup: wait_for_clean 2020-11-16T21:59:06.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:59:06.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:59:06.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:59:06.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:59:06.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:59:06.439 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:59:06.440 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:59:06.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:59:06.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:59:06.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:59:06.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:59:06.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:59:06.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:59:06.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:59:06.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:59:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:59:07.029 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:59:07.030 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:59:07.030 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:59:07.030 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:59:07.030 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:59:07.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:59:07.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:07.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:59:07.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2020-11-16T21:59:07.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2020-11-16T21:59:07.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2020-11-16T21:59:07.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:07.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:59:07.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672970 2020-11-16T21:59:07.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672970 2020-11-16T21:59:07.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2020-11-16T21:59:07.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:07.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:59:07.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542152 2020-11-16T21:59:07.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542152 2020-11-16T21:59:07.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542152' 2020-11-16T21:59:07.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:07.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:59:07.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378631 2020-11-16T21:59:07.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378631 2020-11-16T21:59:07.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542152 3-81604378631' 2020-11-16T21:59:07.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:07.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:59:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215109 2020-11-16T21:59:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215109 2020-11-16T21:59:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542152 3-81604378631 4-103079215109' 2020-11-16T21:59:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:07.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:59:08.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051588 2020-11-16T21:59:08.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051588 2020-11-16T21:59:08.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542152 3-81604378631 4-103079215109 5-124554051588' 2020-11-16T21:59:08.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:08.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2020-11-16T21:59:08.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:08.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:59:08.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:08.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2020-11-16T21:59:08.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2020-11-16T21:59:08.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2020-11-16T21:59:08.045 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836491 2020-11-16T21:59:08.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:59:08.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836491 2020-11-16T21:59:08.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:08.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2020-11-16T21:59:08.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:08.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:59:08.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672970 2020-11-16T21:59:08.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:08.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672970 2020-11-16T21:59:08.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2020-11-16T21:59:08.458 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672970 2020-11-16T21:59:08.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:59:08.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672970 -lt 42949672970 2020-11-16T21:59:08.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:08.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542152 2020-11-16T21:59:08.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:08.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:59:08.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542152 2020-11-16T21:59:08.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:08.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542152 2020-11-16T21:59:08.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2020-11-16T21:59:08.851 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542152 2020-11-16T21:59:08.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:59:09.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542152 -lt 60129542152 2020-11-16T21:59:09.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:09.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378631 2020-11-16T21:59:09.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:09.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:59:09.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378631 2020-11-16T21:59:09.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:09.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378631 2020-11-16T21:59:09.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378631' 2020-11-16T21:59:09.224 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378631 2020-11-16T21:59:09.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:59:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378631 -lt 81604378631 2020-11-16T21:59:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:09.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215109 2020-11-16T21:59:09.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:59:09.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215109 2020-11-16T21:59:09.600 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:09.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215109 2020-11-16T21:59:09.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215109' 2020-11-16T21:59:09.603 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215109 2020-11-16T21:59:09.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:59:09.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215109 -lt 103079215109 2020-11-16T21:59:09.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:09.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051588 2020-11-16T21:59:09.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:09.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:59:09.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051588 2020-11-16T21:59:09.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:09.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051588 2020-11-16T21:59:09.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051588' 2020-11-16T21:59:09.995 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051588 2020-11-16T21:59:09.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:59:10.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051589 -lt 124554051588 2020-11-16T21:59:10.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:59:10.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:10.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:10.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:59:10.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:10.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:10.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:10.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:10.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:10.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:10.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:11.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T21:59:11.221 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:11.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:11.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:11.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T21:59:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T21:59:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T21:59:11.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: seq 1 200 2020-11-16T21:59:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj1 /dev/null 2020-11-16T21:59:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj2 /dev/null 2020-11-16T21:59:11.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:11.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj3 /dev/null 2020-11-16T21:59:11.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:11.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj4 /dev/null 2020-11-16T21:59:12.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj5 /dev/null 2020-11-16T21:59:12.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj6 /dev/null 2020-11-16T21:59:12.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj7 /dev/null 2020-11-16T21:59:12.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj8 /dev/null 2020-11-16T21:59:12.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj9 /dev/null 2020-11-16T21:59:12.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj10 /dev/null 2020-11-16T21:59:12.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj11 /dev/null 2020-11-16T21:59:12.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj12 /dev/null 2020-11-16T21:59:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj13 /dev/null 2020-11-16T21:59:12.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj14 /dev/null 2020-11-16T21:59:12.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj15 /dev/null 2020-11-16T21:59:12.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj16 /dev/null 2020-11-16T21:59:12.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:12.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj17 /dev/null 2020-11-16T21:59:13.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj18 /dev/null 2020-11-16T21:59:13.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj19 /dev/null 2020-11-16T21:59:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj20 /dev/null 2020-11-16T21:59:13.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj21 /dev/null 2020-11-16T21:59:13.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj22 /dev/null 2020-11-16T21:59:13.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj23 /dev/null 2020-11-16T21:59:13.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj24 /dev/null 2020-11-16T21:59:13.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj25 /dev/null 2020-11-16T21:59:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj26 /dev/null 2020-11-16T21:59:13.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj27 /dev/null 2020-11-16T21:59:13.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj28 /dev/null 2020-11-16T21:59:13.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj29 /dev/null 2020-11-16T21:59:13.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj30 /dev/null 2020-11-16T21:59:13.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:13.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj31 /dev/null 2020-11-16T21:59:14.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj32 /dev/null 2020-11-16T21:59:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj33 /dev/null 2020-11-16T21:59:14.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj34 /dev/null 2020-11-16T21:59:14.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj35 /dev/null 2020-11-16T21:59:14.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj36 /dev/null 2020-11-16T21:59:14.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj37 /dev/null 2020-11-16T21:59:14.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj38 /dev/null 2020-11-16T21:59:14.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj39 /dev/null 2020-11-16T21:59:14.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj40 /dev/null 2020-11-16T21:59:14.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj41 /dev/null 2020-11-16T21:59:14.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj42 /dev/null 2020-11-16T21:59:14.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj43 /dev/null 2020-11-16T21:59:14.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:14.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj44 /dev/null 2020-11-16T21:59:15.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj45 /dev/null 2020-11-16T21:59:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj46 /dev/null 2020-11-16T21:59:15.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj47 /dev/null 2020-11-16T21:59:15.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj48 /dev/null 2020-11-16T21:59:15.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj49 /dev/null 2020-11-16T21:59:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj50 /dev/null 2020-11-16T21:59:15.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj51 /dev/null 2020-11-16T21:59:15.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj52 /dev/null 2020-11-16T21:59:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj53 /dev/null 2020-11-16T21:59:15.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj54 /dev/null 2020-11-16T21:59:15.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj55 /dev/null 2020-11-16T21:59:15.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:15.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj56 /dev/null 2020-11-16T21:59:16.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj57 /dev/null 2020-11-16T21:59:16.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj58 /dev/null 2020-11-16T21:59:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj59 /dev/null 2020-11-16T21:59:16.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj60 /dev/null 2020-11-16T21:59:16.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj61 /dev/null 2020-11-16T21:59:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj62 /dev/null 2020-11-16T21:59:16.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj63 /dev/null 2020-11-16T21:59:16.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj64 /dev/null 2020-11-16T21:59:16.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj65 /dev/null 2020-11-16T21:59:16.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj66 /dev/null 2020-11-16T21:59:16.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj67 /dev/null 2020-11-16T21:59:16.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj68 /dev/null 2020-11-16T21:59:16.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:16.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj69 /dev/null 2020-11-16T21:59:17.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj70 /dev/null 2020-11-16T21:59:17.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj71 /dev/null 2020-11-16T21:59:17.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj72 /dev/null 2020-11-16T21:59:17.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj73 /dev/null 2020-11-16T21:59:17.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj74 /dev/null 2020-11-16T21:59:17.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj75 /dev/null 2020-11-16T21:59:17.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj76 /dev/null 2020-11-16T21:59:17.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj77 /dev/null 2020-11-16T21:59:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj78 /dev/null 2020-11-16T21:59:17.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj79 /dev/null 2020-11-16T21:59:17.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj80 /dev/null 2020-11-16T21:59:17.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj81 /dev/null 2020-11-16T21:59:17.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:17.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj82 /dev/null 2020-11-16T21:59:18.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj83 /dev/null 2020-11-16T21:59:18.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj84 /dev/null 2020-11-16T21:59:18.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj85 /dev/null 2020-11-16T21:59:18.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj86 /dev/null 2020-11-16T21:59:18.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj87 /dev/null 2020-11-16T21:59:18.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj88 /dev/null 2020-11-16T21:59:18.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj89 /dev/null 2020-11-16T21:59:18.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj90 /dev/null 2020-11-16T21:59:18.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj91 /dev/null 2020-11-16T21:59:18.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj92 /dev/null 2020-11-16T21:59:18.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj93 /dev/null 2020-11-16T21:59:18.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj94 /dev/null 2020-11-16T21:59:18.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:18.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj95 /dev/null 2020-11-16T21:59:19.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj96 /dev/null 2020-11-16T21:59:19.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj97 /dev/null 2020-11-16T21:59:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj98 /dev/null 2020-11-16T21:59:19.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj99 /dev/null 2020-11-16T21:59:19.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj100 /dev/null 2020-11-16T21:59:19.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj101 /dev/null 2020-11-16T21:59:19.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj102 /dev/null 2020-11-16T21:59:19.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj103 /dev/null 2020-11-16T21:59:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj104 /dev/null 2020-11-16T21:59:19.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj105 /dev/null 2020-11-16T21:59:19.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj106 /dev/null 2020-11-16T21:59:19.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj107 /dev/null 2020-11-16T21:59:19.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:19.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj108 /dev/null 2020-11-16T21:59:20.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj109 /dev/null 2020-11-16T21:59:20.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj110 /dev/null 2020-11-16T21:59:20.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj111 /dev/null 2020-11-16T21:59:20.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj112 /dev/null 2020-11-16T21:59:20.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj113 /dev/null 2020-11-16T21:59:20.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj114 /dev/null 2020-11-16T21:59:20.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj115 /dev/null 2020-11-16T21:59:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj116 /dev/null 2020-11-16T21:59:20.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj117 /dev/null 2020-11-16T21:59:20.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj118 /dev/null 2020-11-16T21:59:20.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj119 /dev/null 2020-11-16T21:59:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj120 /dev/null 2020-11-16T21:59:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:20.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj121 /dev/null 2020-11-16T21:59:21.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:21.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj122 /dev/null 2020-11-16T21:59:21.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:21.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj123 /dev/null 2020-11-16T21:59:21.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:21.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj124 /dev/null 2020-11-16T21:59:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:21.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj125 /dev/null 2020-11-16T21:59:21.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:21.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj126 /dev/null 2020-11-16T21:59:21.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:21.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj127 /dev/null 2020-11-16T21:59:21.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:21.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj128 /dev/null 2020-11-16T21:59:21.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:21.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj129 /dev/null 2020-11-16T21:59:21.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:21.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj130 /dev/null 2020-11-16T21:59:22.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj131 /dev/null 2020-11-16T21:59:22.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj132 /dev/null 2020-11-16T21:59:22.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj133 /dev/null 2020-11-16T21:59:22.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj134 /dev/null 2020-11-16T21:59:22.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj135 /dev/null 2020-11-16T21:59:22.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj136 /dev/null 2020-11-16T21:59:22.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj137 /dev/null 2020-11-16T21:59:22.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj138 /dev/null 2020-11-16T21:59:22.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj139 /dev/null 2020-11-16T21:59:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj140 /dev/null 2020-11-16T21:59:22.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj141 /dev/null 2020-11-16T21:59:22.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj142 /dev/null 2020-11-16T21:59:22.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:22.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj143 /dev/null 2020-11-16T21:59:23.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj144 /dev/null 2020-11-16T21:59:23.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj145 /dev/null 2020-11-16T21:59:23.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj146 /dev/null 2020-11-16T21:59:23.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj147 /dev/null 2020-11-16T21:59:23.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj148 /dev/null 2020-11-16T21:59:23.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj149 /dev/null 2020-11-16T21:59:23.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj150 /dev/null 2020-11-16T21:59:23.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj151 /dev/null 2020-11-16T21:59:23.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj152 /dev/null 2020-11-16T21:59:23.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj153 /dev/null 2020-11-16T21:59:23.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj154 /dev/null 2020-11-16T21:59:23.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj155 /dev/null 2020-11-16T21:59:23.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:23.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj156 /dev/null 2020-11-16T21:59:24.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj157 /dev/null 2020-11-16T21:59:24.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj158 /dev/null 2020-11-16T21:59:24.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj159 /dev/null 2020-11-16T21:59:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj160 /dev/null 2020-11-16T21:59:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj161 /dev/null 2020-11-16T21:59:24.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj162 /dev/null 2020-11-16T21:59:24.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj163 /dev/null 2020-11-16T21:59:24.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj164 /dev/null 2020-11-16T21:59:24.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj165 /dev/null 2020-11-16T21:59:24.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj166 /dev/null 2020-11-16T21:59:24.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj167 /dev/null 2020-11-16T21:59:24.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:24.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj168 /dev/null 2020-11-16T21:59:25.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj169 /dev/null 2020-11-16T21:59:25.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj170 /dev/null 2020-11-16T21:59:25.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj171 /dev/null 2020-11-16T21:59:25.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj172 /dev/null 2020-11-16T21:59:25.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj173 /dev/null 2020-11-16T21:59:25.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj174 /dev/null 2020-11-16T21:59:25.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj175 /dev/null 2020-11-16T21:59:25.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj176 /dev/null 2020-11-16T21:59:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj177 /dev/null 2020-11-16T21:59:25.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj178 /dev/null 2020-11-16T21:59:25.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj179 /dev/null 2020-11-16T21:59:25.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj180 /dev/null 2020-11-16T21:59:25.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:25.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj181 /dev/null 2020-11-16T21:59:26.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj182 /dev/null 2020-11-16T21:59:26.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj183 /dev/null 2020-11-16T21:59:26.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj184 /dev/null 2020-11-16T21:59:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj185 /dev/null 2020-11-16T21:59:26.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj186 /dev/null 2020-11-16T21:59:26.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj187 /dev/null 2020-11-16T21:59:26.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj188 /dev/null 2020-11-16T21:59:26.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj189 /dev/null 2020-11-16T21:59:26.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj190 /dev/null 2020-11-16T21:59:26.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj191 /dev/null 2020-11-16T21:59:26.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj192 /dev/null 2020-11-16T21:59:26.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:26.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj193 /dev/null 2020-11-16T21:59:27.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:27.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj194 /dev/null 2020-11-16T21:59:27.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:27.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj195 /dev/null 2020-11-16T21:59:27.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:27.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj196 /dev/null 2020-11-16T21:59:27.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:27.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj197 /dev/null 2020-11-16T21:59:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj198 /dev/null 2020-11-16T21:59:27.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:27.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj199 /dev/null 2020-11-16T21:59:27.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2020-11-16T21:59:27.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj200 /dev/null 2020-11-16T21:59:27.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:155: TEST_backfill_sizeup: ceph osd set nobackfill 2020-11-16T21:59:28.446 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T21:59:28.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:156: TEST_backfill_sizeup: ceph osd pool set test size 3 2020-11-16T21:59:29.123 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 3 2020-11-16T21:59:29.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:157: TEST_backfill_sizeup: sleep 2 2020-11-16T21:59:31.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:158: TEST_backfill_sizeup: ceph osd unset nobackfill 2020-11-16T21:59:31.833 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T21:59:31.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:160: TEST_backfill_sizeup: wait_for_clean 2020-11-16T21:59:31.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T21:59:31.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T21:59:31.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T21:59:31.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T21:59:31.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T21:59:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T21:59:31.853 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T21:59:31.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T21:59:31.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T21:59:31.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T21:59:32.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T21:59:32.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T21:59:32.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T21:59:32.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T21:59:32.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T21:59:32.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T21:59:32.434 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T21:59:32.434 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T21:59:32.434 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T21:59:32.435 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T21:59:32.435 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T21:59:32.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T21:59:32.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:32.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T21:59:32.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836498 2020-11-16T21:59:32.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836498 2020-11-16T21:59:32.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498' 2020-11-16T21:59:32.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:32.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T21:59:32.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672977 2020-11-16T21:59:32.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672977 2020-11-16T21:59:32.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672977' 2020-11-16T21:59:32.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:32.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T21:59:32.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542159 2020-11-16T21:59:32.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542159 2020-11-16T21:59:32.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-60129542159' 2020-11-16T21:59:32.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:32.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T21:59:33.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378638 2020-11-16T21:59:33.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378638 2020-11-16T21:59:33.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-60129542159 3-81604378638' 2020-11-16T21:59:33.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:33.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T21:59:33.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215116 2020-11-16T21:59:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215116 2020-11-16T21:59:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-60129542159 3-81604378638 4-103079215116' 2020-11-16T21:59:33.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T21:59:33.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T21:59:33.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051595 2020-11-16T21:59:33.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051595 2020-11-16T21:59:33.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-60129542159 3-81604378638 4-103079215116 5-124554051595' 2020-11-16T21:59:33.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:33.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836498 2020-11-16T21:59:33.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:33.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T21:59:33.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836498 2020-11-16T21:59:33.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:33.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836498 2020-11-16T21:59:33.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2020-11-16T21:59:33.440 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836498 2020-11-16T21:59:33.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:59:33.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836498 2020-11-16T21:59:33.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T21:59:34.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T21:59:34.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T21:59:35.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836498 -lt 21474836498 2020-11-16T21:59:35.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:35.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672977 2020-11-16T21:59:35.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:35.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T21:59:35.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672977 2020-11-16T21:59:35.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:35.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672977 2020-11-16T21:59:35.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2020-11-16T21:59:35.296 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672977 2020-11-16T21:59:35.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T21:59:35.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672978 -lt 42949672977 2020-11-16T21:59:35.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:35.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542159 2020-11-16T21:59:35.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:35.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T21:59:35.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542159 2020-11-16T21:59:35.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:35.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542159 2020-11-16T21:59:35.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542159' 2020-11-16T21:59:35.677 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542159 2020-11-16T21:59:35.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T21:59:36.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542159 -lt 60129542159 2020-11-16T21:59:36.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:36.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378638 2020-11-16T21:59:36.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:36.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T21:59:36.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378638 2020-11-16T21:59:36.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:36.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378638 2020-11-16T21:59:36.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378638' 2020-11-16T21:59:36.043 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378638 2020-11-16T21:59:36.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T21:59:36.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378639 -lt 81604378638 2020-11-16T21:59:36.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:36.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215116 2020-11-16T21:59:36.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:36.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T21:59:36.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215116 2020-11-16T21:59:36.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:36.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215116 2020-11-16T21:59:36.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215116' 2020-11-16T21:59:36.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T21:59:36.478 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215116 2020-11-16T21:59:36.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215116 -lt 103079215116 2020-11-16T21:59:36.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T21:59:36.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051595 2020-11-16T21:59:36.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T21:59:36.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T21:59:36.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051595 2020-11-16T21:59:36.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T21:59:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051595 2020-11-16T21:59:36.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051595' 2020-11-16T21:59:36.864 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 124554051595 2020-11-16T21:59:36.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T21:59:37.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051596 -lt 124554051595 2020-11-16T21:59:37.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T21:59:37.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:37.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:37.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T21:59:37.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:37.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:37.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:37.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:37.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:37.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:37.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:38.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:38.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:38.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:38.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:38.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:38.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T21:59:38.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T21:59:38.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T21:59:38.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:38.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:38.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:38.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:38.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:38.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:38.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:38.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:38.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:38.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:39.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:39.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:39.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:39.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:39.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:39.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:39.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:39.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:39.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:39.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:39.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:39.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:39.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:40.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:59:40.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:59:40.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:40.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:40.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:40.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:40.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:40.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:40.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:40.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:40.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:40.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:40.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:40.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:40.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:40.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:40.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:41.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:41.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:41.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:41.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:41.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:41.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:41.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:41.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:41.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:41.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:59:41.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:59:41.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:41.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:41.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:41.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:41.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:41.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:41.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:41.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:41.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:41.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:41.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:42.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:42.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:42.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:42.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:42.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:42.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:42.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:42.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:42.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:42.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:42.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:42.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:42.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:43.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T21:59:43.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T21:59:43.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:43.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:43.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:43.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:43.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:43.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:43.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:43.145 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:43.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:43.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:43.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:43.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:43.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:43.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:43.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:44.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:44.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:44.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:44.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:44.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:44.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:44.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:44.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:44.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T21:59:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T21:59:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:44.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:44.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:44.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:44.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:44.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:44.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:44.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:44.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:44.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:44.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:44.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:44.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:45.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:45.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:45.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:45.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:45.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:45.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:45.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:45.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:45.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:45.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:59:45.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:59:45.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:45.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:45.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:46.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:46.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:46.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:46.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:46.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:46.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:46.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:46.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:46.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:46.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:46.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:46.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:46.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:46.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:46.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:46.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:46.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:46.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:46.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:47.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:59:47.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:59:47.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:47.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:47.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:47.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:47.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:47.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:47.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:47.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:47.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:47.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:47.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:47.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:47.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:52.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:52.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:52.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:52.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:52.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:52.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:52.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:52.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:52.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:52.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:52.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:52.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:59:52.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:59:52.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:52.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:52.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:52.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:52.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:52.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:52.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:52.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:52.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:52.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:52.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:52.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:52.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:52.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:52.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:52.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:52.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:52.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:52.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:52.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:52.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:52.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:52.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:52.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:52.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:59:52.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:59:52.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:52.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:52.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:52.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:52.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:52.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:52.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:52.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:52.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:52.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:52.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:52.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:52.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:52.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:52.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:52.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:52.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:52.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:52.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:52.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:52.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:52.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:52.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:52.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:52.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:59:52.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:59:52.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:52.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:52.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:52.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:52.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:52.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:52.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:52.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:52.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:52.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:52.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:52.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:52.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:52.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:52.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:52.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:52.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:52.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:52.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:52.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:52.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:52.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:52.708 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:52.708 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:59:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:59:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:53.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:53.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:53.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:53.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:53.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:53.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:53.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:53.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:53.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:53.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:53.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:53.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:54.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:54.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:54.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:54.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:54.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:54.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:54.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:54.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:54.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:54.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T21:59:54.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T21:59:54.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:54.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:54.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:54.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:54.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:54.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:54.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:54.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:54.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:54.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:54.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:55.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:55.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:55.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:55.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:55.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:55.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:55.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:55.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:55.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:55.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:55.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:55.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:55.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:59:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:59:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:56.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:56.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:56.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:56.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:56.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:56.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:56.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:56.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:56.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:56.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:56.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:56.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:56.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:56.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:56.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:56.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:56.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:56.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:56.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:56.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:56.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:59:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:59:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:57.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:57.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:57.737 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:57.737 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:57.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:57.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:57.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:57.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:58.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:58.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:58.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:58.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:58.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:58.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:58.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:58.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:58.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:58.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:58.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:58.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:58.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T21:59:59.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T21:59:59.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T21:59:59.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T21:59:59.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T21:59:59.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T21:59:59.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T21:59:59.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T21:59:59.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T21:59:59.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T21:59:59.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T21:59:59.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T21:59:59.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T21:59:59.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T21:59:59.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T21:59:59.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T21:59:59.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T21:59:59.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T21:59:59.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T21:59:59.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T21:59:59.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T21:59:59.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T21:59:59.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T21:59:59.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T21:59:59.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T21:59:59.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T21:59:59.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:00:00.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T22:00:00.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T22:00:00.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:00:00.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:00:00.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:00:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:00:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:00:00.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:00:00.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:00:00.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:00:00.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:00:00.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:00:00.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:00:00.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:00:00.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:00:00.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:00:00.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:00:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:00:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:00:01.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:00:01.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:00:01.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:00:01.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:00:01.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:00:01.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:00:01.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:00:01.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T22:00:01.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T22:00:01.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:00:01.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:00:01.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:00:02.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:00:02.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:00:02.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:00:02.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:00:02.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:00:02.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:00:02.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:00:02.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:00:02.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:00:02.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:00:02.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:00:02.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:00:02.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:00:02.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:00:02.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:00:02.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:00:02.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:00:02.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:00:02.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:00:02.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:00:02.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:00:03.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T22:00:03.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T22:00:03.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:00:03.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:00:03.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:00:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:00:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:00:03.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:00:03.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:00:03.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:00:03.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:00:03.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:00:03.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:00:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:00:03.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:00:03.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:00:03.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:00:04.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:00:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:00:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:00:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:00:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:00:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:00:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:00:04.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:00:04.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:00:04.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T22:00:04.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T22:00:04.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:00:04.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:00:04.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:00:04.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:00:04.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:00:04.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:00:04.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:00:04.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:00:04.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:00:04.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:00:04.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:00:05.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:00:05.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:00:05.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:00:05.358 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:00:05.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:00:05.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:00:05.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:00:05.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:00:05.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:00:05.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:00:05.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:00:05.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:00:05.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:00:06.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T22:00:06.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T22:00:06.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:00:06.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:00:06.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:00:06.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:00:06.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:00:06.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:00:06.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:00:06.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:00:06.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:00:06.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:00:06.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:00:06.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:00:06.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:00:06.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:00:06.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:00:07.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:00:07.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:00:07.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:00:07.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:00:07.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:00:07.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:00:07.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:00:07.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:00:07.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:00:07.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T22:00:07.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T22:00:07.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:00:07.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:00:07.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:00:07.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:00:07.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:00:07.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:00:07.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:00:07.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:00:07.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:00:07.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:00:07.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:00:08.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:00:08.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:00:08.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:00:08.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:00:08.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:00:08.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:00:08.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:00:08.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:00:08.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:00:08.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:00:08.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:00:08.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:00:08.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:00:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T22:00:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T22:00:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:00:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:00:09.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:00:09.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:00:09.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:00:09.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:00:09.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:00:09.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:00:09.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:00:09.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:00:09.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:00:09.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:00:09.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:00:09.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:00:09.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:00:10.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:00:10.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:00:10.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:00:10.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:00:10.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:00:10.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:00:10.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:00:10.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:00:10.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:00:10.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T22:00:10.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T22:00:10.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:00:10.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:00:10.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:00:10.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:00:10.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:00:10.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:00:10.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:00:10.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:00:10.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:00:10.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:00:10.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:00:11.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:00:11.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:00:11.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:00:11.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:00:11.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:00:11.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:00:11.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:00:11.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:00:11.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:00:11.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:00:11.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:00:11.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:00:11.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:00:12.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:00:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:00:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:00:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:00:12.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:00:12.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:00:12.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:00:12.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:00:12.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:00:12.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:00:12.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:00:12.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:00:12.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:00:12.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:00:12.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:00:12.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:00:12.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:00:13.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:00:13.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:00:13.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:00:13.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:00:13.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:00:13.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:00:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:00:13.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:00:13.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:00:13.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:00:13.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:00:13.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:00:13.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:00:13.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:00:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:00:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:00:13.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:00:13.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:00:13.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:00:13.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:00:13.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:00:13.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:00:14.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:00:14.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:00:14.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:00:14.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:00:14.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T22:00:14.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:00:14.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:00:14.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:162: TEST_backfill_sizeup: get_primary test obj1 2020-11-16T22:00:14.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T22:00:14.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T22:00:14.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T22:00:14.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T22:00:14.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:162: TEST_backfill_sizeup: local primary=1 2020-11-16T22:00:14.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:163: TEST_backfill_sizeup: get_pg test obj1 2020-11-16T22:00:14.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T22:00:14.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T22:00:14.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T22:00:14.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T22:00:15.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:163: TEST_backfill_sizeup: local PG=1.0 2020-11-16T22:00:15.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:165: TEST_backfill_sizeup: expr 200 '*' 2 2020-11-16T22:00:15.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:165: TEST_backfill_sizeup: local degraded=400 2020-11-16T22:00:15.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:166: TEST_backfill_sizeup: check td/osd-backfill-stats 1.0 1 replicated 400 0 0 0 2020-11-16T22:00:15.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2020-11-16T22:00:15.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2020-11-16T22:00:15.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2020-11-16T22:00:15.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2020-11-16T22:00:15.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=400 2020-11-16T22:00:15.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2020-11-16T22:00:15.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=0 2020-11-16T22:00:15.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2020-11-16T22:00:15.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2020-11-16T22:00:15.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2020-11-16T22:00:15.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2020-11-16T22:00:15.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2020-11-16T22:00:15.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2020-11-16T22:00:15.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2020-11-16T22:00:15.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2020-11-16T22:00:15.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2020-11-16T22:00:15.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2020-11-16T22:00:15.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2020-11-16T22:00:15.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2020-11-16T22:00:15.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2020-11-16T22:00:15.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2020-11-16T22:00:15.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2020-11-16T22:00:15.431 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:00:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2020-11-16T22:00:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2020-11-16T22:00:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[1,0,2]/[1]' 2020-11-16T22:00:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[1,0,2]/[1]' ']' 2020-11-16T22:00:15.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2020-11-16T22:00:15.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:00:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [1,0,2]/[1] ' 2020-11-16T22:00:15.457 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:00:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2020-11-16T22:00:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2020-11-16T22:00:15.458 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:00:15.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=400 2020-11-16T22:00:15.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 400 ']' 2020-11-16T22:00:15.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 400 400 2020-11-16T22:00:15.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2020-11-16T22:00:15.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T22:00:15.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2020-11-16T22:00:15.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T22:00:15.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:00:15.467 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:00:15.467 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2020-11-16T22:00:15.468 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:00:15.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=2 2020-11-16T22:00:15.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 2 ']' 2020-11-16T22:00:15.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 2 0 2020-11-16T22:00:15.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=2 2020-11-16T22:00:15.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T22:00:15.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T22:00:15.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T22:00:15.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2020-11-16T22:00:15.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:00:15.484 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:00:15.484 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [1,0,2]/[1] ' 2020-11-16T22:00:15.484 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2020-11-16T22:00:15.484 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2020-11-16T22:00:15.484 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:00:15.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=0 2020-11-16T22:00:15.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 0 ']' 2020-11-16T22:00:15.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 0 0 2020-11-16T22:00:15.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2020-11-16T22:00:15.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T22:00:15.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2020-11-16T22:00:15.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T22:00:15.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:00:15.489 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:00:15.489 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2020-11-16T22:00:15.490 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:00:15.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=0 2020-11-16T22:00:15.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 0 ']' 2020-11-16T22:00:15.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 0 0 2020-11-16T22:00:15.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2020-11-16T22:00:15.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T22:00:15.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T22:00:15.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T22:00:15.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2020-11-16T22:00:15.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:168: TEST_backfill_sizeup: delete_pool test 2020-11-16T22:00:15.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T22:00:15.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T22:00:16.076 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T22:00:16.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:169: TEST_backfill_sizeup: kill_daemons td/osd-backfill-stats 2020-11-16T22:00:16.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:00:16.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:00:16.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:00:16.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:00:16.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:00:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:00:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2020-11-16T22:00:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T22:00:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:00:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T22:00:16.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:00:16.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:00:16.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:00:16.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:00:16.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:00:16.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:00:16.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:00:16.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:00:16.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:00:16.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:00:16.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:00:16.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:00:16.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:00:16.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:00:16.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:00:16.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:00:16.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:00:16.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:00:16.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T22:00:16.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:00:16.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:16.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:16.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T22:00:16.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:00:16.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:00:16.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2020-11-16T22:00:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2020-11-16T22:00:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2020-11-16T22:00:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2020-11-16T22:00:16.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T22:00:16.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:00:16.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T22:00:16.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:00:16.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:00:16.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:00:16.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:00:16.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:00:16.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:00:16.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:00:16.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:00:16.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:00:16.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:00:16.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:00:16.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:00:16.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:00:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:00:16.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:00:16.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:00:16.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:00:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:00:16.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T22:00:16.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:00:16.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:16.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:16.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T22:00:16.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:00:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:00:16.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2020-11-16T22:00:16.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:00:16.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:16.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:16.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.465868 2020-11-16T22:00:16.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:00:16.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T22:00:16.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:00:16.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2020-11-16T22:00:16.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_sizeup4_allout td/osd-backfill-stats 2020-11-16T22:00:16.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:389: TEST_backfill_sizeup4_allout: local dir=td/osd-backfill-stats 2020-11-16T22:00:16.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:391: TEST_backfill_sizeup4_allout: run_mon td/osd-backfill-stats a 2020-11-16T22:00:16.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2020-11-16T22:00:16.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:00:16.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:00:16.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:00:16.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2020-11-16T22:00:16.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2020-11-16T22:00:16.779 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:00:16.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:00:16.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:00:16.781 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:00:16.781 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:16.781 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:16.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:16.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:00:16.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:00:16.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:00:16.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:00:16.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:00:16.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:00:16.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:00:16.997 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:00:16.997 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:00:16.997 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:00:16.998 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:00:16.998 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:16.998 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:16.998 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T22:00:16.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:00:16.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get fsid 2020-11-16T22:00:17.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:00:17.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:00:17.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:00:17.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:00:17.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:00:17.106 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:00:17.106 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:00:17.106 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:00:17.107 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:00:17.107 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:17.107 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:17.108 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T22:00:17.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:00:17.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get mon_host 2020-11-16T22:00:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:392: TEST_backfill_sizeup4_allout: run_mgr td/osd-backfill-stats x 2020-11-16T22:00:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2020-11-16T22:00:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:00:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:00:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:00:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2020-11-16T22:00:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:00:17.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:00:17.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:00:17.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:00:17.730 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:00:17.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:17.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:17.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:17.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:00:17.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:00:17.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:393: TEST_backfill_sizeup4_allout: run_osd td/osd-backfill-stats 0 2020-11-16T22:00:17.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T22:00:17.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:00:17.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:00:17.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:00:17.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2020-11-16T22:00:17.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T22:00:17.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:00:17.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:00:17.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:00:17.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2020-11-16T22:00:17.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2020-11-16T22:00:17.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:00:17.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:00:17.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T22:00:17.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:00:17.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:00:17.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:00:17.767 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:00:17.767 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:17.767 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:17.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:17.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:17.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:00:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:00:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:00:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T22:00:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T22:00:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:00:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:00:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:00:17.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:00:17.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:00:17.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2020-11-16T22:00:17.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:00:17.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1a676e61-1f9e-4e4f-9eaf-06da2a0c41fd 2020-11-16T22:00:17.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 1a676e61-1f9e-4e4f-9eaf-06da2a0c41fd' 2020-11-16T22:00:17.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:00:17.770 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 1a676e61-1f9e-4e4f-9eaf-06da2a0c41fd 2020-11-16T22:00:17.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBx9rJfA3F5LxAAD/MDwkXukaXbHbE535nxAQ== 2020-11-16T22:00:17.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBx9rJfA3F5LxAAD/MDwkXukaXbHbE535nxAQ=="}' 2020-11-16T22:00:17.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1a676e61-1f9e-4e4f-9eaf-06da2a0c41fd -i td/osd-backfill-stats/0/new.json 2020-11-16T22:00:18.123 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:00:18.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2020-11-16T22:00:18.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBx9rJfA3F5LxAAD/MDwkXukaXbHbE535nxAQ== --osd-uuid 1a676e61-1f9e-4e4f-9eaf-06da2a0c41fd 2020-11-16T22:00:18.166 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:18.173+0000 7f4181473d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:18.168 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:18.173+0000 7f4181473d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:18.170 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:18.177+0000 7f4181473d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:18.171 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:18.177+0000 7f4181473d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T22:00:18.171 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:18.177+0000 7f4181473d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T22:00:18.171 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:18.177+0000 7f4181473d80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T22:00:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:18.245+0000 7f4181473d80 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2020-11-16T22:00:22.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2020-11-16T22:00:22.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:00:22.596 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:00:22.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:00:22.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:00:23.074 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:00:23.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:00:23.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:00:23.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:00:23.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:00:23.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:00:23.105 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:23.113+0000 7ff0e43b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:23.107 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:23.113+0000 7ff0e43b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:23.109 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:23.117+0000 7ff0e43b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:23.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:00:23.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:00:23.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:00:23.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:00:23.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:00:23.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:23.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:00:23.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:23.461 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:00:23.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:00:23.654 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:23.661+0000 7ff0e43b1d80 -1 Falling back to public interface 2020-11-16T22:00:23.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:00:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:24.489+0000 7ff0e43b1d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:00:24.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:00:24.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:24.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:00:24.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:24.856 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:00:24.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:00:25.260 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/584662,v1:127.0.0.1:6803/584662] [v2:127.0.0.1:6804/584662,v1:127.0.0.1:6805/584662] exists,up 1a676e61-1f9e-4e4f-9eaf-06da2a0c41fd 2020-11-16T22:00:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:00:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:00:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:00:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:394: TEST_backfill_sizeup4_allout: run_osd td/osd-backfill-stats 1 2020-11-16T22:00:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T22:00:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:00:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:00:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:00:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2020-11-16T22:00:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T22:00:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:00:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:00:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:00:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2020-11-16T22:00:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2020-11-16T22:00:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:00:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:00:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T22:00:25.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:00:25.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:00:25.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:00:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:00:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:25.267 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:00:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:00:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:00:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T22:00:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T22:00:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:00:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:00:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:00:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:00:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:00:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2020-11-16T22:00:25.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:00:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6fc5ad7a-fe5e-4cb4-9f9d-98aac4262a7a 2020-11-16T22:00:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 6fc5ad7a-fe5e-4cb4-9f9d-98aac4262a7a' 2020-11-16T22:00:25.269 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 6fc5ad7a-fe5e-4cb4-9f9d-98aac4262a7a 2020-11-16T22:00:25.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:00:25.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB59rJf+5j8ERAA9tIEel5YSmorzLkC6t0Xsw== 2020-11-16T22:00:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB59rJf+5j8ERAA9tIEel5YSmorzLkC6t0Xsw=="}' 2020-11-16T22:00:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6fc5ad7a-fe5e-4cb4-9f9d-98aac4262a7a -i td/osd-backfill-stats/1/new.json 2020-11-16T22:00:25.755 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:00:25.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2020-11-16T22:00:25.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB59rJf+5j8ERAA9tIEel5YSmorzLkC6t0Xsw== --osd-uuid 6fc5ad7a-fe5e-4cb4-9f9d-98aac4262a7a 2020-11-16T22:00:25.798 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:25.805+0000 7f243a5fad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:25.802 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:25.809+0000 7f243a5fad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:25.804 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:25.809+0000 7f243a5fad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:25.804 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:25.809+0000 7f243a5fad80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T22:00:25.804 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:25.809+0000 7f243a5fad80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T22:00:25.804 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:25.809+0000 7f243a5fad80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T22:00:25.897 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:25.905+0000 7f243a5fad80 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2020-11-16T22:00:29.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2020-11-16T22:00:29.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:00:29.971 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:00:29.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:00:29.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:00:30.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:00:30.463 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:00:30.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:00:30.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:00:30.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:00:30.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:00:30.493 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:30.501+0000 7f93aa257d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:30.495 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:30.501+0000 7f93aa257d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:30.497 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:30.505+0000 7f93aa257d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:30.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:00:30.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:00:30.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:00:30.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:00:30.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:00:30.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:30.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:00:30.827 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:00:30.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:30.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:00:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:31.045+0000 7f93aa257d80 -1 Falling back to public interface 2020-11-16T22:00:31.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:00:31.929 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:31.933+0000 7f93aa257d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:00:32.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:00:32.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:32.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:00:32.221 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:00:32.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:32.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:00:32.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:00:33.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:00:33.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:33.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:00:33.641 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:00:33.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:33.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:00:34.039 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/585656,v1:127.0.0.1:6811/585656] [v2:127.0.0.1:6812/585656,v1:127.0.0.1:6813/585656] exists,up 6fc5ad7a-fe5e-4cb4-9f9d-98aac4262a7a 2020-11-16T22:00:34.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:00:34.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:00:34.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:00:34.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:395: TEST_backfill_sizeup4_allout: run_osd td/osd-backfill-stats 2 2020-11-16T22:00:34.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T22:00:34.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:00:34.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T22:00:34.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:00:34.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2020-11-16T22:00:34.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T22:00:34.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:00:34.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:00:34.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:00:34.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2020-11-16T22:00:34.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2020-11-16T22:00:34.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:00:34.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:00:34.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T22:00:34.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:00:34.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:00:34.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:00:34.044 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:00:34.045 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:34.045 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:34.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:34.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:34.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:00:34.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:00:34.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:00:34.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T22:00:34.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T22:00:34.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:00:34.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:00:34.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:00:34.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:00:34.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:00:34.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2020-11-16T22:00:34.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:00:34.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=51e11e09-7250-4380-8452-6b0602b0cd0c 2020-11-16T22:00:34.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 51e11e09-7250-4380-8452-6b0602b0cd0c' 2020-11-16T22:00:34.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:00:34.048 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 51e11e09-7250-4380-8452-6b0602b0cd0c 2020-11-16T22:00:34.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCC9rJf2oSOBBAA5oK94EOlhGEOxxLQalOXSQ== 2020-11-16T22:00:34.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCC9rJf2oSOBBAA5oK94EOlhGEOxxLQalOXSQ=="}' 2020-11-16T22:00:34.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 51e11e09-7250-4380-8452-6b0602b0cd0c -i td/osd-backfill-stats/2/new.json 2020-11-16T22:00:34.531 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:00:34.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2020-11-16T22:00:34.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCC9rJf2oSOBBAA5oK94EOlhGEOxxLQalOXSQ== --osd-uuid 51e11e09-7250-4380-8452-6b0602b0cd0c 2020-11-16T22:00:34.574 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:34.581+0000 7feb7fcd4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:34.577 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:34.585+0000 7feb7fcd4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:34.579 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:34.585+0000 7feb7fcd4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:34.579 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:34.585+0000 7feb7fcd4d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T22:00:34.579 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:34.585+0000 7feb7fcd4d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T22:00:34.579 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:34.585+0000 7feb7fcd4d80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T22:00:34.668 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:34.673+0000 7feb7fcd4d80 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2020-11-16T22:00:39.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2020-11-16T22:00:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:00:39.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T22:00:39.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:00:39.042 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T22:00:39.555 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:00:39.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T22:00:39.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:00:39.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:00:39.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:00:39.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:00:39.585 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:39.593+0000 7f2949883d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:39.588 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:39.593+0000 7f2949883d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:39.589 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:39.597+0000 7f2949883d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:39.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T22:00:39.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:00:39.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:00:39.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:00:39.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:00:39.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:39.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:00:39.964 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:00:39.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:39.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:00:40.147 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:40.153+0000 7f2949883d80 -1 Falling back to public interface 2020-11-16T22:00:40.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:00:40.966 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:40.973+0000 7f2949883d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T22:00:41.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:00:41.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:41.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:00:41.355 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:00:41.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:41.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:00:41.743 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/586676,v1:127.0.0.1:6819/586676] [v2:127.0.0.1:6820/586676,v1:127.0.0.1:6821/586676] exists,up 51e11e09-7250-4380-8452-6b0602b0cd0c 2020-11-16T22:00:41.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:00:41.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:00:41.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:00:41.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:396: TEST_backfill_sizeup4_allout: run_osd td/osd-backfill-stats 3 2020-11-16T22:00:41.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T22:00:41.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:00:41.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T22:00:41.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:00:41.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2020-11-16T22:00:41.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T22:00:41.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:00:41.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:00:41.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:00:41.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2020-11-16T22:00:41.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2020-11-16T22:00:41.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:00:41.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:00:41.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T22:00:41.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:00:41.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:00:41.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:00:41.748 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:00:41.748 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:41.748 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:41.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:41.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:41.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:00:41.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:00:41.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:00:41.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T22:00:41.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T22:00:41.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:00:41.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:00:41.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:00:41.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:00:41.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:00:41.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2020-11-16T22:00:41.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:00:41.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2843a4cd-f013-44cd-aecc-a532fd3f11e7 2020-11-16T22:00:41.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 2843a4cd-f013-44cd-aecc-a532fd3f11e7' 2020-11-16T22:00:41.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:00:41.752 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 2843a4cd-f013-44cd-aecc-a532fd3f11e7 2020-11-16T22:00:41.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCJ9rJflL58LhAAR0ZnQSpeg/dMOA+zfWVBwQ== 2020-11-16T22:00:41.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCJ9rJflL58LhAAR0ZnQSpeg/dMOA+zfWVBwQ=="}' 2020-11-16T22:00:41.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2843a4cd-f013-44cd-aecc-a532fd3f11e7 -i td/osd-backfill-stats/3/new.json 2020-11-16T22:00:42.268 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:00:42.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2020-11-16T22:00:42.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCJ9rJflL58LhAAR0ZnQSpeg/dMOA+zfWVBwQ== --osd-uuid 2843a4cd-f013-44cd-aecc-a532fd3f11e7 2020-11-16T22:00:42.308 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:42.313+0000 7f90718fbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:42.310 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:42.317+0000 7f90718fbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:42.311 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:42.317+0000 7f90718fbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:42.317+0000 7f90718fbd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T22:00:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:42.317+0000 7f90718fbd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T22:00:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:42.317+0000 7f90718fbd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T22:00:42.399 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:42.405+0000 7f90718fbd80 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2020-11-16T22:00:46.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2020-11-16T22:00:46.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:00:46.996 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T22:00:46.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T22:00:46.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:00:47.498 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T22:00:47.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T22:00:47.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:00:47.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:00:47.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:00:47.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:00:47.526 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:47.532+0000 7f056ea67d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:47.530 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:47.536+0000 7f056ea67d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:47.531 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:47.536+0000 7f056ea67d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:47.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T22:00:47.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:00:47.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T22:00:47.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:00:47.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:00:47.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:47.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:00:47.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:47.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:00:47.882 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:00:48.075 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:48.080+0000 7f056ea67d80 -1 Falling back to public interface 2020-11-16T22:00:48.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:00:49.055 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:49.060+0000 7f056ea67d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T22:00:49.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:00:49.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:49.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:00:49.282 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:00:49.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:49.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:00:49.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:00:50.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:00:50.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:50.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:00:50.729 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:00:50.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:50.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:00:51.129 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/587734,v1:127.0.0.1:6827/587734] [v2:127.0.0.1:6828/587734,v1:127.0.0.1:6829/587734] exists,up 2843a4cd-f013-44cd-aecc-a532fd3f11e7 2020-11-16T22:00:51.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:00:51.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:00:51.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:00:51.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:397: TEST_backfill_sizeup4_allout: run_osd td/osd-backfill-stats 4 2020-11-16T22:00:51.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T22:00:51.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:00:51.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T22:00:51.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:00:51.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2020-11-16T22:00:51.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T22:00:51.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:00:51.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:00:51.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:00:51.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2020-11-16T22:00:51.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2020-11-16T22:00:51.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:00:51.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:00:51.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T22:00:51.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:00:51.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:00:51.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:00:51.135 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:00:51.135 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:00:51.135 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:00:51.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:51.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:00:51.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:00:51.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:00:51.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:00:51.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T22:00:51.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T22:00:51.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:00:51.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:00:51.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:00:51.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:00:51.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:00:51.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2020-11-16T22:00:51.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:00:51.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2b6969f5-535a-4aa1-a5ce-7219e510a8dd 2020-11-16T22:00:51.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 2b6969f5-535a-4aa1-a5ce-7219e510a8dd' 2020-11-16T22:00:51.137 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 2b6969f5-535a-4aa1-a5ce-7219e510a8dd 2020-11-16T22:00:51.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:00:51.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCT9rJf0KjqCRAAIox/dX4IY7UziZF4DLt4qA== 2020-11-16T22:00:51.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCT9rJf0KjqCRAAIox/dX4IY7UziZF4DLt4qA=="}' 2020-11-16T22:00:51.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2b6969f5-535a-4aa1-a5ce-7219e510a8dd -i td/osd-backfill-stats/4/new.json 2020-11-16T22:00:51.656 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T22:00:51.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2020-11-16T22:00:51.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCT9rJf0KjqCRAAIox/dX4IY7UziZF4DLt4qA== --osd-uuid 2b6969f5-535a-4aa1-a5ce-7219e510a8dd 2020-11-16T22:00:51.701 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:51.708+0000 7fd60fa62d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:51.703 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:51.708+0000 7fd60fa62d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:51.704 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:51.708+0000 7fd60fa62d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:51.705 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:51.712+0000 7fd60fa62d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T22:00:51.705 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:51.712+0000 7fd60fa62d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T22:00:51.705 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:51.712+0000 7fd60fa62d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T22:00:51.764 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:51.768+0000 7fd60fa62d80 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2020-11-16T22:00:55.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2020-11-16T22:00:55.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:00:55.796 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T22:00:55.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T22:00:55.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:00:56.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T22:00:56.288 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T22:00:56.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:00:56.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:00:56.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:00:56.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:00:56.318 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:56.324+0000 7f0831f5dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:56.319 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:56.324+0000 7f0831f5dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:56.321 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:56.328+0000 7f0831f5dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:00:56.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T22:00:56.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:00:56.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T22:00:56.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:00:56.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:00:56.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:56.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:00:56.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:56.676 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:00:56.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T22:00:56.862 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:56.868+0000 7f0831f5dd80 -1 Falling back to public interface 2020-11-16T22:00:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:00:57.748 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:00:57.744+0000 7f0831f5dd80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T22:00:58.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:00:58.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:00:58.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:00:58.054 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:00:58.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:00:58.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T22:00:58.432 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/588755,v1:127.0.0.1:6835/588755] [v2:127.0.0.1:6836/588755,v1:127.0.0.1:6837/588755] exists,up 2b6969f5-535a-4aa1-a5ce-7219e510a8dd 2020-11-16T22:00:58.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:00:58.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:00:58.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:00:58.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:399: TEST_backfill_sizeup4_allout: create_pool test 1 1 2020-11-16T22:00:58.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T22:00:58.982 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T22:00:58.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:00:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:400: TEST_backfill_sizeup4_allout: ceph osd pool set test size 2 2020-11-16T22:01:00.744 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T22:01:00.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:402: TEST_backfill_sizeup4_allout: wait_for_clean 2020-11-16T22:01:00.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:01:00.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:01:00.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:01:00.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:01:00.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:01:00.755 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:01:00.756 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:01:00.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:01:00.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:01:00.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:01:00.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:01:00.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:01:00.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:01:00.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:01:00.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:01:01.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:01:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:01:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:01:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:01:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T22:01:01.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:01:01.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:01:01.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:01:01.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2020-11-16T22:01:01.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2020-11-16T22:01:01.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2020-11-16T22:01:01.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:01:01.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:01:01.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705672 2020-11-16T22:01:01.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705672 2020-11-16T22:01:01.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-38654705672' 2020-11-16T22:01:01.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:01:01.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:01:01.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542150 2020-11-16T22:01:01.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542150 2020-11-16T22:01:01.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-38654705672 2-60129542150' 2020-11-16T22:01:01.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:01:01.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:01:02.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378629 2020-11-16T22:01:02.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378629 2020-11-16T22:01:02.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-38654705672 2-60129542150 3-81604378629' 2020-11-16T22:01:02.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:01:02.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:01:02.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215107 2020-11-16T22:01:02.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215107 2020-11-16T22:01:02.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-38654705672 2-60129542150 3-81604378629 4-103079215107' 2020-11-16T22:01:02.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:01:02.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2020-11-16T22:01:02.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:01:02.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:01:02.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2020-11-16T22:01:02.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:01:02.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2020-11-16T22:01:02.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-11-16T22:01:02.296 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836490 2020-11-16T22:01:02.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:01:02.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836490 2020-11-16T22:01:02.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:01:03.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:01:03.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:01:04.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2020-11-16T22:01:04.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:01:04.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705672 2020-11-16T22:01:04.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:01:04.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:01:04.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705672 2020-11-16T22:01:04.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:01:04.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705672 2020-11-16T22:01:04.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2020-11-16T22:01:04.140 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705672 2020-11-16T22:01:04.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:01:04.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705673 -lt 38654705672 2020-11-16T22:01:04.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:01:04.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542150 2020-11-16T22:01:04.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:01:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:01:04.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542150 2020-11-16T22:01:04.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:01:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542150 2020-11-16T22:01:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2020-11-16T22:01:04.557 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542150 2020-11-16T22:01:04.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:01:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542151 -lt 60129542150 2020-11-16T22:01:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:01:04.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:01:04.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378629 2020-11-16T22:01:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:01:04.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378629 2020-11-16T22:01:04.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:01:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378629 2020-11-16T22:01:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378629' 2020-11-16T22:01:04.965 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378629 2020-11-16T22:01:04.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:01:05.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378629 -lt 81604378629 2020-11-16T22:01:05.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:01:05.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215107 2020-11-16T22:01:05.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:01:05.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:01:05.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:01:05.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215107 2020-11-16T22:01:05.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215107 2020-11-16T22:01:05.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215107' 2020-11-16T22:01:05.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:01:05.350 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215107 2020-11-16T22:01:05.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215108 -lt 103079215107 2020-11-16T22:01:05.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:01:05.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:05.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:06.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T22:01:06.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:06.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:06.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:06.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:06.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:06.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:06.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:06.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:01:06.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:06.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:06.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T22:01:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:01:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:01:07.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: seq 1 200 2020-11-16T22:01:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj1 /dev/null 2020-11-16T22:01:07.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:07.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj2 /dev/null 2020-11-16T22:01:07.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:07.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj3 /dev/null 2020-11-16T22:01:07.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:07.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj4 /dev/null 2020-11-16T22:01:07.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:07.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj5 /dev/null 2020-11-16T22:01:07.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:07.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj6 /dev/null 2020-11-16T22:01:07.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:07.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj7 /dev/null 2020-11-16T22:01:07.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:07.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj8 /dev/null 2020-11-16T22:01:07.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:07.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj9 /dev/null 2020-11-16T22:01:08.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:08.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj10 /dev/null 2020-11-16T22:01:08.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:08.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj11 /dev/null 2020-11-16T22:01:08.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:08.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj12 /dev/null 2020-11-16T22:01:08.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:08.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj13 /dev/null 2020-11-16T22:01:08.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:08.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj14 /dev/null 2020-11-16T22:01:08.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:08.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj15 /dev/null 2020-11-16T22:01:08.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:08.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj16 /dev/null 2020-11-16T22:01:08.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:08.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj17 /dev/null 2020-11-16T22:01:08.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:08.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj18 /dev/null 2020-11-16T22:01:08.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:08.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj19 /dev/null 2020-11-16T22:01:09.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:09.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj20 /dev/null 2020-11-16T22:01:09.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:09.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj21 /dev/null 2020-11-16T22:01:09.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj22 /dev/null 2020-11-16T22:01:09.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:09.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj23 /dev/null 2020-11-16T22:01:09.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:09.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj24 /dev/null 2020-11-16T22:01:09.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:09.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj25 /dev/null 2020-11-16T22:01:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj26 /dev/null 2020-11-16T22:01:09.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:09.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj27 /dev/null 2020-11-16T22:01:09.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:09.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj28 /dev/null 2020-11-16T22:01:09.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:09.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj29 /dev/null 2020-11-16T22:01:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:10.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj30 /dev/null 2020-11-16T22:01:10.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:10.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj31 /dev/null 2020-11-16T22:01:10.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:10.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj32 /dev/null 2020-11-16T22:01:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj33 /dev/null 2020-11-16T22:01:10.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:10.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj34 /dev/null 2020-11-16T22:01:10.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:10.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj35 /dev/null 2020-11-16T22:01:10.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:10.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj36 /dev/null 2020-11-16T22:01:10.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj37 /dev/null 2020-11-16T22:01:10.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:10.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj38 /dev/null 2020-11-16T22:01:10.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:10.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj39 /dev/null 2020-11-16T22:01:11.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:11.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj40 /dev/null 2020-11-16T22:01:11.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:11.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj41 /dev/null 2020-11-16T22:01:11.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:11.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj42 /dev/null 2020-11-16T22:01:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:11.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj43 /dev/null 2020-11-16T22:01:11.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:11.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj44 /dev/null 2020-11-16T22:01:11.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:11.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj45 /dev/null 2020-11-16T22:01:11.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:11.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj46 /dev/null 2020-11-16T22:01:11.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:11.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj47 /dev/null 2020-11-16T22:01:11.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:11.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj48 /dev/null 2020-11-16T22:01:11.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:11.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj49 /dev/null 2020-11-16T22:01:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj50 /dev/null 2020-11-16T22:01:12.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:12.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj51 /dev/null 2020-11-16T22:01:12.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:12.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj52 /dev/null 2020-11-16T22:01:12.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:12.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj53 /dev/null 2020-11-16T22:01:12.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:12.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj54 /dev/null 2020-11-16T22:01:12.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:12.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj55 /dev/null 2020-11-16T22:01:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj56 /dev/null 2020-11-16T22:01:12.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:12.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj57 /dev/null 2020-11-16T22:01:12.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:12.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj58 /dev/null 2020-11-16T22:01:13.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:13.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj59 /dev/null 2020-11-16T22:01:13.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:13.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj60 /dev/null 2020-11-16T22:01:13.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:13.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj61 /dev/null 2020-11-16T22:01:13.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:13.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj62 /dev/null 2020-11-16T22:01:13.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:13.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj63 /dev/null 2020-11-16T22:01:13.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:13.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj64 /dev/null 2020-11-16T22:01:13.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:13.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj65 /dev/null 2020-11-16T22:01:13.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:13.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj66 /dev/null 2020-11-16T22:01:13.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:13.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj67 /dev/null 2020-11-16T22:01:13.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:13.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj68 /dev/null 2020-11-16T22:01:14.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:14.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj69 /dev/null 2020-11-16T22:01:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:14.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj70 /dev/null 2020-11-16T22:01:14.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:14.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj71 /dev/null 2020-11-16T22:01:14.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:14.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj72 /dev/null 2020-11-16T22:01:14.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:14.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj73 /dev/null 2020-11-16T22:01:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj74 /dev/null 2020-11-16T22:01:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj75 /dev/null 2020-11-16T22:01:14.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:14.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj76 /dev/null 2020-11-16T22:01:14.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:14.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj77 /dev/null 2020-11-16T22:01:15.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:15.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj78 /dev/null 2020-11-16T22:01:15.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:15.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj79 /dev/null 2020-11-16T22:01:15.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:15.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj80 /dev/null 2020-11-16T22:01:15.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:15.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj81 /dev/null 2020-11-16T22:01:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:15.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj82 /dev/null 2020-11-16T22:01:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj83 /dev/null 2020-11-16T22:01:15.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:15.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj84 /dev/null 2020-11-16T22:01:15.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:15.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj85 /dev/null 2020-11-16T22:01:15.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:15.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj86 /dev/null 2020-11-16T22:01:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj87 /dev/null 2020-11-16T22:01:16.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:16.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj88 /dev/null 2020-11-16T22:01:16.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:16.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj89 /dev/null 2020-11-16T22:01:16.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:16.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj90 /dev/null 2020-11-16T22:01:16.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:16.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj91 /dev/null 2020-11-16T22:01:16.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:16.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj92 /dev/null 2020-11-16T22:01:16.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:16.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj93 /dev/null 2020-11-16T22:01:16.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:16.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj94 /dev/null 2020-11-16T22:01:16.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:16.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj95 /dev/null 2020-11-16T22:01:16.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:16.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj96 /dev/null 2020-11-16T22:01:16.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:16.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj97 /dev/null 2020-11-16T22:01:17.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:17.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj98 /dev/null 2020-11-16T22:01:17.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:17.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj99 /dev/null 2020-11-16T22:01:17.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj100 /dev/null 2020-11-16T22:01:17.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:17.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj101 /dev/null 2020-11-16T22:01:17.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:17.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj102 /dev/null 2020-11-16T22:01:17.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:17.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj103 /dev/null 2020-11-16T22:01:17.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:17.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj104 /dev/null 2020-11-16T22:01:17.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:17.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj105 /dev/null 2020-11-16T22:01:17.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:17.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj106 /dev/null 2020-11-16T22:01:18.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:18.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj107 /dev/null 2020-11-16T22:01:18.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:18.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj108 /dev/null 2020-11-16T22:01:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj109 /dev/null 2020-11-16T22:01:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj110 /dev/null 2020-11-16T22:01:18.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:18.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj111 /dev/null 2020-11-16T22:01:18.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:18.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj112 /dev/null 2020-11-16T22:01:18.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:18.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj113 /dev/null 2020-11-16T22:01:18.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:18.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj114 /dev/null 2020-11-16T22:01:18.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:18.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj115 /dev/null 2020-11-16T22:01:19.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:19.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj116 /dev/null 2020-11-16T22:01:19.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:19.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj117 /dev/null 2020-11-16T22:01:19.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:19.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj118 /dev/null 2020-11-16T22:01:19.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:19.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj119 /dev/null 2020-11-16T22:01:19.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:19.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj120 /dev/null 2020-11-16T22:01:19.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:19.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj121 /dev/null 2020-11-16T22:01:19.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:19.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj122 /dev/null 2020-11-16T22:01:19.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:19.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj123 /dev/null 2020-11-16T22:01:20.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:20.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj124 /dev/null 2020-11-16T22:01:20.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:20.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj125 /dev/null 2020-11-16T22:01:20.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:20.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj126 /dev/null 2020-11-16T22:01:20.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:20.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj127 /dev/null 2020-11-16T22:01:20.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:20.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj128 /dev/null 2020-11-16T22:01:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:20.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj129 /dev/null 2020-11-16T22:01:20.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:20.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj130 /dev/null 2020-11-16T22:01:20.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:20.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj131 /dev/null 2020-11-16T22:01:20.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:20.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj132 /dev/null 2020-11-16T22:01:20.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:20.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj133 /dev/null 2020-11-16T22:01:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj134 /dev/null 2020-11-16T22:01:21.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:21.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj135 /dev/null 2020-11-16T22:01:21.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:21.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj136 /dev/null 2020-11-16T22:01:21.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:21.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj137 /dev/null 2020-11-16T22:01:21.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:21.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj138 /dev/null 2020-11-16T22:01:21.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:21.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj139 /dev/null 2020-11-16T22:01:21.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj140 /dev/null 2020-11-16T22:01:21.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:21.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj141 /dev/null 2020-11-16T22:01:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:21.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj142 /dev/null 2020-11-16T22:01:22.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:22.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj143 /dev/null 2020-11-16T22:01:22.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:22.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj144 /dev/null 2020-11-16T22:01:22.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:22.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj145 /dev/null 2020-11-16T22:01:22.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:22.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj146 /dev/null 2020-11-16T22:01:22.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:22.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj147 /dev/null 2020-11-16T22:01:22.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:22.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj148 /dev/null 2020-11-16T22:01:22.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:22.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj149 /dev/null 2020-11-16T22:01:22.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:22.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj150 /dev/null 2020-11-16T22:01:22.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:22.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj151 /dev/null 2020-11-16T22:01:22.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:22.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj152 /dev/null 2020-11-16T22:01:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj153 /dev/null 2020-11-16T22:01:23.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:23.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj154 /dev/null 2020-11-16T22:01:23.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:23.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj155 /dev/null 2020-11-16T22:01:23.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:23.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj156 /dev/null 2020-11-16T22:01:23.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:23.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj157 /dev/null 2020-11-16T22:01:23.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:23.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj158 /dev/null 2020-11-16T22:01:23.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:23.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj159 /dev/null 2020-11-16T22:01:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj160 /dev/null 2020-11-16T22:01:23.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:23.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj161 /dev/null 2020-11-16T22:01:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj162 /dev/null 2020-11-16T22:01:24.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:24.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj163 /dev/null 2020-11-16T22:01:24.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:24.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj164 /dev/null 2020-11-16T22:01:24.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:24.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj165 /dev/null 2020-11-16T22:01:24.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:24.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj166 /dev/null 2020-11-16T22:01:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj167 /dev/null 2020-11-16T22:01:24.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:24.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj168 /dev/null 2020-11-16T22:01:24.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:24.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj169 /dev/null 2020-11-16T22:01:24.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:24.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj170 /dev/null 2020-11-16T22:01:24.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:24.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj171 /dev/null 2020-11-16T22:01:24.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:24.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj172 /dev/null 2020-11-16T22:01:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj173 /dev/null 2020-11-16T22:01:25.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:25.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj174 /dev/null 2020-11-16T22:01:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj175 /dev/null 2020-11-16T22:01:25.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:25.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj176 /dev/null 2020-11-16T22:01:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj177 /dev/null 2020-11-16T22:01:25.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:25.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj178 /dev/null 2020-11-16T22:01:25.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:25.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj179 /dev/null 2020-11-16T22:01:25.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:25.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj180 /dev/null 2020-11-16T22:01:25.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:25.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj181 /dev/null 2020-11-16T22:01:26.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:26.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj182 /dev/null 2020-11-16T22:01:26.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:26.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj183 /dev/null 2020-11-16T22:01:26.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:26.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj184 /dev/null 2020-11-16T22:01:26.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:26.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj185 /dev/null 2020-11-16T22:01:26.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:26.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj186 /dev/null 2020-11-16T22:01:26.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:26.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj187 /dev/null 2020-11-16T22:01:26.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:26.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj188 /dev/null 2020-11-16T22:01:26.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:26.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj189 /dev/null 2020-11-16T22:01:26.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:26.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj190 /dev/null 2020-11-16T22:01:26.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:26.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj191 /dev/null 2020-11-16T22:01:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj192 /dev/null 2020-11-16T22:01:27.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:27.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj193 /dev/null 2020-11-16T22:01:27.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:27.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj194 /dev/null 2020-11-16T22:01:27.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:27.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj195 /dev/null 2020-11-16T22:01:27.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:27.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj196 /dev/null 2020-11-16T22:01:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj197 /dev/null 2020-11-16T22:01:27.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:27.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj198 /dev/null 2020-11-16T22:01:27.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:27.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj199 /dev/null 2020-11-16T22:01:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2020-11-16T22:01:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj200 /dev/null 2020-11-16T22:01:28.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:409: TEST_backfill_sizeup4_allout: get_pg test obj1 2020-11-16T22:01:28.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T22:01:28.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T22:01:28.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T22:01:28.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T22:01:28.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:409: TEST_backfill_sizeup4_allout: local PG=1.0 2020-11-16T22:01:28.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:411: TEST_backfill_sizeup4_allout: get_primary test obj1 2020-11-16T22:01:28.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T22:01:28.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T22:01:28.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T22:01:28.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T22:01:28.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:411: TEST_backfill_sizeup4_allout: local primary=1 2020-11-16T22:01:28.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:412: TEST_backfill_sizeup4_allout: get_not_primary test obj1 2020-11-16T22:01:28.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2020-11-16T22:01:28.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2020-11-16T22:01:28.903 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2020-11-16T22:01:28.903 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T22:01:28.903 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T22:01:28.903 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T22:01:28.904 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T22:01:29.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T22:01:29.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2020-11-16T22:01:29.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T22:01:29.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:412: TEST_backfill_sizeup4_allout: local otherosd=0 2020-11-16T22:01:29.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:414: TEST_backfill_sizeup4_allout: ceph osd set nobackfill 2020-11-16T22:01:30.377 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T22:01:30.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:415: TEST_backfill_sizeup4_allout: ceph osd out osd.0 2020-11-16T22:01:31.018 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T22:01:31.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:416: TEST_backfill_sizeup4_allout: ceph osd out osd.1 2020-11-16T22:01:31.809 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. 2020-11-16T22:01:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:417: TEST_backfill_sizeup4_allout: ceph osd pool set test size 4 2020-11-16T22:01:32.748 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 4 2020-11-16T22:01:32.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:419: TEST_backfill_sizeup4_allout: sleep 2 2020-11-16T22:01:34.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:420: TEST_backfill_sizeup4_allout: get_primary test obj1 2020-11-16T22:01:34.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T22:01:34.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T22:01:34.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T22:01:34.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T22:01:35.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:420: TEST_backfill_sizeup4_allout: primary=1 2020-11-16T22:01:35.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:421: TEST_backfill_sizeup4_allout: ceph osd unset nobackfill 2020-11-16T22:01:35.838 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T22:01:35.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:422: TEST_backfill_sizeup4_allout: ceph tell osd.1 get_latest_osdmap 2020-11-16T22:01:36.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:423: TEST_backfill_sizeup4_allout: ceph tell osd.1 debug kick_recovery_wq 0 2020-11-16T22:01:36.143 INFO:tasks.workunit.client.0.smithi189.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-11-16T22:01:36.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:424: TEST_backfill_sizeup4_allout: sleep 2 2020-11-16T22:01:38.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:426: TEST_backfill_sizeup4_allout: wait_for_clean 2020-11-16T22:01:38.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:01:38.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:01:38.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:01:38.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:01:38.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:01:38.156 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:01:38.156 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:01:38.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:01:38.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:01:38.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:01:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:01:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:01:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:01:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:01:38.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:01:38.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:01:38.760 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:01:38.761 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:01:38.761 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:01:38.761 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T22:01:38.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:01:38.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:01:38.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:01:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836499 2020-11-16T22:01:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836499 2020-11-16T22:01:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499' 2020-11-16T22:01:38.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:01:38.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:01:39.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705682 2020-11-16T22:01:39.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705682 2020-11-16T22:01:39.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-38654705682' 2020-11-16T22:01:39.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:01:39.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:01:39.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542160 2020-11-16T22:01:39.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542160 2020-11-16T22:01:39.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-60129542160' 2020-11-16T22:01:39.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:01:39.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:01:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378638 2020-11-16T22:01:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378638 2020-11-16T22:01:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-60129542160 3-81604378638' 2020-11-16T22:01:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:01:39.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:01:39.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215117 2020-11-16T22:01:39.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215117 2020-11-16T22:01:39.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-38654705682 2-60129542160 3-81604378638 4-103079215117' 2020-11-16T22:01:39.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:01:39.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836499 2020-11-16T22:01:39.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:01:39.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:01:39.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836499 2020-11-16T22:01:39.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:01:39.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836499 2020-11-16T22:01:39.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2020-11-16T22:01:39.596 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836499 2020-11-16T22:01:39.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:01:39.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836499 -lt 21474836499 2020-11-16T22:01:39.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:01:39.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705682 2020-11-16T22:01:39.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:01:39.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:01:40.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:01:40.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705682 2020-11-16T22:01:40.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705682 2020-11-16T22:01:40.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705682' 2020-11-16T22:01:40.002 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705682 2020-11-16T22:01:40.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:01:40.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705682 -lt 38654705682 2020-11-16T22:01:40.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:01:40.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542160 2020-11-16T22:01:40.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:01:40.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:01:40.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542160 2020-11-16T22:01:40.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:01:40.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542160 2020-11-16T22:01:40.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2020-11-16T22:01:40.400 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 60129542160 2020-11-16T22:01:40.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:01:40.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542160 -lt 60129542160 2020-11-16T22:01:40.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:01:40.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378638 2020-11-16T22:01:40.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:01:40.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:01:40.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378638 2020-11-16T22:01:40.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:01:40.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378638 2020-11-16T22:01:40.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378638' 2020-11-16T22:01:40.823 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378638 2020-11-16T22:01:40.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:01:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378638 -lt 81604378638 2020-11-16T22:01:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:01:41.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215117 2020-11-16T22:01:41.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:01:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:01:41.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215117 2020-11-16T22:01:41.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:01:41.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215117 2020-11-16T22:01:41.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215117' 2020-11-16T22:01:41.220 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 103079215117 2020-11-16T22:01:41.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:01:41.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215117 -lt 103079215117 2020-11-16T22:01:41.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:01:41.698 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:41.698 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:42.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T22:01:42.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:42.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:42.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:42.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:42.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:42.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:42.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:42.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:42.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:42.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:42.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:43.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:43.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T22:01:43.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T22:01:43.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T22:01:43.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:43.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:43.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:43.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:43.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:43.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:43.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:43.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:43.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:43.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:43.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:43.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:43.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:43.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:44.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:44.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:01:44.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:01:44.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:01:44.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:01:44.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:01:44.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:01:44.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:01:44.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:01:44.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T22:01:44.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T22:01:44.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:01:44.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:44.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:44.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:44.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:44.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:44.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:44.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:44.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:44.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:45.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:45.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:01:45.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:01:45.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:01:45.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:01:45.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:01:45.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:01:45.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:01:45.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:01:45.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:01:45.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:01:45.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:01:45.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:45.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:45.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:45.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:45.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:45.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:45.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:45.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:46.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:46.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:46.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:46.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:46.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:46.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:46.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:46.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:01:46.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:01:46.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:01:46.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:01:46.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:01:46.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:01:46.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:01:46.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:01:47.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:01:47.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:01:47.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:01:47.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:47.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:47.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:47.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:47.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:47.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:47.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:47.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:47.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:47.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:47.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:47.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:47.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:47.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:48.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:48.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:01:48.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:01:48.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:01:48.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:01:48.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:01:48.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:01:48.301 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:01:48.301 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:01:48.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T22:01:48.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T22:01:48.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:01:48.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:48.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:48.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:48.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:48.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:48.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:48.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:48.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:48.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:48.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:49.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:49.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:49.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:49.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:49.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:49.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:01:49.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:01:49.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:01:49.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:01:49.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:01:49.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:01:49.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:01:49.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:01:50.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T22:01:50.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T22:01:50.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:01:50.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:50.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:50.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:50.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:50.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:50.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:50.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:50.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:50.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:50.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:50.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:50.852 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:50.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:50.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:51.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:51.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:01:51.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:01:51.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:01:51.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:01:51.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:01:51.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:01:51.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:01:51.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:01:51.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:01:51.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:01:51.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:01:51.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:51.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:51.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:51.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:51.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:51.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:51.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:51.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:51.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:51.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:52.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:52.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:52.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:52.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:52.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:01:52.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:01:52.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:01:52.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:01:52.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:01:52.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:01:52.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:01:52.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:01:53.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:01:53.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:01:53.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:01:53.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:53.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:53.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:53.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:53.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:53.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:53.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:53.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:53.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:53.367 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:53.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:53.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:53.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:53.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:54.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:54.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:01:54.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:01:54.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:01:54.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:01:54.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:01:54.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:01:54.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:01:54.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:01:54.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:01:54.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:01:54.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:01:54.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:54.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:54.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:54.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:54.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:54.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:54.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:54.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:54.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:54.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:55.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:55.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:55.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:55.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:55.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:55.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:01:55.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:01:55.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:01:55.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:01:55.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:01:55.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:01:55.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:01:55.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:01:56.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:01:56.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:01:56.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:01:56.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:56.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:56.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:56.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:56.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:56.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:56.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:56.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:56.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:56.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:56.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:56.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:56.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:56.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:01:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:01:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:01:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:01:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:01:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:01:57.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:01:57.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:01:57.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:01:57.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:01:57.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:01:57.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:57.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:57.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:57.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:57.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:57.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:57.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:57.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:57.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:58.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:58.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:58.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:01:58.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:58.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:01:58.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:01:58.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:01:58.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:01:58.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:01:58.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:01:58.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:01:58.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:01:58.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:01:59.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:01:59.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:01:59.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:01:59.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:01:59.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:01:59.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:01:59.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:01:59.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:01:59.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:01:59.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:01:59.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:01:59.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:01:59.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:01:59.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:01:59.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:01:59.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:01:59.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:00.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:00.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:00.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:00.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:00.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:00.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:00.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:00.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:00.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:00.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:02:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:02:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:00.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:00.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:00.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:00.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:00.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:00.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:00.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:00.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:00.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:01.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:01.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:01.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:01.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:01.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:01.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:01.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:01.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:01.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:01.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:01.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:01.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:01.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:02.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:02:02.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:02:02.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:02.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:02.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:02.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:02.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:02.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:02.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:02.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:02.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:02.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:02.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:02.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:02.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:02.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:03.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:03.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:03.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:03.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:03.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:03.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:03.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:03.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:03.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:03.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:02:03.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:02:03.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:03.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:03.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:03.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:03.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:03.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:03.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:03.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:03.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:03.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:03.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:04.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:04.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:04.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:04.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:04.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:04.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:04.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:04.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:04.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:04.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:04.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:04.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:04.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:04.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:02:04.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:02:04.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:04.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:04.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:05.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:05.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:05.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:05.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:05.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:05.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:05.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:05.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:05.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:05.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:05.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:05.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:05.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:05.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:05.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:05.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:05.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:05.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:05.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:05.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:05.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:06.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:02:06.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:02:06.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:06.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:06.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:06.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:06.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:06.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:06.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:06.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:06.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:06.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:06.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:06.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:06.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:06.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:06.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:07.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:07.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:07.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:07.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:07.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:07.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:07.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:07.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:07.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:07.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:02:07.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:02:07.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:07.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:07.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:07.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:07.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:07.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:07.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:07.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:07.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:07.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:07.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:08.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:08.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:08.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:08.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:08.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:08.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:08.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:08.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:02:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:02:09.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:09.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:09.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:09.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:09.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:09.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:09.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:09.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:09.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:09.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:09.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:09.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:09.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:09.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:09.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:10.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:10.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:10.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:10.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:10.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:10.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:10.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:10.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:10.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:10.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:02:10.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:02:10.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:10.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:10.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:10.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:10.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:10.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:10.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:10.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:10.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:10.733 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:10.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:11.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:11.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:11.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:11.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:11.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:11.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:11.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:12.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:02:12.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:02:12.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:12.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:12.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:12.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:12.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:12.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:12.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:12.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:12.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:12.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:12.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:12.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:12.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:12.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:12.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:13.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:13.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:13.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:13.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:13.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:13.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:13.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:13.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:13.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:13.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:02:13.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:02:13.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:13.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:13.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:13.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:13.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:13.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:13.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:13.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:13.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:13.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:13.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:14.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:14.057 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:14.057 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:14.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:14.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:14.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:14.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:14.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:14.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:15.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:02:15.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:02:15.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:15.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:15.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:15.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:15.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:15.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:15.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:15.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:15.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:15.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:15.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:15.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:15.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:15.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:15.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:16.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:16.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:16.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:16.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:16.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:16.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:16.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:16.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:16.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:16.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:02:16.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:02:16.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:16.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:16.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:16.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:16.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:16.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:16.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:16.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:16.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:16.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:16.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:17.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:17.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:17.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:17.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:17.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:17.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:17.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:17.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:17.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:17.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:17.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:17.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:17.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:18.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:02:18.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:02:18.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:18.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:18.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:18.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:18.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:18.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:18.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:18.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:18.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:18.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:18.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:18.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:18.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:18.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:18.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:18.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:18.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:18.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:18.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:18.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:18.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:18.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:18.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:18.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:19.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:02:19.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:02:19.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:19.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:19.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:19.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:19.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:19.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:19.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:19.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:19.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:19.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:19.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:19.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:19.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:19.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:19.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:20.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:20.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:20.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:20.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:20.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:20.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:20.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:20.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:20.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:21.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:02:21.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:02:21.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:21.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:21.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:21.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:21.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:21.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:21.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:21.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:21.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:21.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:21.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:21.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:21.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:21.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:21.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:21.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:21.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:21.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:21.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:21.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:21.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:21.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:21.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:21.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:22.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T22:02:22.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T22:02:22.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:02:22.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:22.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:02:22.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:22.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:22.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:22.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:22.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:22.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:22.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:22.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:22.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:02:22.906 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:22.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:22.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:23.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:02:23.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:02:23.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:02:23.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:02:23.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:02:23.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:02:23.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:02:23.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:02:23.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:02:23.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T22:02:23.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T22:02:23.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2020-11-16T22:02:23.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:02:23.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-11-16T22:02:24.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:02:24.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:02:24.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:02:24.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:02:24.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:02:24.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:02:24.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:02:24.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:02:24.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:02:24.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:02:24.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:02:24.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:02:24.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T22:02:24.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:02:24.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:02:24.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:428: TEST_backfill_sizeup4_allout: expr 200 '*' 2 2020-11-16T22:02:24.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:428: TEST_backfill_sizeup4_allout: local misdeg=400 2020-11-16T22:02:24.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:429: TEST_backfill_sizeup4_allout: check td/osd-backfill-stats 1.0 1 replicated 400 0 400 200 2020-11-16T22:02:24.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2020-11-16T22:02:24.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2020-11-16T22:02:24.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2020-11-16T22:02:24.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2020-11-16T22:02:24.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=400 2020-11-16T22:02:24.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2020-11-16T22:02:24.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=400 2020-11-16T22:02:24.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=200 2020-11-16T22:02:24.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2020-11-16T22:02:24.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2020-11-16T22:02:24.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2020-11-16T22:02:24.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2020-11-16T22:02:24.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2020-11-16T22:02:24.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2020-11-16T22:02:24.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log 2020-11-16T22:02:24.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2020-11-16T22:02:24.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2020-11-16T22:02:24.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2020-11-16T22:02:24.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2020-11-16T22:02:24.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2020-11-16T22:02:24.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2020-11-16T22:02:24.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2020-11-16T22:02:24.950 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:02:24.951 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2020-11-16T22:02:24.951 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2020-11-16T22:02:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[2,4,3]/[1,0]' 2020-11-16T22:02:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[2,4,3]/[1,0]' ']' 2020-11-16T22:02:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2020-11-16T22:02:24.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:02:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:02:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [2,4,3]/[1,0] ' 2020-11-16T22:02:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2020-11-16T22:02:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2020-11-16T22:02:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:02:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=400 2020-11-16T22:02:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 400 ']' 2020-11-16T22:02:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 400 400 2020-11-16T22:02:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2020-11-16T22:02:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T22:02:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2020-11-16T22:02:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T22:02:24.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:02:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:02:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2020-11-16T22:02:24.986 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:02:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=2 2020-11-16T22:02:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 2 ']' 2020-11-16T22:02:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 2 0 2020-11-16T22:02:25.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=2 2020-11-16T22:02:25.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T22:02:25.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T22:02:25.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T22:02:25.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2020-11-16T22:02:25.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:02:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [2,4,3]/[1,0] ' 2020-11-16T22:02:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:02:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2020-11-16T22:02:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:02:25.004 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2020-11-16T22:02:25.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=400 2020-11-16T22:02:25.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 400 ']' 2020-11-16T22:02:25.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 400 400 2020-11-16T22:02:25.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2020-11-16T22:02:25.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T22:02:25.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2020-11-16T22:02:25.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T22:02:25.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:02:25.015 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:02:25.015 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2020-11-16T22:02:25.015 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:02:25.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=201 2020-11-16T22:02:25.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 201 ']' 2020-11-16T22:02:25.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 201 200 2020-11-16T22:02:25.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=201 2020-11-16T22:02:25.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T22:02:25.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=200 2020-11-16T22:02:25.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T22:02:25.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2020-11-16T22:02:25.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:431: TEST_backfill_sizeup4_allout: delete_pool test 2020-11-16T22:02:25.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T22:02:25.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T22:02:25.517 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T22:02:25.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:432: TEST_backfill_sizeup4_allout: kill_daemons td/osd-backfill-stats 2020-11-16T22:02:25.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:02:25.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:02:25.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:02:25.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:02:25.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:02:25.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:02:25.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2020-11-16T22:02:25.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T22:02:25.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:02:25.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T22:02:25.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:02:25.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:02:25.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:02:25.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:02:25.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:02:25.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:02:25.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:02:25.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:02:25.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:02:25.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:02:25.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:02:25.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:02:25.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:02:25.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:02:25.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:02:25.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:02:25.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:02:25.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:02:25.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T22:02:25.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:02:25.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:25.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:25.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T22:02:25.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:02:25.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:02:25.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2020-11-16T22:02:25.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2020-11-16T22:02:25.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2020-11-16T22:02:25.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2020-11-16T22:02:25.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T22:02:25.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:02:25.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T22:02:25.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:02:25.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:02:25.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:02:25.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:02:25.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:02:25.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:02:25.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:02:25.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:02:25.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:02:25.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:02:25.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:02:25.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:02:25.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:02:25.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:02:25.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:02:25.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:02:25.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:02:25.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:02:25.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T22:02:25.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:02:25.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:25.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:25.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T22:02:25.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:02:25.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:02:25.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2020-11-16T22:02:25.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:02:25.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:25.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:25.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.465868 2020-11-16T22:02:25.706 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:02:25.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T22:02:25.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:02:25.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2020-11-16T22:02:25.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_sizeup_out td/osd-backfill-stats 2020-11-16T22:02:25.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:182: TEST_backfill_sizeup_out: local dir=td/osd-backfill-stats 2020-11-16T22:02:25.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:184: TEST_backfill_sizeup_out: run_mon td/osd-backfill-stats a 2020-11-16T22:02:25.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2020-11-16T22:02:25.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:02:25.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:02:25.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:02:25.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2020-11-16T22:02:25.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2020-11-16T22:02:26.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:02:26.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:02:26.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:02:26.020 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:02:26.020 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:26.021 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:26.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:26.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:02:26.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:02:26.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:02:26.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:02:26.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:02:26.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:02:26.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:02:26.242 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:02:26.242 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:02:26.242 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:02:26.243 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:02:26.243 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:26.243 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:26.243 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T22:02:26.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:02:26.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get fsid 2020-11-16T22:02:26.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:02:26.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:02:26.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:02:26.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:02:26.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:02:26.355 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:02:26.356 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:02:26.356 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:02:26.356 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:02:26.356 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:26.356 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:26.356 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.465868/ceph-mon.a.asok 2020-11-16T22:02:26.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:02:26.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.465868/ceph-mon.a.asok config get mon_host 2020-11-16T22:02:26.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:185: TEST_backfill_sizeup_out: run_mgr td/osd-backfill-stats x 2020-11-16T22:02:26.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2020-11-16T22:02:26.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:02:26.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:02:26.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:02:26.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2020-11-16T22:02:26.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:02:27.004 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:02:27.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:02:27.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:02:27.005 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:02:27.006 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:27.006 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:27.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:27.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:02:27.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:02:27.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:186: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 0 2020-11-16T22:02:27.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T22:02:27.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:02:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:02:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:02:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2020-11-16T22:02:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T22:02:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:02:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:02:27.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:02:27.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2020-11-16T22:02:27.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2020-11-16T22:02:27.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:02:27.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:02:27.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T22:02:27.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:02:27.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:02:27.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:02:27.037 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:02:27.037 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:27.037 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:27.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:27.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:27.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:02:27.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:02:27.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:02:27.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T22:02:27.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T22:02:27.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:02:27.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:02:27.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:02:27.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:02:27.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:02:27.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2020-11-16T22:02:27.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:02:27.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e81e2891-d767-4d8b-a3f7-770dd87f77f8 2020-11-16T22:02:27.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 e81e2891-d767-4d8b-a3f7-770dd87f77f8' 2020-11-16T22:02:27.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:02:27.041 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 e81e2891-d767-4d8b-a3f7-770dd87f77f8 2020-11-16T22:02:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDz9rJfzFxLBBAAREK7smMPCR5wyDtBWbBxBQ== 2020-11-16T22:02:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDz9rJfzFxLBBAAREK7smMPCR5wyDtBWbBxBQ=="}' 2020-11-16T22:02:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e81e2891-d767-4d8b-a3f7-770dd87f77f8 -i td/osd-backfill-stats/0/new.json 2020-11-16T22:02:27.408 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:02:27.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2020-11-16T22:02:27.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDz9rJfzFxLBBAAREK7smMPCR5wyDtBWbBxBQ== --osd-uuid e81e2891-d767-4d8b-a3f7-770dd87f77f8 2020-11-16T22:02:27.450 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:27.458+0000 7fb8116dad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:27.453 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:27.462+0000 7fb8116dad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:27.455 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:27.462+0000 7fb8116dad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:27.456 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:27.462+0000 7fb8116dad80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T22:02:27.456 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:27.462+0000 7fb8116dad80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T22:02:27.456 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:27.462+0000 7fb8116dad80 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2020-11-16T22:02:27.498 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:27.506+0000 7fb8116dad80 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2020-11-16T22:02:31.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2020-11-16T22:02:31.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:02:31.977 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:02:31.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:02:31.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:02:32.483 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:02:32.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:02:32.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:02:32.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:02:32.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:02:32.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:02:32.513 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:32.522+0000 7f844a049d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:32.515 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:32.522+0000 7f844a049d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:32.516 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:32.526+0000 7f844a049d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:32.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:02:32.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:02:32.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:02:32.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:02:32.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:02:32.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:02:32.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:02:32.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:02:32.878 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:02:32.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:02:33.054 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:33.062+0000 7f844a049d80 -1 Falling back to public interface 2020-11-16T22:02:33.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:02:33.875 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:33.878+0000 7f844a049d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:02:34.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:02:34.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:02:34.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:02:34.255 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:02:34.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:02:34.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:02:34.670 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/597705,v1:127.0.0.1:6803/597705] [v2:127.0.0.1:6804/597705,v1:127.0.0.1:6805/597705] exists,up e81e2891-d767-4d8b-a3f7-770dd87f77f8 2020-11-16T22:02:34.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:02:34.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:02:34.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:02:34.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:187: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 1 2020-11-16T22:02:34.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T22:02:34.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:02:34.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:02:34.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:02:34.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2020-11-16T22:02:34.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T22:02:34.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:02:34.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:02:34.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:02:34.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2020-11-16T22:02:34.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2020-11-16T22:02:34.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:02:34.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:02:34.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T22:02:34.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:02:34.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:02:34.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:02:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:02:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:34.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:34.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:02:34.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:02:34.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:02:34.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T22:02:34.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T22:02:34.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:02:34.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:02:34.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:02:34.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:02:34.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:02:34.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2020-11-16T22:02:34.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:02:34.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c7975ea1-4828-472d-8d82-cdb84d8bc7ce 2020-11-16T22:02:34.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 c7975ea1-4828-472d-8d82-cdb84d8bc7ce' 2020-11-16T22:02:34.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:02:34.678 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 c7975ea1-4828-472d-8d82-cdb84d8bc7ce 2020-11-16T22:02:34.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD69rJfoqL0KRAAJX61EskRiE/VqDQrlOy74Q== 2020-11-16T22:02:34.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD69rJfoqL0KRAAJX61EskRiE/VqDQrlOy74Q=="}' 2020-11-16T22:02:34.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c7975ea1-4828-472d-8d82-cdb84d8bc7ce -i td/osd-backfill-stats/1/new.json 2020-11-16T22:02:35.159 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:02:35.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2020-11-16T22:02:35.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD69rJfoqL0KRAAJX61EskRiE/VqDQrlOy74Q== --osd-uuid c7975ea1-4828-472d-8d82-cdb84d8bc7ce 2020-11-16T22:02:35.206 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:35.214+0000 7fa3c2bced80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:35.209 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:35.218+0000 7fa3c2bced80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:35.210 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:35.218+0000 7fa3c2bced80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:35.211 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:35.218+0000 7fa3c2bced80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T22:02:35.211 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:35.218+0000 7fa3c2bced80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T22:02:35.212 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:35.218+0000 7fa3c2bced80 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2020-11-16T22:02:35.292 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:35.298+0000 7fa3c2bced80 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2020-11-16T22:02:39.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2020-11-16T22:02:39.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:02:39.359 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:02:39.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:02:39.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:02:39.854 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:02:39.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:02:39.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:02:39.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:02:39.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:02:39.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:02:39.883 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:39.890+0000 7fa5d3a8cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:39.885 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:39.894+0000 7fa5d3a8cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:39.887 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:39.894+0000 7fa5d3a8cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:40.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:02:40.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:02:40.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:02:40.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:02:40.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:02:40.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:02:40.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:02:40.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:02:40.242 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:02:40.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:02:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:40.438+0000 7fa5d3a8cd80 -1 Falling back to public interface 2020-11-16T22:02:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:02:41.318 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:41.326+0000 7fa5d3a8cd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:02:41.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:02:41.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:02:41.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:02:41.619 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:02:41.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:02:41.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:02:42.031 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/598699,v1:127.0.0.1:6811/598699] [v2:127.0.0.1:6812/598699,v1:127.0.0.1:6813/598699] exists,up c7975ea1-4828-472d-8d82-cdb84d8bc7ce 2020-11-16T22:02:42.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:02:42.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:02:42.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:02:42.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:188: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 2 2020-11-16T22:02:42.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T22:02:42.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:02:42.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T22:02:42.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:02:42.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2020-11-16T22:02:42.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T22:02:42.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:02:42.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:02:42.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:02:42.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2020-11-16T22:02:42.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2020-11-16T22:02:42.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:02:42.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:02:42.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T22:02:42.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:02:42.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:02:42.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:02:42.036 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:02:42.036 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:42.036 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:42.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:42.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:42.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:02:42.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:02:42.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:02:42.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T22:02:42.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T22:02:42.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:02:42.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:02:42.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:02:42.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:02:42.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:02:42.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2020-11-16T22:02:42.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:02:42.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=112a5119-57ee-4569-ac94-344d5f2bc292 2020-11-16T22:02:42.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 112a5119-57ee-4569-ac94-344d5f2bc292' 2020-11-16T22:02:42.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:02:42.039 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 112a5119-57ee-4569-ac94-344d5f2bc292 2020-11-16T22:02:42.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAC97Jfgp4TBBAARC6rbBYnTQNC+9tef7QeWw== 2020-11-16T22:02:42.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAC97Jfgp4TBBAARC6rbBYnTQNC+9tef7QeWw=="}' 2020-11-16T22:02:42.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 112a5119-57ee-4569-ac94-344d5f2bc292 -i td/osd-backfill-stats/2/new.json 2020-11-16T22:02:42.538 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:02:42.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2020-11-16T22:02:42.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAC97Jfgp4TBBAARC6rbBYnTQNC+9tef7QeWw== --osd-uuid 112a5119-57ee-4569-ac94-344d5f2bc292 2020-11-16T22:02:42.585 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:42.594+0000 7fba06accd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:42.587 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:42.594+0000 7fba06accd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:42.588 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:42.594+0000 7fba06accd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:42.589 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:42.598+0000 7fba06accd80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T22:02:42.589 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:42.598+0000 7fba06accd80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T22:02:42.589 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:42.598+0000 7fba06accd80 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2020-11-16T22:02:42.636 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:42.642+0000 7fba06accd80 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2020-11-16T22:02:46.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2020-11-16T22:02:46.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:02:46.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T22:02:46.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:02:46.684 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T22:02:47.195 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:02:47.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T22:02:47.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:02:47.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:02:47.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:02:47.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:02:47.225 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:47.234+0000 7feee8bc5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:47.250 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:47.258+0000 7feee8bc5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:47.251 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:47.258+0000 7feee8bc5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:47.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T22:02:47.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:02:47.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:02:47.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:02:47.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:02:47.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:02:47.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:02:47.574 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:02:47.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:02:47.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:02:47.787 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:47.794+0000 7feee8bc5d80 -1 Falling back to public interface 2020-11-16T22:02:47.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:02:48.637 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:48.638+0000 7feee8bc5d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T22:02:48.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:02:48.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:02:48.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:02:48.963 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:02:48.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:02:48.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:02:49.398 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/599693,v1:127.0.0.1:6819/599693] [v2:127.0.0.1:6820/599693,v1:127.0.0.1:6821/599693] exists,up 112a5119-57ee-4569-ac94-344d5f2bc292 2020-11-16T22:02:49.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:02:49.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:02:49.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:02:49.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:189: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 3 2020-11-16T22:02:49.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T22:02:49.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:02:49.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T22:02:49.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:02:49.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2020-11-16T22:02:49.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T22:02:49.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:02:49.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:02:49.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:02:49.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2020-11-16T22:02:49.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2020-11-16T22:02:49.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:02:49.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:02:49.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T22:02:49.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:02:49.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:02:49.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:02:49.402 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:02:49.403 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:49.403 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:49.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:49.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:49.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:02:49.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:02:49.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:02:49.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T22:02:49.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T22:02:49.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:02:49.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:02:49.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:02:49.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:02:49.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:02:49.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2020-11-16T22:02:49.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:02:49.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f1e10fd0-f669-4f5f-aa20-81c1ac9a58db 2020-11-16T22:02:49.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 f1e10fd0-f669-4f5f-aa20-81c1ac9a58db' 2020-11-16T22:02:49.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:02:49.406 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 f1e10fd0-f669-4f5f-aa20-81c1ac9a58db 2020-11-16T22:02:49.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAJ97JfXzYZGhAAJ7f6joi/0xElUcp8xd3jjw== 2020-11-16T22:02:49.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAJ97JfXzYZGhAAJ7f6joi/0xElUcp8xd3jjw=="}' 2020-11-16T22:02:49.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f1e10fd0-f669-4f5f-aa20-81c1ac9a58db -i td/osd-backfill-stats/3/new.json 2020-11-16T22:02:49.923 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:02:49.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2020-11-16T22:02:49.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAJ97JfXzYZGhAAJ7f6joi/0xElUcp8xd3jjw== --osd-uuid f1e10fd0-f669-4f5f-aa20-81c1ac9a58db 2020-11-16T22:02:49.963 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:49.970+0000 7f8068debd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:49.965 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:49.974+0000 7f8068debd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:49.974+0000 7f8068debd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:49.974+0000 7f8068debd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T22:02:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:49.974+0000 7f8068debd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T22:02:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:49.974+0000 7f8068debd80 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2020-11-16T22:02:50.073 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:50.082+0000 7f8068debd80 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2020-11-16T22:02:54.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2020-11-16T22:02:54.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:02:54.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T22:02:54.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:02:54.152 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T22:02:54.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T22:02:54.642 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T22:02:54.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:02:54.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:02:54.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:02:54.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:02:54.670 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:54.678+0000 7fb4c2891d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:54.672 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:54.682+0000 7fb4c2891d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:54.674 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:54.682+0000 7fb4c2891d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:55.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T22:02:55.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:02:55.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T22:02:55.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:02:55.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:02:55.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:02:55.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:02:55.049 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:02:55.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:02:55.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:02:55.207 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:55.214+0000 7fb4c2891d80 -1 Falling back to public interface 2020-11-16T22:02:55.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:02:56.054 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:56.062+0000 7fb4c2891d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T22:02:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:02:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:02:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:02:56.429 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:02:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:02:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:02:56.875 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/600687,v1:127.0.0.1:6827/600687] [v2:127.0.0.1:6828/600687,v1:127.0.0.1:6829/600687] exists,up f1e10fd0-f669-4f5f-aa20-81c1ac9a58db 2020-11-16T22:02:56.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:02:56.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:02:56.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:02:56.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:190: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 4 2020-11-16T22:02:56.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T22:02:56.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:02:56.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T22:02:56.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:02:56.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2020-11-16T22:02:56.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T22:02:56.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:02:56.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:02:56.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:02:56.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2020-11-16T22:02:56.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2020-11-16T22:02:56.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:02:56.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:02:56.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T22:02:56.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:02:56.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:02:56.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:02:56.880 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:02:56.880 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:02:56.880 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:02:56.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:56.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:02:56.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:02:56.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:02:56.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:02:56.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T22:02:56.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T22:02:56.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:02:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:02:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:02:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:02:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:02:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2020-11-16T22:02:56.882 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:02:56.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5f0a5514-4a88-45ec-9d72-bea69d5c4f02 2020-11-16T22:02:56.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 5f0a5514-4a88-45ec-9d72-bea69d5c4f02' 2020-11-16T22:02:56.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:02:56.883 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 5f0a5514-4a88-45ec-9d72-bea69d5c4f02 2020-11-16T22:02:56.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAQ97Jf/lIWNhAAP1vbO7q3/6P7hIlWkYPJGA== 2020-11-16T22:02:56.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAQ97Jf/lIWNhAAP1vbO7q3/6P7hIlWkYPJGA=="}' 2020-11-16T22:02:56.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5f0a5514-4a88-45ec-9d72-bea69d5c4f02 -i td/osd-backfill-stats/4/new.json 2020-11-16T22:02:57.446 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T22:02:57.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2020-11-16T22:02:57.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAQ97Jf/lIWNhAAP1vbO7q3/6P7hIlWkYPJGA== --osd-uuid 5f0a5514-4a88-45ec-9d72-bea69d5c4f02 2020-11-16T22:02:57.491 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:57.498+0000 7fbeed938d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:57.494 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:57.502+0000 7fbeed938d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:57.495 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:57.502+0000 7fbeed938d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:02:57.496 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:57.502+0000 7fbeed938d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T22:02:57.496 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:57.502+0000 7fbeed938d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T22:02:57.496 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:57.502+0000 7fbeed938d80 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2020-11-16T22:02:57.535 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:02:57.542+0000 7fbeed938d80 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2020-11-16T22:03:01.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2020-11-16T22:03:01.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:03:01.944 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T22:03:01.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T22:03:01.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:03:02.438 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T22:03:02.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T22:03:02.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:03:02.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:03:02.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:03:02.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:03:02.459 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:02.466+0000 7f7b27205d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:03:02.462 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:02.470+0000 7f7b27205d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:03:02.463 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:02.470+0000 7f7b27205d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:03:02.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T22:03:02.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:03:02.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T22:03:02.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:03:02.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:03:02.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:03:02.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:03:02.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:03:02.842 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:03:02.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T22:03:03.015 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:03.022+0000 7f7b27205d80 -1 Falling back to public interface 2020-11-16T22:03:03.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:03:03.856 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:03.858+0000 7f7b27205d80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T22:03:04.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:03:04.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:03:04.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:03:04.244 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:03:04.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:03:04.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T22:03:04.669 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/601681,v1:127.0.0.1:6835/601681] [v2:127.0.0.1:6836/601681,v1:127.0.0.1:6837/601681] exists,up 5f0a5514-4a88-45ec-9d72-bea69d5c4f02 2020-11-16T22:03:04.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:03:04.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:03:04.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:03:04.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:191: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 5 2020-11-16T22:03:04.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2020-11-16T22:03:04.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:03:04.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-11-16T22:03:04.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:03:04.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2020-11-16T22:03:04.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-11-16T22:03:04.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:03:04.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:03:04.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:03:04.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2020-11-16T22:03:04.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2020-11-16T22:03:04.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:03:04.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:03:04.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2020-11-16T22:03:04.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:03:04.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:03:04.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:03:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:03:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:03:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:03:04.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:03:04.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' 2020-11-16T22:03:04.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:03:04.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:03:04.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:03:04.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2020-11-16T22:03:04.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2020-11-16T22:03:04.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:03:04.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:03:04.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:03:04.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:03:04.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:03:04.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2020-11-16T22:03:04.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:03:04.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b7f69017-3fa9-4848-9653-a5eb57632f74 2020-11-16T22:03:04.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 b7f69017-3fa9-4848-9653-a5eb57632f74' 2020-11-16T22:03:04.677 INFO:tasks.workunit.client.0.smithi189.stdout:add osd5 b7f69017-3fa9-4848-9653-a5eb57632f74 2020-11-16T22:03:04.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:03:04.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAY97JfB4/jKRAAJ5MoBatv5mpHgONDEMvgqQ== 2020-11-16T22:03:04.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAY97JfB4/jKRAAJ5MoBatv5mpHgONDEMvgqQ=="}' 2020-11-16T22:03:04.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b7f69017-3fa9-4848-9653-a5eb57632f74 -i td/osd-backfill-stats/5/new.json 2020-11-16T22:03:05.240 INFO:tasks.workunit.client.0.smithi189.stdout:5 2020-11-16T22:03:05.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2020-11-16T22:03:05.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAY97JfB4/jKRAAJ5MoBatv5mpHgONDEMvgqQ== --osd-uuid b7f69017-3fa9-4848-9653-a5eb57632f74 2020-11-16T22:03:05.287 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:05.294+0000 7f7af27add80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:03:05.318 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:05.326+0000 7f7af27add80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:03:05.320 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:05.326+0000 7f7af27add80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:03:05.320 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:05.326+0000 7f7af27add80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T22:03:05.320 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:05.326+0000 7f7af27add80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T22:03:05.320 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:05.326+0000 7f7af27add80 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2020-11-16T22:03:05.413 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:05.422+0000 7f7af27add80 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2020-11-16T22:03:09.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2020-11-16T22:03:09.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:03:09.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-11-16T22:03:09.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:03:09.980 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd5 key to auth repository 2020-11-16T22:03:10.489 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.5 2020-11-16T22:03:10.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-11-16T22:03:10.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=947ccb7e-60d8-4f3a-bade-b334025cdb29 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.465868/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:03:10.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:03:10.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:03:10.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:03:10.518 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:10.526+0000 7f8d7d3bcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:03:10.521 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:10.530+0000 7f8d7d3bcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:03:10.522 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:10.530+0000 7f8d7d3bcd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:03:10.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-11-16T22:03:10.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:03:10.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-11-16T22:03:10.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:03:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:03:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:03:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:03:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:03:10.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T22:03:10.884 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:03:11.059 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:11.066+0000 7f8d7d3bcd80 -1 Falling back to public interface 2020-11-16T22:03:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:03:11.911 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:03:11.914+0000 7f8d7d3bcd80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-16T22:03:12.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:03:12.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:03:12.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:03:12.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:03:12.275 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:03:12.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-11-16T22:03:12.696 INFO:tasks.workunit.client.0.smithi189.stdout:osd.5 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/602675,v1:127.0.0.1:6843/602675] [v2:127.0.0.1:6844/602675,v1:127.0.0.1:6845/602675] exists,up b7f69017-3fa9-4848-9653-a5eb57632f74 2020-11-16T22:03:12.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:03:12.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:03:12.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:03:12.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:193: TEST_backfill_sizeup_out: create_pool test 1 1 2020-11-16T22:03:12.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2020-11-16T22:03:13.373 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' already exists 2020-11-16T22:03:13.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:03:14.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:194: TEST_backfill_sizeup_out: ceph osd pool set test size 1 --yes-i-really-mean-it 2020-11-16T22:03:15.422 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 1 2020-11-16T22:03:15.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:196: TEST_backfill_sizeup_out: wait_for_clean 2020-11-16T22:03:15.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:03:15.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:03:15.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:03:15.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:03:15.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:03:15.438 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:03:15.438 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:03:15.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:03:15.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:03:15.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:03:15.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:03:15.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:03:15.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:03:15.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:03:15.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:03:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:03:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:03:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:03:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:03:16.045 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T22:03:16.046 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T22:03:16.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:03:16.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:16.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:03:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2020-11-16T22:03:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2020-11-16T22:03:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2020-11-16T22:03:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:16.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:03:16.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705674 2020-11-16T22:03:16.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705674 2020-11-16T22:03:16.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-38654705674' 2020-11-16T22:03:16.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:16.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:03:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574857 2020-11-16T22:03:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574857 2020-11-16T22:03:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857' 2020-11-16T22:03:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:16.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:03:16.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=73014444038 2020-11-16T22:03:16.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 73014444038 2020-11-16T22:03:16.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-73014444038' 2020-11-16T22:03:16.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:16.728 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:03:16.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=94489280517 2020-11-16T22:03:16.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 94489280517 2020-11-16T22:03:16.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-73014444038 4-94489280517' 2020-11-16T22:03:16.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:16.893 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T22:03:17.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964116996 2020-11-16T22:03:17.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964116996 2020-11-16T22:03:17.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574857 3-73014444038 4-94489280517 5-115964116996' 2020-11-16T22:03:17.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:17.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2020-11-16T22:03:17.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:17.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:03:17.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2020-11-16T22:03:17.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:17.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2020-11-16T22:03:17.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2020-11-16T22:03:17.071 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836491 2020-11-16T22:03:17.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:03:17.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836491 2020-11-16T22:03:17.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:17.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705674 2020-11-16T22:03:17.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:17.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:03:17.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705674 2020-11-16T22:03:17.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:17.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705674 2020-11-16T22:03:17.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2020-11-16T22:03:17.470 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705674 2020-11-16T22:03:17.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:03:17.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705674 -lt 38654705674 2020-11-16T22:03:17.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:17.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-55834574857 2020-11-16T22:03:17.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:17.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:03:17.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-55834574857 2020-11-16T22:03:17.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:17.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574857 2020-11-16T22:03:17.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 55834574857' 2020-11-16T22:03:17.866 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 55834574857 2020-11-16T22:03:17.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:03:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574857 -lt 55834574857 2020-11-16T22:03:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:18.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-73014444038 2020-11-16T22:03:18.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:03:18.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:18.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-73014444038 2020-11-16T22:03:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=73014444038 2020-11-16T22:03:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 73014444038' 2020-11-16T22:03:18.259 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 73014444038 2020-11-16T22:03:18.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:03:18.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 73014444036 -lt 73014444038 2020-11-16T22:03:18.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:03:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:03:19.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:03:20.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 73014444039 -lt 73014444038 2020-11-16T22:03:20.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:20.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-94489280517 2020-11-16T22:03:20.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:20.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:03:20.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:20.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-94489280517 2020-11-16T22:03:20.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=94489280517 2020-11-16T22:03:20.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 94489280517' 2020-11-16T22:03:20.047 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 94489280517 2020-11-16T22:03:20.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:03:20.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 94489280517 -lt 94489280517 2020-11-16T22:03:20.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:20.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-115964116996 2020-11-16T22:03:20.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:20.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T22:03:20.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-115964116996 2020-11-16T22:03:20.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:20.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964116996 2020-11-16T22:03:20.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 115964116996' 2020-11-16T22:03:20.413 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 115964116996 2020-11-16T22:03:20.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T22:03:20.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964116997 -lt 115964116996 2020-11-16T22:03:20.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:03:20.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:20.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:21.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T22:03:21.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:03:21.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:03:21.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:03:21.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:03:21.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:03:21.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:03:21.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:03:21.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:03:21.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:03:21.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:21.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:22.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T22:03:22.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:03:22.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:03:22.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: seq 1 200 2020-11-16T22:03:22.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj1 /dev/null 2020-11-16T22:03:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj2 /dev/null 2020-11-16T22:03:22.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj3 /dev/null 2020-11-16T22:03:22.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj4 /dev/null 2020-11-16T22:03:22.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj5 /dev/null 2020-11-16T22:03:22.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj6 /dev/null 2020-11-16T22:03:22.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj7 /dev/null 2020-11-16T22:03:22.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj8 /dev/null 2020-11-16T22:03:22.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj9 /dev/null 2020-11-16T22:03:22.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj10 /dev/null 2020-11-16T22:03:22.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj11 /dev/null 2020-11-16T22:03:22.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:22.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj12 /dev/null 2020-11-16T22:03:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj13 /dev/null 2020-11-16T22:03:23.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj14 /dev/null 2020-11-16T22:03:23.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj15 /dev/null 2020-11-16T22:03:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj16 /dev/null 2020-11-16T22:03:23.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj17 /dev/null 2020-11-16T22:03:23.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj18 /dev/null 2020-11-16T22:03:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj19 /dev/null 2020-11-16T22:03:23.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj20 /dev/null 2020-11-16T22:03:23.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj21 /dev/null 2020-11-16T22:03:23.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj22 /dev/null 2020-11-16T22:03:23.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj23 /dev/null 2020-11-16T22:03:23.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj24 /dev/null 2020-11-16T22:03:23.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj25 /dev/null 2020-11-16T22:03:23.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:23.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj26 /dev/null 2020-11-16T22:03:24.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj27 /dev/null 2020-11-16T22:03:24.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj28 /dev/null 2020-11-16T22:03:24.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj29 /dev/null 2020-11-16T22:03:24.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj30 /dev/null 2020-11-16T22:03:24.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj31 /dev/null 2020-11-16T22:03:24.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj32 /dev/null 2020-11-16T22:03:24.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj33 /dev/null 2020-11-16T22:03:24.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj34 /dev/null 2020-11-16T22:03:24.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj35 /dev/null 2020-11-16T22:03:24.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj36 /dev/null 2020-11-16T22:03:24.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj37 /dev/null 2020-11-16T22:03:24.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj38 /dev/null 2020-11-16T22:03:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj39 /dev/null 2020-11-16T22:03:25.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj40 /dev/null 2020-11-16T22:03:25.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj41 /dev/null 2020-11-16T22:03:25.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj42 /dev/null 2020-11-16T22:03:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj43 /dev/null 2020-11-16T22:03:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj44 /dev/null 2020-11-16T22:03:25.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj45 /dev/null 2020-11-16T22:03:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj46 /dev/null 2020-11-16T22:03:25.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj47 /dev/null 2020-11-16T22:03:25.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj48 /dev/null 2020-11-16T22:03:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj49 /dev/null 2020-11-16T22:03:25.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj50 /dev/null 2020-11-16T22:03:25.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj51 /dev/null 2020-11-16T22:03:25.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:25.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj52 /dev/null 2020-11-16T22:03:26.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj53 /dev/null 2020-11-16T22:03:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj54 /dev/null 2020-11-16T22:03:26.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj55 /dev/null 2020-11-16T22:03:26.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj56 /dev/null 2020-11-16T22:03:26.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj57 /dev/null 2020-11-16T22:03:26.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj58 /dev/null 2020-11-16T22:03:26.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj59 /dev/null 2020-11-16T22:03:26.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj60 /dev/null 2020-11-16T22:03:26.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj61 /dev/null 2020-11-16T22:03:26.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj62 /dev/null 2020-11-16T22:03:26.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj63 /dev/null 2020-11-16T22:03:26.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj64 /dev/null 2020-11-16T22:03:26.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:26.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj65 /dev/null 2020-11-16T22:03:27.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:27.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj66 /dev/null 2020-11-16T22:03:27.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:27.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj67 /dev/null 2020-11-16T22:03:27.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:27.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj68 /dev/null 2020-11-16T22:03:27.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:27.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj69 /dev/null 2020-11-16T22:03:27.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:27.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj70 /dev/null 2020-11-16T22:03:27.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:27.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj71 /dev/null 2020-11-16T22:03:27.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:27.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj72 /dev/null 2020-11-16T22:03:27.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:27.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj73 /dev/null 2020-11-16T22:03:27.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:27.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj74 /dev/null 2020-11-16T22:03:27.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:27.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj75 /dev/null 2020-11-16T22:03:27.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:27.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj76 /dev/null 2020-11-16T22:03:28.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj77 /dev/null 2020-11-16T22:03:28.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj78 /dev/null 2020-11-16T22:03:28.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj79 /dev/null 2020-11-16T22:03:28.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj80 /dev/null 2020-11-16T22:03:28.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj81 /dev/null 2020-11-16T22:03:28.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj82 /dev/null 2020-11-16T22:03:28.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj83 /dev/null 2020-11-16T22:03:28.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj84 /dev/null 2020-11-16T22:03:28.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj85 /dev/null 2020-11-16T22:03:28.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj86 /dev/null 2020-11-16T22:03:28.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj87 /dev/null 2020-11-16T22:03:28.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj88 /dev/null 2020-11-16T22:03:28.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:28.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj89 /dev/null 2020-11-16T22:03:29.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj90 /dev/null 2020-11-16T22:03:29.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj91 /dev/null 2020-11-16T22:03:29.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj92 /dev/null 2020-11-16T22:03:29.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj93 /dev/null 2020-11-16T22:03:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj94 /dev/null 2020-11-16T22:03:29.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj95 /dev/null 2020-11-16T22:03:29.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj96 /dev/null 2020-11-16T22:03:29.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj97 /dev/null 2020-11-16T22:03:29.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj98 /dev/null 2020-11-16T22:03:29.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj99 /dev/null 2020-11-16T22:03:29.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj100 /dev/null 2020-11-16T22:03:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj101 /dev/null 2020-11-16T22:03:29.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:29.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj102 /dev/null 2020-11-16T22:03:30.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj103 /dev/null 2020-11-16T22:03:30.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj104 /dev/null 2020-11-16T22:03:30.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj105 /dev/null 2020-11-16T22:03:30.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj106 /dev/null 2020-11-16T22:03:30.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj107 /dev/null 2020-11-16T22:03:30.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj108 /dev/null 2020-11-16T22:03:30.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj109 /dev/null 2020-11-16T22:03:30.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj110 /dev/null 2020-11-16T22:03:30.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj111 /dev/null 2020-11-16T22:03:30.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj112 /dev/null 2020-11-16T22:03:30.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj113 /dev/null 2020-11-16T22:03:30.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj114 /dev/null 2020-11-16T22:03:30.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:30.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj115 /dev/null 2020-11-16T22:03:31.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj116 /dev/null 2020-11-16T22:03:31.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj117 /dev/null 2020-11-16T22:03:31.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj118 /dev/null 2020-11-16T22:03:31.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj119 /dev/null 2020-11-16T22:03:31.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj120 /dev/null 2020-11-16T22:03:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj121 /dev/null 2020-11-16T22:03:31.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj122 /dev/null 2020-11-16T22:03:31.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj123 /dev/null 2020-11-16T22:03:31.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj124 /dev/null 2020-11-16T22:03:31.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj125 /dev/null 2020-11-16T22:03:31.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj126 /dev/null 2020-11-16T22:03:31.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj127 /dev/null 2020-11-16T22:03:31.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:31.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj128 /dev/null 2020-11-16T22:03:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj129 /dev/null 2020-11-16T22:03:32.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:32.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj130 /dev/null 2020-11-16T22:03:32.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:32.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj131 /dev/null 2020-11-16T22:03:32.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:32.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj132 /dev/null 2020-11-16T22:03:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj133 /dev/null 2020-11-16T22:03:32.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:32.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj134 /dev/null 2020-11-16T22:03:32.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:32.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj135 /dev/null 2020-11-16T22:03:32.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:32.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj136 /dev/null 2020-11-16T22:03:32.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:32.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj137 /dev/null 2020-11-16T22:03:32.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:32.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj138 /dev/null 2020-11-16T22:03:32.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:32.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj139 /dev/null 2020-11-16T22:03:33.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj140 /dev/null 2020-11-16T22:03:33.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj141 /dev/null 2020-11-16T22:03:33.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj142 /dev/null 2020-11-16T22:03:33.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj143 /dev/null 2020-11-16T22:03:33.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj144 /dev/null 2020-11-16T22:03:33.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj145 /dev/null 2020-11-16T22:03:33.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj146 /dev/null 2020-11-16T22:03:33.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj147 /dev/null 2020-11-16T22:03:33.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj148 /dev/null 2020-11-16T22:03:33.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj149 /dev/null 2020-11-16T22:03:33.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj150 /dev/null 2020-11-16T22:03:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj151 /dev/null 2020-11-16T22:03:33.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:33.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj152 /dev/null 2020-11-16T22:03:34.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj153 /dev/null 2020-11-16T22:03:34.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj154 /dev/null 2020-11-16T22:03:34.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj155 /dev/null 2020-11-16T22:03:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj156 /dev/null 2020-11-16T22:03:34.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj157 /dev/null 2020-11-16T22:03:34.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj158 /dev/null 2020-11-16T22:03:34.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj159 /dev/null 2020-11-16T22:03:34.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj160 /dev/null 2020-11-16T22:03:34.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj161 /dev/null 2020-11-16T22:03:34.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj162 /dev/null 2020-11-16T22:03:34.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj163 /dev/null 2020-11-16T22:03:34.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj164 /dev/null 2020-11-16T22:03:34.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:34.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj165 /dev/null 2020-11-16T22:03:35.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj166 /dev/null 2020-11-16T22:03:35.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj167 /dev/null 2020-11-16T22:03:35.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj168 /dev/null 2020-11-16T22:03:35.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj169 /dev/null 2020-11-16T22:03:35.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj170 /dev/null 2020-11-16T22:03:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj171 /dev/null 2020-11-16T22:03:35.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj172 /dev/null 2020-11-16T22:03:35.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj173 /dev/null 2020-11-16T22:03:35.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj174 /dev/null 2020-11-16T22:03:35.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj175 /dev/null 2020-11-16T22:03:35.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj176 /dev/null 2020-11-16T22:03:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj177 /dev/null 2020-11-16T22:03:35.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:35.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj178 /dev/null 2020-11-16T22:03:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj179 /dev/null 2020-11-16T22:03:36.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj180 /dev/null 2020-11-16T22:03:36.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj181 /dev/null 2020-11-16T22:03:36.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj182 /dev/null 2020-11-16T22:03:36.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj183 /dev/null 2020-11-16T22:03:36.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj184 /dev/null 2020-11-16T22:03:36.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj185 /dev/null 2020-11-16T22:03:36.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj186 /dev/null 2020-11-16T22:03:36.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj187 /dev/null 2020-11-16T22:03:36.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj188 /dev/null 2020-11-16T22:03:36.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj189 /dev/null 2020-11-16T22:03:36.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:36.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj190 /dev/null 2020-11-16T22:03:37.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:37.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj191 /dev/null 2020-11-16T22:03:37.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:37.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj192 /dev/null 2020-11-16T22:03:37.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:37.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj193 /dev/null 2020-11-16T22:03:37.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:37.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj194 /dev/null 2020-11-16T22:03:37.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:37.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj195 /dev/null 2020-11-16T22:03:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj196 /dev/null 2020-11-16T22:03:37.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:37.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj197 /dev/null 2020-11-16T22:03:37.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:37.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj198 /dev/null 2020-11-16T22:03:37.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:37.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj199 /dev/null 2020-11-16T22:03:37.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2020-11-16T22:03:37.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj200 /dev/null 2020-11-16T22:03:37.805 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:203: TEST_backfill_sizeup_out: get_pg test obj1 2020-11-16T22:03:37.805 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2020-11-16T22:03:37.805 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2020-11-16T22:03:37.806 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2020-11-16T22:03:37.806 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2020-11-16T22:03:38.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:203: TEST_backfill_sizeup_out: local PG=1.0 2020-11-16T22:03:38.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:205: TEST_backfill_sizeup_out: get_primary test obj1 2020-11-16T22:03:38.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2020-11-16T22:03:38.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2020-11-16T22:03:38.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2020-11-16T22:03:38.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T22:03:38.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:205: TEST_backfill_sizeup_out: local primary=1 2020-11-16T22:03:38.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:207: TEST_backfill_sizeup_out: ceph osd set nobackfill 2020-11-16T22:03:39.360 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is set 2020-11-16T22:03:39.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:208: TEST_backfill_sizeup_out: ceph osd out osd.1 2020-11-16T22:03:39.905 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already out. 2020-11-16T22:03:39.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:209: TEST_backfill_sizeup_out: ceph osd pool set test size 3 2020-11-16T22:03:40.684 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 3 2020-11-16T22:03:40.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:210: TEST_backfill_sizeup_out: sleep 2 2020-11-16T22:03:42.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:211: TEST_backfill_sizeup_out: ceph osd unset nobackfill 2020-11-16T22:03:43.416 INFO:tasks.workunit.client.0.smithi189.stderr:nobackfill is unset 2020-11-16T22:03:43.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:213: TEST_backfill_sizeup_out: wait_for_clean 2020-11-16T22:03:43.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:03:43.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:03:43.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:03:43.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:03:43.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:03:43.432 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:03:43.432 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:03:43.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:03:43.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:03:43.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:03:43.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:03:43.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:03:43.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:03:43.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:03:43.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:03:44.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:03:44.032 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:03:44.032 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:03:44.032 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:03:44.033 INFO:tasks.workunit.client.0.smithi189.stderr:4 2020-11-16T22:03:44.033 INFO:tasks.workunit.client.0.smithi189.stderr:5' 2020-11-16T22:03:44.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:03:44.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:44.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:03:44.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836498 2020-11-16T22:03:44.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836498 2020-11-16T22:03:44.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498' 2020-11-16T22:03:44.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:44.193 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:03:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705682 2020-11-16T22:03:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705682 2020-11-16T22:03:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-38654705682' 2020-11-16T22:03:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:44.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:03:44.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574864 2020-11-16T22:03:44.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574864 2020-11-16T22:03:44.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-38654705682 2-55834574864' 2020-11-16T22:03:44.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:44.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:03:44.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=73014444046 2020-11-16T22:03:44.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 73014444046 2020-11-16T22:03:44.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-38654705682 2-55834574864 3-73014444046' 2020-11-16T22:03:44.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:44.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:03:44.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=94489280524 2020-11-16T22:03:44.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 94489280524 2020-11-16T22:03:44.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-38654705682 2-55834574864 3-73014444046 4-94489280524' 2020-11-16T22:03:44.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:03:44.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-11-16T22:03:45.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964117004 2020-11-16T22:03:45.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964117004 2020-11-16T22:03:45.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-38654705682 2-55834574864 3-73014444046 4-94489280524 5-115964117004' 2020-11-16T22:03:45.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:45.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836498 2020-11-16T22:03:45.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:45.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:03:45.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836498 2020-11-16T22:03:45.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:45.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836498 2020-11-16T22:03:45.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2020-11-16T22:03:45.035 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836498 2020-11-16T22:03:45.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:03:45.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836498 -lt 21474836498 2020-11-16T22:03:45.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:45.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705682 2020-11-16T22:03:45.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:45.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:03:45.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705682 2020-11-16T22:03:45.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:45.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705682 2020-11-16T22:03:45.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705682' 2020-11-16T22:03:45.430 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705682 2020-11-16T22:03:45.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:03:45.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705682 -lt 38654705682 2020-11-16T22:03:45.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:45.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-55834574864 2020-11-16T22:03:45.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:45.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:03:45.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-55834574864 2020-11-16T22:03:45.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:45.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574864 2020-11-16T22:03:45.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 55834574864' 2020-11-16T22:03:45.859 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 55834574864 2020-11-16T22:03:45.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:03:46.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574864 -lt 55834574864 2020-11-16T22:03:46.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:46.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-73014444046 2020-11-16T22:03:46.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:46.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:03:46.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-73014444046 2020-11-16T22:03:46.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:46.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=73014444046 2020-11-16T22:03:46.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 73014444046' 2020-11-16T22:03:46.259 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 73014444046 2020-11-16T22:03:46.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:03:46.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 73014444046 -lt 73014444046 2020-11-16T22:03:46.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:46.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-94489280524 2020-11-16T22:03:46.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:46.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:03:46.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:46.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-94489280524 2020-11-16T22:03:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=94489280524 2020-11-16T22:03:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 94489280524' 2020-11-16T22:03:46.662 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 94489280524 2020-11-16T22:03:46.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:03:47.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 94489280525 -lt 94489280524 2020-11-16T22:03:47.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:03:47.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-115964117004 2020-11-16T22:03:47.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:03:47.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-11-16T22:03:47.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-115964117004 2020-11-16T22:03:47.057 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:03:47.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964117004 2020-11-16T22:03:47.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 115964117004' 2020-11-16T22:03:47.059 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.5 seq 115964117004 2020-11-16T22:03:47.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-11-16T22:03:47.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964117004 -lt 115964117004 2020-11-16T22:03:47.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:03:47.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:47.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:47.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T22:03:47.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:03:47.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:03:47.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:03:47.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:03:47.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:03:47.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:03:47.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:03:48.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:03:48.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:03:48.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:48.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:48.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:03:48.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T22:03:48.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T22:03:48.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T22:03:48.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:03:48.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:03:48.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:03:48.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:03:48.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:03:48.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:03:48.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:03:48.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:03:48.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:03:48.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:03:49.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:03:49.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:03:49.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:49.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:49.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:03:49.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:03:49.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:03:49.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:03:49.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:03:49.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:03:49.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:03:49.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:03:49.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:03:50.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T22:03:50.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T22:03:50.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:03:50.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:03:50.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:03:50.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:03:50.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:03:50.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:03:50.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:03:50.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:03:50.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:03:50.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:03:50.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:03:50.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:03:50.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:03:50.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:50.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:51.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:03:51.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:03:51.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:03:51.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:03:51.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:03:51.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:03:51.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:03:51.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:03:51.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:03:51.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T22:03:51.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T22:03:51.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:03:51.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:03:51.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:03:51.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:03:51.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:03:51.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:03:51.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:03:51.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:03:51.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:03:51.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:03:51.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:03:52.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:03:52.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:03:52.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:52.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:52.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:03:52.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:03:52.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:03:52.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:03:52.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:03:52.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:03:52.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:03:52.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:03:52.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:03:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T22:03:53.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T22:03:53.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:03:53.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:03:53.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:03:53.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:03:53.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:03:53.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:03:53.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:03:53.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:03:53.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:03:53.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:03:53.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:03:53.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:03:53.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:03:53.782 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:53.782 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:54.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:03:54.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:03:54.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:03:54.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:03:54.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:03:54.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:03:54.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:03:54.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:03:54.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:03:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T22:03:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T22:03:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:03:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:03:54.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:03:54.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:03:54.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:03:54.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:03:54.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:03:54.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:03:54.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:03:54.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:03:54.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:03:55.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:03:55.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:03:55.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:55.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:55.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:03:55.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:03:55.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:03:55.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:03:55.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:03:55.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:03:55.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:03:55.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:03:55.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:03:56.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:03:56.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:03:56.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:03:56.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:03:56.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:03:56.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:03:56.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:03:56.308 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:03:56.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:03:56.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:03:56.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:03:56.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:03:56.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:03:56.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:03:56.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:03:56.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:56.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:57.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:03:57.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:03:57.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:03:57.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:03:57.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:03:57.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:03:57.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:03:57.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:03:57.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:03:57.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:03:57.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:03:57.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:03:57.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:03:57.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:03:57.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:03:57.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:03:57.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:03:57.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:03:57.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:03:57.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:03:57.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:03:57.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:03:58.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:03:58.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:03:58.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:58.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:58.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:03:58.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:03:58.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:03:58.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:03:58.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:03:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:03:58.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:03:58.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:03:58.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:03:59.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:03:59.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:03:59.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:03:59.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:03:59.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:03:59.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:03:59.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:03:59.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:03:59.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:03:59.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:03:59.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:03:59.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:03:59.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:03:59.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:03:59.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:03:59.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:03:59.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:03:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:03:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:03:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:03:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:03:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:03:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:03:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:03:59.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:03:59.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:00.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:04:00.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:04:00.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:00.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:00.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:00.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:00.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:00.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:00.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:00.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:00.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:00.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:00.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:00.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:00.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:00.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:00.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:01.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:01.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:01.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:01.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:01.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:01.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:01.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:01.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:01.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:01.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:04:01.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:04:01.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:01.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:01.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:02.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:02.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:02.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:02.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:02.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:02.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:02.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:02.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:02.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:02.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:02.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:02.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:02.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:02.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:03.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:04:03.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:04:03.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:03.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:03.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:03.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:03.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:03.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:03.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:03.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:03.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:03.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:03.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:03.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:03.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:03.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:03.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:04.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:04.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:04.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:04.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:04.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:04.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:04.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:04.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:04.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:04.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:04:04.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:04:04.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:04.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:04.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:04.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:04.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:04.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:04.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:04.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:04.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:04.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:04.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:05.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:05.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:05.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:05.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:05.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:05.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:05.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:05.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:05.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:05.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:05.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:05.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:05.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:06.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T22:04:06.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T22:04:06.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:06.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:06.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:06.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:06.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:06.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:06.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:06.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:06.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:06.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:06.307 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:06.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:06.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:06.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:06.692 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:07.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:07.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:07.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:07.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:07.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:07.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:07.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:07.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:07.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:07.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2020-11-16T22:04:07.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2020-11-16T22:04:07.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:07.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:07.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:07.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:07.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:07.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:07.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:07.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:07.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:07.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:07.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:08.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:08.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:08.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:08.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:08.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:08.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:08.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:08.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:08.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:08.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:08.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:08.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:08.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:09.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T22:04:09.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T22:04:09.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:09.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:09.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:09.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:09.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:09.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:09.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:09.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:09.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:09.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:09.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:09.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:09.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:09.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:09.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:10.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:10.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:10.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:10.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:10.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:10.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:10.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:10.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:10.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T22:04:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T22:04:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:10.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:10.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:10.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:10.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:10.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:10.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:10.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:10.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:11.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:11.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:11.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:11.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:11.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:11.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:12.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:04:12.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:04:12.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:12.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:12.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:12.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:12.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:12.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:12.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:12.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:12.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:12.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:12.149 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:12.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:12.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:12.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:12.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:12.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:12.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:12.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:12.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:12.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:12.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:12.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:12.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:12.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:13.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2020-11-16T22:04:13.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2020-11-16T22:04:13.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:13.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:13.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:13.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:13.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:13.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:13.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:13.591 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:13.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:13.592 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:13.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:13.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:13.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:13.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:14.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:14.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:14.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:14.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:14.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:14.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:14.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:14.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:04:14.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:04:14.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:14.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:14.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:15.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:15.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:15.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:15.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:15.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:15.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:15.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:15.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:15.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:15.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:15.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:15.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:15.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:15.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:15.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:15.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:15.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:04:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:04:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:16.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:16.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:16.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:16.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:16.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:16.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:16.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:16.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:16.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:16.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:16.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:16.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:17.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:17.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:17.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:17.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:17.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:17.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:17.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:17.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:17.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:17.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:04:17.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:04:17.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:17.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:17.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:17.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:17.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:17.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:17.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:17.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:17.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:17.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:17.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:18.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:18.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:18.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:18.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:18.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:18.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:18.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:18.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:18.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:18.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:18.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:18.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:18.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:19.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:04:19.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:04:19.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:19.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:19.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:19.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:19.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:19.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:19.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:19.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:19.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:19.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:19.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:19.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:19.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:19.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:19.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:20.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:20.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:20.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:20.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:20.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:20.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:20.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:20.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:20.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:20.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:04:20.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:04:20.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:20.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:20.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:20.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:20.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:20.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:20.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:20.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:20.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:20.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:20.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:21.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:21.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:21.239 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:21.239 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:21.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:21.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:21.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:21.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:22.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:04:22.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:04:22.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:22.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:22.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:22.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:22.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:22.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:22.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:22.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:22.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:22.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:22.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:22.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:22.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:22.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:22.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:23.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:23.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:23.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:23.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:23.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:23.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:23.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:23.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:23.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:23.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2020-11-16T22:04:23.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2020-11-16T22:04:23.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:23.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:23.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:23.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:23.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:23.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:23.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:23.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:23.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:23.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:24.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:24.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:24.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:24.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:24.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:24.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:24.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:24.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:24.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:24.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:24.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:25.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:04:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:04:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:25.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:25.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:25.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:25.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:25.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:25.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:25.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:25.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:25.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:25.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:25.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:25.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:26.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:26.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:26.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:26.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:26.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:26.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:26.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:26.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:26.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:26.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2020-11-16T22:04:26.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2020-11-16T22:04:26.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:26.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:26.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:26.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:26.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:26.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:26.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:26.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:26.743 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:26.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:26.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:27.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:27.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:27.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:27.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:27.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:27.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:27.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:27.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:27.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:27.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:27.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:27.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:27.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:28.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T22:04:28.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T22:04:28.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:28.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:28.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:28.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:28.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:28.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:28.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:28.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:28.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:28.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:28.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:28.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:28.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:28.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:28.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:29.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:29.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:29.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:29.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:29.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2020-11-16T22:04:29.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2020-11-16T22:04:29.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:29.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:29.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:29.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:29.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:29.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:29.634 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:29.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:29.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:29.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:29.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:30.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:04:30.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:30.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:30.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:30.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2020-11-16T22:04:30.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:04:30.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:04:30.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:04:30.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:04:30.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:04:30.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:04:30.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:04:30.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:04:31.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=1 2020-11-16T22:04:31.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 1 '!=' null 2020-11-16T22:04:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:04:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:04:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:04:31.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:04:31.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:04:31.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:04:31.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:04:31.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:04:31.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:04:31.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:04:31.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:04:31.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:04:31.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:04:31.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:04:31.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:04:31.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T22:04:31.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:04:31.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:04:31.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:215: TEST_backfill_sizeup_out: expr 200 '*' 2 2020-11-16T22:04:31.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:215: TEST_backfill_sizeup_out: local degraded=400 2020-11-16T22:04:31.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:216: TEST_backfill_sizeup_out: check td/osd-backfill-stats 1.0 1 replicated 400 0 200 0 2020-11-16T22:04:31.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2020-11-16T22:04:31.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2020-11-16T22:04:31.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2020-11-16T22:04:31.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2020-11-16T22:04:31.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=400 2020-11-16T22:04:31.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2020-11-16T22:04:31.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=200 2020-11-16T22:04:31.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2020-11-16T22:04:31.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2020-11-16T22:04:31.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2020-11-16T22:04:31.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2020-11-16T22:04:31.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2020-11-16T22:04:31.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2020-11-16T22:04:31.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2020-11-16T22:04:31.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2020-11-16T22:04:32.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2020-11-16T22:04:32.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2020-11-16T22:04:32.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2020-11-16T22:04:32.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2020-11-16T22:04:32.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2020-11-16T22:04:32.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2020-11-16T22:04:32.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2020-11-16T22:04:32.015 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:04:32.015 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2020-11-16T22:04:32.016 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2020-11-16T22:04:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[0,2,4]/[1]' 2020-11-16T22:04:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[0,2,4]/[1]' ']' 2020-11-16T22:04:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2020-11-16T22:04:32.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:04:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:04:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [0,2,4]/[1] ' 2020-11-16T22:04:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2020-11-16T22:04:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2020-11-16T22:04:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:04:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=400 2020-11-16T22:04:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 400 ']' 2020-11-16T22:04:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 400 400 2020-11-16T22:04:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2020-11-16T22:04:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T22:04:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2020-11-16T22:04:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T22:04:32.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:04:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:04:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2020-11-16T22:04:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:04:32.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=2 2020-11-16T22:04:32.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 2 ']' 2020-11-16T22:04:32.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 2 0 2020-11-16T22:04:32.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=2 2020-11-16T22:04:32.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T22:04:32.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T22:04:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T22:04:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2020-11-16T22:04:32.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:04:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:04:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2020-11-16T22:04:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [0,2,4]/[1] ' 2020-11-16T22:04:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2020-11-16T22:04:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:04:32.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=200 2020-11-16T22:04:32.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 200 ']' 2020-11-16T22:04:32.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 200 200 2020-11-16T22:04:32.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2020-11-16T22:04:32.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2020-11-16T22:04:32.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2020-11-16T22:04:32.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2020-11-16T22:04:32.074 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2020-11-16T22:04:32.074 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2020-11-16T22:04:32.074 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2020-11-16T22:04:32.074 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2020-11-16T22:04:32.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=1 2020-11-16T22:04:32.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 1 ']' 2020-11-16T22:04:32.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 1 0 2020-11-16T22:04:32.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2020-11-16T22:04:32.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2020-11-16T22:04:32.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2020-11-16T22:04:32.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2020-11-16T22:04:32.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2020-11-16T22:04:32.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:218: TEST_backfill_sizeup_out: delete_pool test 2020-11-16T22:04:32.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2020-11-16T22:04:32.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-11-16T22:04:32.615 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test' does not exist 2020-11-16T22:04:32.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:219: TEST_backfill_sizeup_out: kill_daemons td/osd-backfill-stats 2020-11-16T22:04:32.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:04:32.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:04:32.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:04:32.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:04:32.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:04:32.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:04:32.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2020-11-16T22:04:32.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T22:04:32.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:04:32.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T22:04:32.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:04:32.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:04:32.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:04:32.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:04:32.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:04:32.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:04:32.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:04:32.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:04:32.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:04:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:04:32.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:04:32.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:04:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:04:32.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:04:32.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:04:32.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:04:32.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:04:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:04:32.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T22:04:32.789 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:04:32.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:04:32.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:04:32.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T22:04:32.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:04:32.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:04:32.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T22:04:32.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-backfill-stats 0 2020-11-16T22:04:32.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2020-11-16T22:04:32.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T22:04:32.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2020-11-16T22:04:32.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:04:32.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:04:32.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:04:32.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:04:32.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:04:32.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:04:32.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:04:32.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:04:32.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:04:32.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:04:32.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:04:32.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:04:32.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:04:32.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:04:32.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:04:32.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:04:32.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:04:32.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T22:04:32.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2020-11-16T22:04:32.806 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:04:32.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:04:32.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.465868 2020-11-16T22:04:32.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.465868 2020-11-16T22:04:32.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:04:32.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:04:32.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T22:04:32.812 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T22:04:32.813 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T22:04:32.866 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2020-11-16T22:04:32.867 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2020-11-16T22:04:32.927 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T22:04:32.939 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T22:04:32.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:04:32.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:04:32.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T22:04:32.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T22:04:32.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T22:04:32.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T22:04:32.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T22:04:32.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T22:04:32.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-bench 2020-11-16T22:04:32.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2020-11-16T22:04:32.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2020-11-16T22:04:32.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2020-11-16T22:04:32.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2020-11-16T22:04:32.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2020-11-16T22:04:32.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2020-11-16T22:04:32.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=f80be67a-325f-464b-8534-b638143f539c --auth-supported=none ' 2020-11-16T22:04:32.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2020-11-16T22:04:32.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2020-11-16T22:04:32.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2020-11-16T22:04:32.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T22:04:32.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2020-11-16T22:04:32.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2020-11-16T22:04:32.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2020-11-16T22:04:32.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2020-11-16T22:04:32.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2020-11-16T22:04:32.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2020-11-16T22:04:32.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:04:32.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2020-11-16T22:04:32.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:04:32.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:04:32.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:04:32.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:04:32.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:04:32.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:04:32.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:04:32.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:04:32.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:04:32.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:04:32.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:04:32.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:04:32.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:04:32.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:04:32.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:04:32.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:04:32.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:04:32.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:04:32.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2020-11-16T22:04:32.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:04:32.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:04:32.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.610845 2020-11-16T22:04:32.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.610845 2020-11-16T22:04:32.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:04:32.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:04:32.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2020-11-16T22:04:32.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:04:32.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:04:32.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.610845 2020-11-16T22:04:32.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.610845 2020-11-16T22:04:32.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:04:32.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T22:04:32.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T22:04:32.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:04:32.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2020-11-16T22:04:32.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2020-11-16T22:04:32.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2020-11-16T22:04:32.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2020-11-16T22:04:32.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2020-11-16T22:04:32.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:04:32.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:04:32.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:04:32.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2020-11-16T22:04:32.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2020-11-16T22:04:33.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:04:33.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:04:33.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:04:33.250 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:04:33.250 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:04:33.250 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.610845 2020-11-16T22:04:33.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.610845/$cluster-$name.asok' 2020-11-16T22:04:33.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.610845/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:04:33.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:04:33.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:04:33.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:04:33.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:04:33.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:04:33.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:04:33.446 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:04:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:04:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:04:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:04:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:04:33.447 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.610845 2020-11-16T22:04:33.448 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.610845/ceph-mon.a.asok 2020-11-16T22:04:33.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:04:33.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.610845/ceph-mon.a.asok config get fsid 2020-11-16T22:04:33.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:04:33.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:04:33.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:04:33.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:04:33.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:04:33.555 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:04:33.555 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:04:33.555 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:04:33.556 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:04:33.556 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:04:33.556 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.610845 2020-11-16T22:04:33.556 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.610845/ceph-mon.a.asok 2020-11-16T22:04:33.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:04:33.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.610845/ceph-mon.a.asok config get mon_host 2020-11-16T22:04:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2020-11-16T22:04:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-bench 2020-11-16T22:04:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:04:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:04:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:04:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-bench/x 2020-11-16T22:04:33.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:04:34.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:04:34.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:04:34.289 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:04:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:04:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:04:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.610845 2020-11-16T22:04:34.290 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.610845/$cluster-$name.asok' 2020-11-16T22:04:34.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:04:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.610845/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:04:34.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2020-11-16T22:04:34.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-bench 2020-11-16T22:04:34.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:04:34.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:04:34.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:04:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-bench/0 2020-11-16T22:04:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f80be67a-325f-464b-8534-b638143f539c --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2020-11-16T22:04:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:04:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:04:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:04:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2020-11-16T22:04:34.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2020-11-16T22:04:34.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:04:34.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:04:34.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2020-11-16T22:04:34.317 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:04:34.317 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:04:34.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:04:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:04:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:04:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.610845 2020-11-16T22:04:34.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.610845/$cluster-$name.asok' 2020-11-16T22:04:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.610845/$cluster-$name.asok' 2020-11-16T22:04:34.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:04:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:04:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:04:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2020-11-16T22:04:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2020-11-16T22:04:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:04:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:04:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:04:34.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:04:34.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:04:34.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-bench/0 2020-11-16T22:04:34.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:04:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=94e8b0bd-9e9c-47ff-b35e-b31dcdfa2bc1 2020-11-16T22:04:34.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 94e8b0bd-9e9c-47ff-b35e-b31dcdfa2bc1' 2020-11-16T22:04:34.321 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 94e8b0bd-9e9c-47ff-b35e-b31dcdfa2bc1 2020-11-16T22:04:34.321 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:04:34.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBy97JfspLEFBAABWdF7AzdmWHy2Xig7LbeBw== 2020-11-16T22:04:34.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBy97JfspLEFBAABWdF7AzdmWHy2Xig7LbeBw=="}' 2020-11-16T22:04:34.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 94e8b0bd-9e9c-47ff-b35e-b31dcdfa2bc1 -i td/osd-bench/0/new.json 2020-11-16T22:04:34.675 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:04:34.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-bench/0/new.json 2020-11-16T22:04:34.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=f80be67a-325f-464b-8534-b638143f539c --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.610845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBy97JfspLEFBAABWdF7AzdmWHy2Xig7LbeBw== --osd-uuid 94e8b0bd-9e9c-47ff-b35e-b31dcdfa2bc1 2020-11-16T22:04:34.716 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:34.720+0000 7fabb08c2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:04:34.718 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:34.724+0000 7fabb08c2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:04:34.720 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:34.728+0000 7fabb08c2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:04:34.721 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:34.728+0000 7fabb08c2d80 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2020-11-16T22:04:34.721 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:34.728+0000 7fabb08c2d80 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2020-11-16T22:04:34.721 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:34.728+0000 7fabb08c2d80 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2020-11-16T22:04:34.752 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:34.760+0000 7fabb08c2d80 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2020-11-16T22:04:39.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-bench/0/keyring 2020-11-16T22:04:39.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:04:39.071 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:04:39.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:04:39.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:04:39.574 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:04:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:04:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:04:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:04:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:04:39.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=f80be67a-325f-464b-8534-b638143f539c --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.610845/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:04:39.599 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:39.604+0000 7f165e727d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:04:39.601 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:39.608+0000 7f165e727d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:04:39.602 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:39.608+0000 7f165e727d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:04:39.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:04:39.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:04:39.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:04:39.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:04:39.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:04:39.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:04:39.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:04:39.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:04:39.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:04:39.977 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:04:40.142 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:40.148+0000 7f165e727d80 -1 Falling back to public interface 2020-11-16T22:04:40.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:04:40.973 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:04:40.980+0000 7f165e727d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:04:41.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:04:41.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:04:41.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:04:41.344 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:04:41.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:04:41.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:04:41.730 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/611698,v1:127.0.0.1:6803/611698] [v2:127.0.0.1:6804/611698,v1:127.0.0.1:6805/611698] exists,up 94e8b0bd-9e9c-47ff-b35e-b31dcdfa2bc1 2020-11-16T22:04:41.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:04:41.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:04:41.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:04:41.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2020-11-16T22:04:41.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2020-11-16T22:04:41.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2020-11-16T22:04:41.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2020-11-16T22:04:41.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2020-11-16T22:04:41.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2020-11-16T22:04:41.757 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2020-11-16T22:04:41.757 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2020-11-16T22:04:41.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2020-11-16T22:04:41.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2020-11-16T22:04:41.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2020-11-16T22:04:41.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2020-11-16T22:04:41.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2020-11-16T22:04:41.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: expect_failure: local dir=td/osd-bench 2020-11-16T22:04:41.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: shift 2020-11-16T22:04:41.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: local expected=osd_bench_max_block_size 2020-11-16T22:04:41.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: shift 2020-11-16T22:04:41.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: local success 2020-11-16T22:04:41.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: expect_failure: ceph tell osd.0 bench 1024 67108865 2020-11-16T22:04:41.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: expect_failure: success=false 2020-11-16T22:04:41.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: false 2020-11-16T22:04:41.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2020-11-16T22:04:41.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: return 0 2020-11-16T22:04:41.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2020-11-16T22:04:41.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2020-11-16T22:04:41.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2020-11-16T22:04:41.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: expect_failure: local dir=td/osd-bench 2020-11-16T22:04:41.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: shift 2020-11-16T22:04:41.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: local expected=bench_small_size_max_iops 2020-11-16T22:04:41.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: shift 2020-11-16T22:04:41.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: local success 2020-11-16T22:04:41.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: expect_failure: ceph tell osd.0 bench 3072001 1024 2020-11-16T22:04:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: expect_failure: success=false 2020-11-16T22:04:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: false 2020-11-16T22:04:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2020-11-16T22:04:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: return 0 2020-11-16T22:04:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2020-11-16T22:04:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2020-11-16T22:04:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2020-11-16T22:04:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: expect_failure: local dir=td/osd-bench 2020-11-16T22:04:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: shift 2020-11-16T22:04:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: local expected=osd_bench_large_size_max_throughput 2020-11-16T22:04:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: shift 2020-11-16T22:04:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: local success 2020-11-16T22:04:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2020-11-16T22:04:42.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: expect_failure: success=false 2020-11-16T22:04:42.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: false 2020-11-16T22:04:42.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2020-11-16T22:04:42.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: return 0 2020-11-16T22:04:42.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2020-11-16T22:05:54.473 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:05:54.473 INFO:tasks.workunit.client.0.smithi189.stdout: "bytes_written": 1073741824, 2020-11-16T22:05:54.473 INFO:tasks.workunit.client.0.smithi189.stdout: "blocksize": 4194304, 2020-11-16T22:05:54.474 INFO:tasks.workunit.client.0.smithi189.stdout: "elapsed_sec": 42.664437528999997, 2020-11-16T22:05:54.474 INFO:tasks.workunit.client.0.smithi189.stdout: "bytes_per_sec": 25167138.867590908, 2020-11-16T22:05:54.474 INFO:tasks.workunit.client.0.smithi189.stdout: "iops": 6.000313488862731 2020-11-16T22:05:54.474 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:05:54.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2020-11-16T22:05:56.391 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:05:56.392 INFO:tasks.workunit.client.0.smithi189.stdout: "bytes_written": 10, 2020-11-16T22:05:56.392 INFO:tasks.workunit.client.0.smithi189.stdout: "blocksize": 4444, 2020-11-16T22:05:56.392 INFO:tasks.workunit.client.0.smithi189.stdout: "elapsed_sec": 0.33341587, 2020-11-16T22:05:56.392 INFO:tasks.workunit.client.0.smithi189.stdout: "bytes_per_sec": 29.99257353886604, 2020-11-16T22:05:56.392 INFO:tasks.workunit.client.0.smithi189.stdout: "iops": 0.0067490039466395228 2020-11-16T22:05:56.393 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:05:56.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2020-11-16T22:05:57.301 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:05:57.301 INFO:tasks.workunit.client.0.smithi189.stdout: "bytes_written": 1, 2020-11-16T22:05:57.301 INFO:tasks.workunit.client.0.smithi189.stdout: "blocksize": 14456, 2020-11-16T22:05:57.302 INFO:tasks.workunit.client.0.smithi189.stdout: "elapsed_sec": 0.125096123, 2020-11-16T22:05:57.302 INFO:tasks.workunit.client.0.smithi189.stdout: "bytes_per_sec": 7.9938528550561072, 2020-11-16T22:05:57.302 INFO:tasks.workunit.client.0.smithi189.stdout: "iops": 0.00055297819971334447 2020-11-16T22:05:57.302 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:05:57.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2020-11-16T22:05:57.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2020-11-16T22:05:57.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:05:57.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2020-11-16T22:05:57.317 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:05:57.317 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:05:57.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:05:57.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:05:57.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:05:57.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:05:57.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:05:57.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:05:57.633 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:05:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:05:57.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:05:57.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:05:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:05:57.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:05:57.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:05:57.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:05:57.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:05:57.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:05:57.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2020-11-16T22:05:57.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:05:57.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:05:57.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.610845 2020-11-16T22:05:57.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.610845 2020-11-16T22:05:57.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:05:57.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:05:57.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T22:05:57.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-bench 0 2020-11-16T22:05:57.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2020-11-16T22:05:57.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T22:05:57.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2020-11-16T22:05:57.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:05:57.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:05:57.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:05:57.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:05:57.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:05:57.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:05:57.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:05:57.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:05:57.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:05:57.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:05:57.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:05:57.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:05:57.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:05:57.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:05:57.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:05:57.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:05:57.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:05:57.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T22:05:57.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2020-11-16T22:05:57.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:05:57.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:05:57.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.610845 2020-11-16T22:05:57.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.610845 2020-11-16T22:05:57.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:05:57.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:05:57.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T22:05:57.687 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T22:05:57.687 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T22:05:57.746 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2020-11-16T22:05:57.747 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2020-11-16T22:05:57.806 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T22:05:57.821 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T22:05:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:05:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:05:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T22:05:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T22:05:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T22:05:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T22:05:57.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T22:05:57.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T22:05:57.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-bluefs-volume-ops 2020-11-16T22:05:57.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:05:57.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2020-11-16T22:05:57.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: export CEPH_MON=127.0.0.1:7146 2020-11-16T22:05:57.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: CEPH_MON=127.0.0.1:7146 2020-11-16T22:05:57.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: export CEPH_ARGS 2020-11-16T22:05:57.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: uuidgen 2020-11-16T22:05:57.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: CEPH_ARGS+='--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none ' 2020-11-16T22:05:57.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2020-11-16T22:05:57.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2020-11-16T22:05:57.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:16: run: CEPH_ARGS+='--bluestore_block_db_create=true ' 2020-11-16T22:05:57.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:17: run: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2020-11-16T22:05:57.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:18: run: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2020-11-16T22:05:57.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:19: run: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2020-11-16T22:05:57.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: run: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2020-11-16T22:05:57.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: set 2020-11-16T22:05:57.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T22:05:57.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: local funcs=TEST_bluestore 2020-11-16T22:05:57.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: run: for func in $funcs 2020-11-16T22:05:57.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: run: setup td/osd-bluefs-volume-ops 2020-11-16T22:05:57.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:05:57.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2020-11-16T22:05:57.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:05:57.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:05:57.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2020-11-16T22:05:57.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:05:57.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:05:57.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:05:57.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:05:57.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:05:57.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:05:57.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:05:57.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:05:57.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:05:57.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:05:57.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:05:57.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:05:57.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:05:57.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:05:57.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:05:57.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:05:57.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:05:57.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:05:57.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2020-11-16T22:05:57.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:05:57.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:05:57.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:05:57.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.612441 2020-11-16T22:05:57.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:05:57.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:05:57.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2020-11-16T22:05:57.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:05:57.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:05:57.846 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:05:57.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.612441 2020-11-16T22:05:57.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:05:57.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T22:05:57.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T22:05:57.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:05:57.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2020-11-16T22:05:57.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:24: run: TEST_bluestore td/osd-bluefs-volume-ops 2020-11-16T22:05:57.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:05:57.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: ulimit -n 2020-11-16T22:05:57.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: local flimit=4096 2020-11-16T22:05:57.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: '[' 4096 -lt 1536 ']' 2020-11-16T22:05:57.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2020-11-16T22:05:57.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:05:57.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:05:57.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:05:57.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:05:57.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2020-11-16T22:05:57.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2020-11-16T22:05:58.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:05:58.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:05:58.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:05:58.164 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:05:58.165 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:05:58.165 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:05:58.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:05:58.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:05:58.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:05:58.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:05:58.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:05:58.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:05:58.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:05:58.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:05:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:05:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:05:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:05:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:05:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:05:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:05:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.612441/ceph-mon.a.asok 2020-11-16T22:05:58.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:05:58.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.612441/ceph-mon.a.asok config get fsid 2020-11-16T22:05:58.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:05:58.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:05:58.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:05:58.474 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:05:58.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:05:58.476 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:05:58.476 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:05:58.476 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:05:58.476 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:05:58.476 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:05:58.476 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:05:58.477 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.612441/ceph-mon.a.asok 2020-11-16T22:05:58.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:05:58.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.612441/ceph-mon.a.asok config get mon_host 2020-11-16T22:05:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2020-11-16T22:05:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:05:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:05:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:05:58.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:05:58.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-bluefs-volume-ops/x 2020-11-16T22:05:58.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:05:59.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:05:59.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:05:59.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:05:59.142 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:05:59.142 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:05:59.142 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:05:59.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:05:59.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:05:59.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:05:59.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2020-11-16T22:05:59.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:05:59.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:05:59.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:05:59.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:05:59.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2020-11-16T22:05:59.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:05:59.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:05:59.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:05:59.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:05:59.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2020-11-16T22:05:59.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2020-11-16T22:05:59.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:05:59.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:05:59.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:05:59.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:05:59.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:05:59.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:05:59.171 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:05:59.171 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:05:59.172 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:05:59.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:05:59.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:05:59.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:05:59.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:05:59.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:05:59.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:05:59.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:05:59.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:05:59.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:05:59.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:05:59.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:05:59.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:05:59.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2020-11-16T22:05:59.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:05:59.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1f6242ad-048f-45f5-818b-c23c7b6373fd 2020-11-16T22:05:59.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 1f6242ad-048f-45f5-818b-c23c7b6373fd' 2020-11-16T22:05:59.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:05:59.175 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 1f6242ad-048f-45f5-818b-c23c7b6373fd 2020-11-16T22:05:59.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDH97JfPIGzCxAApPl/TyDg22P8DJDzOxMJxw== 2020-11-16T22:05:59.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDH97JfPIGzCxAApPl/TyDg22P8DJDzOxMJxw=="}' 2020-11-16T22:05:59.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1f6242ad-048f-45f5-818b-c23c7b6373fd -i td/osd-bluefs-volume-ops/0/new.json 2020-11-16T22:05:59.523 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:05:59.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2020-11-16T22:05:59.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDH97JfPIGzCxAApPl/TyDg22P8DJDzOxMJxw== --osd-uuid 1f6242ad-048f-45f5-818b-c23c7b6373fd 2020-11-16T22:05:59.563 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:05:59.570+0000 7f698457dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:05:59.566 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:05:59.574+0000 7f698457dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:05:59.567 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:05:59.574+0000 7f698457dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:05:59.568 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:05:59.574+0000 7f698457dd80 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2020-11-16T22:05:59.568 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:05:59.574+0000 7f698457dd80 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2020-11-16T22:05:59.568 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:05:59.574+0000 7f698457dd80 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2020-11-16T22:05:59.609 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:05:59.618+0000 7f698457dd80 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2020-11-16T22:06:07.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2020-11-16T22:06:07.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:06:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:06:07.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:06:07.748 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:06:08.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:06:08.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:06:08.254 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:06:08.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:06:08.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:06:08.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:06:08.274 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:08.282+0000 7f4d1f858d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:08.276 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:08.282+0000 7f4d1f858d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:08.277 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:08.286+0000 7f4d1f858d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:08.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:06:08.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:06:08.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:06:08.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:06:08.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:06:08.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:08.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:06:08.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:08.630 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:06:08.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:06:08.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:09.327 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:09.334+0000 7f4d1f858d80 -1 Falling back to public interface 2020-11-16T22:06:09.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:09.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:09.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:06:09.980 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:06:09.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:09.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:06:10.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:11.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:11.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:11.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:06:11.362 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:06:11.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:11.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:06:11.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:12.455 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:12.454+0000 7f4d1f858d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:06:12.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:12.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:12.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:06:12.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:12.880 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:06:12.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:06:13.263 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/613573,v1:127.0.0.1:6803/613573] [v2:127.0.0.1:6804/613573,v1:127.0.0.1:6805/613573] exists,up 1f6242ad-048f-45f5-818b-c23c7b6373fd 2020-11-16T22:06:13.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:06:13.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:06:13.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:06:13.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2020-11-16T22:06:13.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=613573 2020-11-16T22:06:13.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2020-11-16T22:06:13.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:06:13.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:06:13.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:06:13.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:06:13.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2020-11-16T22:06:13.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:06:13.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:06:13.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:06:13.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:06:13.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2020-11-16T22:06:13.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2020-11-16T22:06:13.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:06:13.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:06:13.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:06:13.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:06:13.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:06:13.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:06:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:06:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:06:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:06:13.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:06:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:06:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:06:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:06:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:06:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:06:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:06:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:06:13.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:06:13.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:06:13.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:06:13.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:06:13.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2020-11-16T22:06:13.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:06:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=dabc47d8-e723-4845-87ab-b470967b9c5f 2020-11-16T22:06:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 dabc47d8-e723-4845-87ab-b470967b9c5f' 2020-11-16T22:06:13.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:06:13.273 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 dabc47d8-e723-4845-87ab-b470967b9c5f 2020-11-16T22:06:13.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDV97Jfxk/6ERAA6ED5f5q3ju+/XvMtGugDVg== 2020-11-16T22:06:13.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDV97Jfxk/6ERAA6ED5f5q3ju+/XvMtGugDVg=="}' 2020-11-16T22:06:13.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new dabc47d8-e723-4845-87ab-b470967b9c5f -i td/osd-bluefs-volume-ops/1/new.json 2020-11-16T22:06:13.897 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:06:13.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2020-11-16T22:06:13.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDV97Jfxk/6ERAA6ED5f5q3ju+/XvMtGugDVg== --osd-uuid dabc47d8-e723-4845-87ab-b470967b9c5f 2020-11-16T22:06:13.942 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:13.950+0000 7f5b8ab39d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:13.945 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:13.954+0000 7f5b8ab39d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:13.946 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:13.954+0000 7f5b8ab39d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:13.947 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:13.954+0000 7f5b8ab39d80 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2020-11-16T22:06:13.947 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:13.954+0000 7f5b8ab39d80 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2020-11-16T22:06:13.947 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:13.954+0000 7f5b8ab39d80 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2020-11-16T22:06:13.995 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:14.002+0000 7f5b8ab39d80 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2020-11-16T22:06:22.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2020-11-16T22:06:22.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:06:22.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:06:22.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:06:22.191 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:06:22.692 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:06:22.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:06:22.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:06:22.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:06:22.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:06:22.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:06:22.717 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:22.726+0000 7faa0163fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:22.720 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:22.726+0000 7faa0163fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:22.721 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:22.730+0000 7faa0163fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:23.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:06:23.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:06:23.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:06:23.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:06:23.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:06:23.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:23.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:06:23.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:23.075 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:06:23.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:06:23.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:23.755 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:23.762+0000 7faa0163fd80 -1 Falling back to public interface 2020-11-16T22:06:24.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:24.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:24.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:06:24.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:24.427 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:06:24.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:06:24.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:25.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:25.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:25.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:06:25.794 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:06:25.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:25.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:06:26.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:26.897 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:26.906+0000 7faa0163fd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:06:27.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:27.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:27.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:06:27.179 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:06:27.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:27.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:06:27.619 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/615166,v1:127.0.0.1:6811/615166] [v2:127.0.0.1:6812/615166,v1:127.0.0.1:6813/615166] exists,up dabc47d8-e723-4845-87ab-b470967b9c5f 2020-11-16T22:06:27.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:06:27.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:06:27.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:06:27.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2020-11-16T22:06:27.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=615166 2020-11-16T22:06:27.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2020-11-16T22:06:27.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:06:27.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:06:27.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T22:06:27.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:06:27.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2020-11-16T22:06:27.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:06:27.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:06:27.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:06:27.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:06:27.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2020-11-16T22:06:27.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2020-11-16T22:06:27.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:06:27.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:06:27.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:06:27.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:06:27.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:06:27.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:06:27.628 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:06:27.628 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:06:27.628 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:06:27.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:06:27.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:06:27.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:06:27.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:06:27.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:06:27.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:06:27.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:06:27.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:06:27.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:06:27.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:06:27.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:06:27.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:06:27.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2020-11-16T22:06:27.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:06:27.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=538b02af-9658-4dba-ac56-d474d8d8b6b0 2020-11-16T22:06:27.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 538b02af-9658-4dba-ac56-d474d8d8b6b0' 2020-11-16T22:06:27.631 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 538b02af-9658-4dba-ac56-d474d8d8b6b0 2020-11-16T22:06:27.631 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:06:27.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDj97JfIrcXJxAAhpzp52K2Xw9+y9kiv/FxZw== 2020-11-16T22:06:27.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDj97JfIrcXJxAAhpzp52K2Xw9+y9kiv/FxZw=="}' 2020-11-16T22:06:27.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 538b02af-9658-4dba-ac56-d474d8d8b6b0 -i td/osd-bluefs-volume-ops/2/new.json 2020-11-16T22:06:28.264 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:06:28.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2020-11-16T22:06:28.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDj97JfIrcXJxAAhpzp52K2Xw9+y9kiv/FxZw== --osd-uuid 538b02af-9658-4dba-ac56-d474d8d8b6b0 2020-11-16T22:06:28.301 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:28.310+0000 7f37a76e1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:28.304 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:28.310+0000 7f37a76e1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:28.305 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:28.314+0000 7f37a76e1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:28.306 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:28.314+0000 7f37a76e1d80 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2020-11-16T22:06:28.306 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:28.314+0000 7f37a76e1d80 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2020-11-16T22:06:28.306 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:28.314+0000 7f37a76e1d80 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2020-11-16T22:06:28.357 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:28.366+0000 7f37a76e1d80 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2020-11-16T22:06:36.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2020-11-16T22:06:36.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:06:36.444 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T22:06:36.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T22:06:36.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:06:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T22:06:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:06:36.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:06:36.948 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:06:36.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:06:36.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:06:36.973 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:36.982+0000 7f24f370dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:36.975 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:36.982+0000 7f24f370dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:36.977 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:36.986+0000 7f24f370dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:37.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T22:06:37.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:06:37.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:06:37.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:06:37.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:06:37.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:37.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:06:37.293 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:06:37.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:37.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:06:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:38.011 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:38.018+0000 7f24f370dd80 -1 Falling back to public interface 2020-11-16T22:06:38.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:38.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:38.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:06:38.677 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:06:38.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:38.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:06:39.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:40.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:40.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:40.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:06:40.063 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:06:40.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:40.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:06:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:41.161 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:41.166+0000 7f24f370dd80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T22:06:41.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:41.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:41.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:06:41.445 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:06:41.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:41.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:06:41.825 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/616760,v1:127.0.0.1:6819/616760] [v2:127.0.0.1:6820/616760,v1:127.0.0.1:6821/616760] exists,up 538b02af-9658-4dba-ac56-d474d8d8b6b0 2020-11-16T22:06:41.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:06:41.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:06:41.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:06:41.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2020-11-16T22:06:41.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=616760 2020-11-16T22:06:41.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2020-11-16T22:06:41.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:06:41.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:06:41.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T22:06:41.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:06:41.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2020-11-16T22:06:41.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:06:41.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:06:41.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:06:41.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:06:41.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2020-11-16T22:06:41.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2020-11-16T22:06:41.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:06:41.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:06:41.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:06:41.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:06:41.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:06:41.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:06:41.834 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:06:41.834 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:06:41.834 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:06:41.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:06:41.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:06:41.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:06:41.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:06:41.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:06:41.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:06:41.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:06:41.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:06:41.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:06:41.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:06:41.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:06:41.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:06:41.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2020-11-16T22:06:41.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:06:41.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f8e9f455-b9f3-4cfe-8bb9-33830abd1395 2020-11-16T22:06:41.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 f8e9f455-b9f3-4cfe-8bb9-33830abd1395' 2020-11-16T22:06:41.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:06:41.838 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 f8e9f455-b9f3-4cfe-8bb9-33830abd1395 2020-11-16T22:06:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDx97Jfn7WIMxAAwhGiu0WN2N2N8IbC+MR0Fw== 2020-11-16T22:06:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDx97Jfn7WIMxAAwhGiu0WN2N2N8IbC+MR0Fw=="}' 2020-11-16T22:06:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f8e9f455-b9f3-4cfe-8bb9-33830abd1395 -i td/osd-bluefs-volume-ops/3/new.json 2020-11-16T22:06:42.407 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:06:42.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2020-11-16T22:06:42.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDx97Jfn7WIMxAAwhGiu0WN2N2N8IbC+MR0Fw== --osd-uuid f8e9f455-b9f3-4cfe-8bb9-33830abd1395 2020-11-16T22:06:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:42.461+0000 7fd51e080d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:42.463 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:42.469+0000 7fd51e080d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:42.464 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:42.469+0000 7fd51e080d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:42.465 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:42.473+0000 7fd51e080d80 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2020-11-16T22:06:42.465 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:42.473+0000 7fd51e080d80 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2020-11-16T22:06:42.465 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:42.473+0000 7fd51e080d80 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2020-11-16T22:06:42.546 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:42.553+0000 7fd51e080d80 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2020-11-16T22:06:50.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2020-11-16T22:06:50.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:06:50.915 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T22:06:50.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T22:06:50.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:06:51.416 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T22:06:51.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T22:06:51.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:06:51.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:06:51.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:06:51.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:06:51.444 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:51.449+0000 7f938f651d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:51.446 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:51.453+0000 7f938f651d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:51.447 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:51.453+0000 7f938f651d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:06:51.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T22:06:51.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:06:51.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T22:06:51.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:06:51.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:06:51.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:51.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:06:51.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:51.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:06:51.790 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:06:52.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:52.499 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:52.505+0000 7f938f651d80 -1 Falling back to public interface 2020-11-16T22:06:53.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:53.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:53.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:06:53.150 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:06:53.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:53.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:06:53.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:54.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:54.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:54.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:06:54.551 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:06:54.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:06:54.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:06:55.670 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:06:55.669+0000 7f938f651d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T22:06:55.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:06:55.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:06:55.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:06:55.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:06:55.947 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:06:55.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:06:56.339 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/618353,v1:127.0.0.1:6827/618353] [v2:127.0.0.1:6828/618353,v1:127.0.0.1:6829/618353] exists,up f8e9f455-b9f3-4cfe-8bb9-33830abd1395 2020-11-16T22:06:56.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:06:56.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:06:56.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:06:56.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2020-11-16T22:06:56.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=618353 2020-11-16T22:06:56.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2020-11-16T22:07:01.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2020-11-16T22:07:01.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create foo 16 2020-11-16T22:07:01.895 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' already exists 2020-11-16T22:07:01.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:07:02.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2020-11-16T22:07:02.973 INFO:tasks.workunit.client.0.smithi189.stdout:hints = 1 2020-11-16T22:07:02.974 INFO:tasks.workunit.client.0.smithi189.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2020-11-16T22:07:02.974 INFO:tasks.workunit.client.0.smithi189.stdout:Object prefix: benchmark_data_smithi189_619087 2020-11-16T22:07:02.974 INFO:tasks.workunit.client.0.smithi189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T22:07:02.974 INFO:tasks.workunit.client.0.smithi189.stdout: 0 0 0 0 0 0 - 0 2020-11-16T22:07:03.974 INFO:tasks.workunit.client.0.smithi189.stdout: 1 16 60 44 0.171864 0.171875 0.267478 0.317241 2020-11-16T22:07:04.974 INFO:tasks.workunit.client.0.smithi189.stdout: 2 16 113 97 0.189434 0.207031 0.279668 0.293595 2020-11-16T22:07:05.974 INFO:tasks.workunit.client.0.smithi189.stdout: 3 16 193 177 0.230445 0.3125 0.220314 0.26519 2020-11-16T22:07:06.974 INFO:tasks.workunit.client.0.smithi189.stdout: 4 16 257 241 0.235327 0.25 0.258796 0.255207 2020-11-16T22:07:07.974 INFO:tasks.workunit.client.0.smithi189.stdout: 5 16 331 315 0.246065 0.289062 0.223599 0.249124 2020-11-16T22:07:08.974 INFO:tasks.workunit.client.0.smithi189.stdout: 6 16 395 379 0.246714 0.25 0.286944 0.248814 2020-11-16T22:07:09.974 INFO:tasks.workunit.client.0.smithi189.stdout: 7 16 465 449 0.250527 0.273438 0.210008 0.246774 2020-11-16T22:07:10.974 INFO:tasks.workunit.client.0.smithi189.stdout: 8 16 529 513 0.250457 0.25 0.235647 0.245136 2020-11-16T22:07:11.975 INFO:tasks.workunit.client.0.smithi189.stdout: 9 16 593 577 0.250403 0.25 0.234824 0.246422 2020-11-16T22:07:12.975 INFO:tasks.workunit.client.0.smithi189.stdout: 10 16 657 641 0.25036 0.25 0.230759 0.245163 2020-11-16T22:07:13.975 INFO:tasks.workunit.client.0.smithi189.stdout: 11 16 721 705 0.250325 0.25 0.265048 0.247365 2020-11-16T22:07:14.975 INFO:tasks.workunit.client.0.smithi189.stdout: 12 16 785 769 0.250296 0.25 0.26683 0.246706 2020-11-16T22:07:15.975 INFO:tasks.workunit.client.0.smithi189.stdout: 13 16 849 833 0.25027 0.25 0.264506 0.246677 2020-11-16T22:07:16.975 INFO:tasks.workunit.client.0.smithi189.stdout: 14 16 913 897 0.250248 0.25 0.250472 0.247449 2020-11-16T22:07:17.975 INFO:tasks.workunit.client.0.smithi189.stdout: 15 16 977 961 0.25023 0.25 0.267874 0.248056 2020-11-16T22:07:18.976 INFO:tasks.workunit.client.0.smithi189.stdout: 16 16 1037 1021 0.249238 0.234375 0.297056 0.249063 2020-11-16T22:07:19.975 INFO:tasks.workunit.client.0.smithi189.stdout: 17 16 1103 1087 0.249741 0.257812 0.204951 0.248875 2020-11-16T22:07:20.976 INFO:tasks.workunit.client.0.smithi189.stdout: 18 16 1168 1152 0.249971 0.253906 0.258936 0.248626 2020-11-16T22:07:21.976 INFO:tasks.workunit.client.0.smithi189.stdout: 19 16 1230 1214 0.24956 0.242188 0.254593 0.248933 2020-11-16T22:07:22.976 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T22:07:22.987486+0000 min lat: 0.148644 max lat: 0.432229 avg lat: 0.248112 2020-11-16T22:07:22.976 INFO:tasks.workunit.client.0.smithi189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T22:07:22.977 INFO:tasks.workunit.client.0.smithi189.stdout: 20 16 1297 1281 0.250167 0.261719 0.250984 0.248112 2020-11-16T22:07:23.976 INFO:tasks.workunit.client.0.smithi189.stdout: 21 16 1358 1342 0.249599 0.238281 0.260717 0.248769 2020-11-16T22:07:24.976 INFO:tasks.workunit.client.0.smithi189.stdout: 22 16 1421 1405 0.249439 0.246094 0.284435 0.24913 2020-11-16T22:07:25.976 INFO:tasks.workunit.client.0.smithi189.stdout: 23 16 1477 1461 0.248103 0.21875 0.266731 0.24977 2020-11-16T22:07:26.976 INFO:tasks.workunit.client.0.smithi189.stdout: 24 16 1541 1525 0.24818 0.25 0.243152 0.250042 2020-11-16T22:07:27.977 INFO:tasks.workunit.client.0.smithi189.stdout: 25 16 1605 1589 0.248251 0.25 0.243372 0.250196 2020-11-16T22:07:28.977 INFO:tasks.workunit.client.0.smithi189.stdout: 26 16 1662 1646 0.247266 0.222656 0.270224 0.250826 2020-11-16T22:07:29.977 INFO:tasks.workunit.client.0.smithi189.stdout: 27 16 1726 1710 0.247366 0.25 0.257112 0.251612 2020-11-16T22:07:30.977 INFO:tasks.workunit.client.0.smithi189.stdout: 28 16 1788 1772 0.24718 0.242188 0.287124 0.251734 2020-11-16T22:07:31.977 INFO:tasks.workunit.client.0.smithi189.stdout: 29 16 1845 1829 0.246333 0.222656 0.296527 0.252224 2020-11-16T22:07:32.977 INFO:tasks.workunit.client.0.smithi189.stdout: 30 16 1909 1893 0.246454 0.25 0.267917 0.252549 2020-11-16T22:07:33.978 INFO:tasks.workunit.client.0.smithi189.stdout:Total time run: 30.184 2020-11-16T22:07:33.978 INFO:tasks.workunit.client.0.smithi189.stdout:Total writes made: 1909 2020-11-16T22:07:33.978 INFO:tasks.workunit.client.0.smithi189.stdout:Write size: 4096 2020-11-16T22:07:33.978 INFO:tasks.workunit.client.0.smithi189.stdout:Object size: 4096 2020-11-16T22:07:33.978 INFO:tasks.workunit.client.0.smithi189.stdout:Bandwidth (MB/sec): 0.247052 2020-11-16T22:07:33.979 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Bandwidth: 0.0240437 2020-11-16T22:07:33.979 INFO:tasks.workunit.client.0.smithi189.stdout:Max bandwidth (MB/sec): 0.3125 2020-11-16T22:07:33.979 INFO:tasks.workunit.client.0.smithi189.stdout:Min bandwidth (MB/sec): 0.171875 2020-11-16T22:07:33.979 INFO:tasks.workunit.client.0.smithi189.stdout:Average IOPS: 63 2020-11-16T22:07:33.979 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev IOPS: 6.15518 2020-11-16T22:07:33.979 INFO:tasks.workunit.client.0.smithi189.stdout:Max IOPS: 80 2020-11-16T22:07:33.979 INFO:tasks.workunit.client.0.smithi189.stdout:Min IOPS: 44 2020-11-16T22:07:33.979 INFO:tasks.workunit.client.0.smithi189.stdout:Average Latency(s): 0.252477 2020-11-16T22:07:33.980 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Latency(s): 0.0289262 2020-11-16T22:07:33.980 INFO:tasks.workunit.client.0.smithi189.stdout:Max latency(s): 0.432229 2020-11-16T22:07:33.980 INFO:tasks.workunit.client.0.smithi189.stdout:Min latency(s): 0.148644 2020-11-16T22:07:34.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2020-11-16T22:07:34.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 613573 2020-11-16T22:07:34.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2020-11-16T22:07:34.530 INFO:tasks.workunit.client.0.smithi189.stdout:after bench 2020-11-16T22:07:35.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 613573 2020-11-16T22:07:35.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (613573) - No such process 2020-11-16T22:07:35.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2020-11-16T22:07:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already down. 2020-11-16T22:07:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 615166 2020-11-16T22:07:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2020-11-16T22:07:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 615166 2020-11-16T22:07:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (615166) - No such process 2020-11-16T22:07:36.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2020-11-16T22:07:37.352 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already down. 2020-11-16T22:07:37.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 616760 2020-11-16T22:07:37.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2020-11-16T22:07:38.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 616760 2020-11-16T22:07:38.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (616760) - No such process 2020-11-16T22:07:38.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2020-11-16T22:07:38.727 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already down. 2020-11-16T22:07:38.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 618353 2020-11-16T22:07:38.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2020-11-16T22:07:39.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 618353 2020-11-16T22:07:39.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (618353) - No such process 2020-11-16T22:07:39.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2020-11-16T22:07:40.237 INFO:tasks.workunit.client.0.smithi189.stderr:osd.3 is already down. 2020-11-16T22:07:40.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2020-11-16T22:07:42.049 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:07:42.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2020-11-16T22:07:43.860 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:07:43.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2020-11-16T22:07:45.673 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:07:45.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2020-11-16T22:07:47.485 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:07:47.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2020-11-16T22:07:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2020-11-16T22:07:47.524 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:07:48.446 INFO:tasks.workunit.client.0.smithi189.stdout:0 : device size 0x1ffff000 : using 0xc00000(12 MiB) 2020-11-16T22:07:48.447 INFO:tasks.workunit.client.0.smithi189.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2020-11-16T22:07:48.447 INFO:tasks.workunit.client.0.smithi189.stdout:2 : device size 0x100000000 : using 0x805dc000(2.0 GiB) 2020-11-16T22:07:48.447 INFO:tasks.workunit.client.0.smithi189.stdout:Expanding DB/WAL... 2020-11-16T22:07:48.447 INFO:tasks.workunit.client.0.smithi189.stdout:0 : expanding to 0x536870912 2020-11-16T22:07:48.470 INFO:tasks.workunit.client.0.smithi189.stdout:0 : size label updated to 536870912 2020-11-16T22:07:48.471 INFO:tasks.workunit.client.0.smithi189.stdout:1 : expanding to 0x1073741824 2020-11-16T22:07:48.496 INFO:tasks.workunit.client.0.smithi189.stdout:1 : size label updated to 1073741824 2020-11-16T22:07:48.496 INFO:tasks.workunit.client.0.smithi189.stdout:2 : expanding from 0x80000000 to 0x100000000 2020-11-16T22:07:48.705 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:07:48.716+0000 7f0529c980c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2020-11-16T22:07:48.706 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:07:48.716+0000 7f0529c980c0 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2020-11-16T22:07:53.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2020-11-16T22:07:53.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2020-11-16T22:07:53.576 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:07:54.491 INFO:tasks.workunit.client.0.smithi189.stdout:0 : device size 0x1ffff000 : using 0xc00000(12 MiB) 2020-11-16T22:07:54.491 INFO:tasks.workunit.client.0.smithi189.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2020-11-16T22:07:54.492 INFO:tasks.workunit.client.0.smithi189.stdout:2 : device size 0x101000000 : using 0x8159f000(2.0 GiB) 2020-11-16T22:07:54.492 INFO:tasks.workunit.client.0.smithi189.stdout:Expanding DB/WAL... 2020-11-16T22:07:54.492 INFO:tasks.workunit.client.0.smithi189.stdout:0 : expanding to 0x536870912 2020-11-16T22:07:54.517 INFO:tasks.workunit.client.0.smithi189.stdout:0 : size label updated to 536870912 2020-11-16T22:07:54.517 INFO:tasks.workunit.client.0.smithi189.stdout:1 : expanding to 0x1073741824 2020-11-16T22:07:54.542 INFO:tasks.workunit.client.0.smithi189.stdout:1 : size label updated to 1073741824 2020-11-16T22:07:54.542 INFO:tasks.workunit.client.0.smithi189.stdout:2 : expanding from 0x80000000 to 0x101000000 2020-11-16T22:07:54.726 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:07:54.736+0000 7f04b9ce50c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2020-11-16T22:07:54.727 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:07:54.736+0000 7f04b9ce50c0 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2020-11-16T22:07:59.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2020-11-16T22:07:59.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2020-11-16T22:07:59.635 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:08:00.538 INFO:tasks.workunit.client.0.smithi189.stdout:0 : device size 0x1ffff000 : using 0xc00000(12 MiB) 2020-11-16T22:08:00.539 INFO:tasks.workunit.client.0.smithi189.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2020-11-16T22:08:00.539 INFO:tasks.workunit.client.0.smithi189.stdout:2 : device size 0x100020000 : using 0x80627000(2.0 GiB) 2020-11-16T22:08:00.539 INFO:tasks.workunit.client.0.smithi189.stdout:Expanding DB/WAL... 2020-11-16T22:08:00.539 INFO:tasks.workunit.client.0.smithi189.stdout:0 : expanding to 0x536870912 2020-11-16T22:08:00.563 INFO:tasks.workunit.client.0.smithi189.stdout:0 : size label updated to 536870912 2020-11-16T22:08:00.563 INFO:tasks.workunit.client.0.smithi189.stdout:1 : expanding to 0x1073741824 2020-11-16T22:08:00.596 INFO:tasks.workunit.client.0.smithi189.stdout:1 : size label updated to 1073741824 2020-11-16T22:08:00.597 INFO:tasks.workunit.client.0.smithi189.stdout:2 : expanding from 0x80000000 to 0x100020000 2020-11-16T22:08:00.781 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:00.788+0000 7f3f94bf30c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2020-11-16T22:08:00.781 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:00.788+0000 7f3f94bf30c0 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2020-11-16T22:08:05.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2020-11-16T22:08:05.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2020-11-16T22:08:05.871 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:08:06.793 INFO:tasks.workunit.client.0.smithi189.stdout:0 : device size 0x1ffff000 : using 0xc00000(12 MiB) 2020-11-16T22:08:06.794 INFO:tasks.workunit.client.0.smithi189.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2020-11-16T22:08:06.794 INFO:tasks.workunit.client.0.smithi189.stdout:2 : device size 0xfff00000 : using 0x804e2000(2.0 GiB) 2020-11-16T22:08:06.794 INFO:tasks.workunit.client.0.smithi189.stdout:Expanding DB/WAL... 2020-11-16T22:08:06.794 INFO:tasks.workunit.client.0.smithi189.stdout:0 : expanding to 0x536870912 2020-11-16T22:08:06.809 INFO:tasks.workunit.client.0.smithi189.stdout:0 : size label updated to 536870912 2020-11-16T22:08:06.809 INFO:tasks.workunit.client.0.smithi189.stdout:1 : expanding to 0x1073741824 2020-11-16T22:08:06.850 INFO:tasks.workunit.client.0.smithi189.stdout:1 : size label updated to 1073741824 2020-11-16T22:08:06.851 INFO:tasks.workunit.client.0.smithi189.stdout:2 : expanding from 0x80000000 to 0xfff00000 2020-11-16T22:08:07.093 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:07.100+0000 7fd48d9130c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2020-11-16T22:08:07.094 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:07.100+0000 7fd48d9130c0 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2020-11-16T22:08:11.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2020-11-16T22:08:13.736 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:08:13.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2020-11-16T22:08:15.548 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:08:15.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2020-11-16T22:08:17.364 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:08:17.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2020-11-16T22:08:19.180 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:08:19.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2020-11-16T22:08:19.201 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:08:20.999 INFO:tasks.workunit.client.0.smithi189.stdout:0 : device size 0x1ffff000 : using 0x500000(5 MiB) 2020-11-16T22:08:20.999 INFO:tasks.workunit.client.0.smithi189.stdout:1 : device size 0x3fffe000 : using 0xf00000(15 MiB) 2020-11-16T22:08:20.999 INFO:tasks.workunit.client.0.smithi189.stdout:2 : device size 0x100000000 : using 0x5dc000(5.9 MiB) 2020-11-16T22:08:21.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2020-11-16T22:08:21.016 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:08:22.553 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2020-11-16T22:08:22.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2020-11-16T22:08:23.872 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:08:23.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2020-11-16T22:08:23.894 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:08:25.681 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2020-11-16T22:08:25.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2020-11-16T22:08:27.000 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:08:27.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2020-11-16T22:08:27.022 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:08:28.812 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2020-11-16T22:08:28.813 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2020-11-16T22:08:28.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2020-11-16T22:08:29.624 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:08:29.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2020-11-16T22:08:29.643 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:08:29.649 INFO:tasks.workunit.client.0.smithi189.stderr:Migrate to WAL device isn't supported. 2020-11-16T22:08:29.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2020-11-16T22:08:29.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2020-11-16T22:08:31.464 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:08:31.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2020-11-16T22:08:31.488 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:08:33.028 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2020-11-16T22:08:33.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2020-11-16T22:08:34.366 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:08:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2020-11-16T22:08:34.384 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:08:35.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2020-11-16T22:08:36.977 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:08:36.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2020-11-16T22:08:36.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:08:36.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:08:36.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T22:08:36.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:08:36.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2020-11-16T22:08:36.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:08:36.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:08:36.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:08:36.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:08:36.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2020-11-16T22:08:36.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2020-11-16T22:08:36.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:08:36.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:08:36.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:08:36.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:08:36.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:08:36.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:08:36.985 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:08:36.985 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:08:36.985 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:08:36.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:08:36.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:08:36.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:08:36.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:08:36.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:08:36.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:08:36.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:08:36.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:08:36.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:08:36.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:08:36.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2020-11-16T22:08:36.988 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:08:36.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T22:08:36.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:08:36.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2020-11-16T22:08:36.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T22:08:36.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:08:36.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:08:36.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:08:37.007 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:37.015+0000 7fea2b637d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:37.010 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:37.019+0000 7fea2b637d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:37.011 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:37.019+0000 7fea2b637d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T22:08:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:08:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:08:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:08:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:08:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:08:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:08:37.402 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:08:37.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:08:37.799 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:37.807+0000 7fea2b637d80 -1 Falling back to public interface 2020-11-16T22:08:38.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:08:38.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:38.781 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:08:38.781 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:08:38.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:38.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:08:39.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:08:40.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:08:40.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:40.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:08:40.155 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:08:40.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:40.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:08:40.386 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:40.395+0000 7fea2b637d80 -1 osd.0 26 log_to_monitors {default=true} 2020-11-16T22:08:40.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:08:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:08:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:08:41.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:41.531 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:08:41.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:08:41.922 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 36 up_thru 36 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6802/627581,v1:127.0.0.1:6803/627581] [v2:127.0.0.1:6804/627581,v1:127.0.0.1:6805/627581] exists,up 1f6242ad-048f-45f5-818b-c23c7b6373fd 2020-11-16T22:08:41.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:08:41.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:08:41.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:08:41.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2020-11-16T22:08:41.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=627581 2020-11-16T22:08:41.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2020-11-16T22:08:41.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:08:41.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:08:41.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T22:08:41.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:08:41.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2020-11-16T22:08:41.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:08:41.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:08:41.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:08:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:08:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2020-11-16T22:08:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2020-11-16T22:08:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:08:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:08:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:08:41.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:08:41.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:08:41.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:08:41.931 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:08:41.931 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:08:41.931 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:08:41.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:08:41.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:08:41.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:08:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:08:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:08:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:08:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:08:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:08:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:08:41.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:08:41.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2020-11-16T22:08:41.933 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:08:41.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T22:08:41.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:08:41.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2020-11-16T22:08:41.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T22:08:41.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:08:41.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:08:41.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:08:41.953 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:41.963+0000 7f953112cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:41.956 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:41.963+0000 7f953112cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:41.957 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:41.967+0000 7f953112cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:42.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T22:08:42.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:08:42.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:08:42.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:08:42.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:08:42.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:42.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:08:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:42.317 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:08:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:08:42.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:08:42.754 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:42.763+0000 7f953112cd80 -1 Falling back to public interface 2020-11-16T22:08:43.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:08:43.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:43.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:08:43.687 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:08:43.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:08:43.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:44.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:08:45.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:08:45.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:45.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:08:45.063 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:08:45.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:45.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:08:45.291 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:45.299+0000 7f953112cd80 -1 osd.1 29 log_to_monitors {default=true} 2020-11-16T22:08:45.484 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 38 up_thru 27 down_at 30 last_clean_interval [10,29) [v2:127.0.0.1:6810/628292,v1:127.0.0.1:6811/628292] [v2:127.0.0.1:6812/628292,v1:127.0.0.1:6813/628292] exists,up dabc47d8-e723-4845-87ab-b470967b9c5f 2020-11-16T22:08:45.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:08:45.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:08:45.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:08:45.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2020-11-16T22:08:45.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=628292 2020-11-16T22:08:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2020-11-16T22:08:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:08:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:08:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-11-16T22:08:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:08:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2020-11-16T22:08:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:08:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:08:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:08:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:08:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2020-11-16T22:08:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2020-11-16T22:08:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:08:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:08:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:08:45.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:08:45.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:08:45.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:08:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:08:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:08:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:08:45.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:08:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:08:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:08:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:08:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:08:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:08:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:08:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:08:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:08:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:08:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2020-11-16T22:08:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-11-16T22:08:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:08:45.494 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:08:45.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2020-11-16T22:08:45.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-11-16T22:08:45.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:08:45.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:08:45.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:08:45.523 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:45.531+0000 7f678874cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:45.529 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:45.539+0000 7f678874cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:45.531 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:45.539+0000 7f678874cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:45.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-11-16T22:08:45.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:08:45.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:08:45.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:08:45.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:08:45.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:45.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:08:45.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:45.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:08:45.972 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:08:46.070 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:46.079+0000 7f678874cd80 -1 Falling back to public interface 2020-11-16T22:08:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:08:47.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:08:47.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:47.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:08:47.359 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:08:47.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:47.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:08:47.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:08:48.007 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:48.015+0000 7f678874cd80 -1 osd.2 32 log_to_monitors {default=true} 2020-11-16T22:08:48.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:08:48.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:48.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:08:48.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:48.719 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:08:48.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:08:49.141 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 41 up_thru 41 down_at 33 last_clean_interval [15,32) [v2:127.0.0.1:6818/628977,v1:127.0.0.1:6819/628977] [v2:127.0.0.1:6820/628977,v1:127.0.0.1:6821/628977] exists,up 538b02af-9658-4dba-ac56-d474d8d8b6b0 2020-11-16T22:08:49.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:08:49.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:08:49.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:08:49.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2020-11-16T22:08:49.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=628977 2020-11-16T22:08:49.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2020-11-16T22:08:49.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:08:49.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:08:49.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2020-11-16T22:08:49.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:08:49.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2020-11-16T22:08:49.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:08:49.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:08:49.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:08:49.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:08:49.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2020-11-16T22:08:49.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2020-11-16T22:08:49.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:08:49.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:08:49.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:08:49.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:08:49.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:08:49.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:08:49.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:08:49.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:08:49.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:08:49.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:08:49.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:08:49.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:08:49.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:08:49.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:08:49.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:08:49.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:08:49.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:08:49.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:08:49.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:08:49.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2020-11-16T22:08:49.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2020-11-16T22:08:49.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:08:49.153 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T22:08:49.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2020-11-16T22:08:49.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2020-11-16T22:08:49.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:08:49.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:08:49.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:08:49.170 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:49.179+0000 7f1658b61d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:49.173 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:49.183+0000 7f1658b61d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:49.175 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:49.183+0000 7f1658b61d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:08:49.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2020-11-16T22:08:49.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:08:49.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T22:08:49.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:08:49.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:08:49.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:49.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:08:49.540 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:08:49.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:49.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:08:49.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:08:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:49.983+0000 7f1658b61d80 -1 Falling back to public interface 2020-11-16T22:08:50.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:08:50.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:50.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:08:50.924 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:08:50.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:50.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:08:51.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:08:52.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:08:52.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:52.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:08:52.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:52.313 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:08:52.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:08:52.570 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:08:52.579+0000 7f1658b61d80 -1 osd.3 34 log_to_monitors {default=true} 2020-11-16T22:08:52.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:08:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:08:53.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:08:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:08:53.786 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:08:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:08:53.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:08:54.252 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 44 up_thru 44 down_at 35 last_clean_interval [20,34) [v2:127.0.0.1:6826/629652,v1:127.0.0.1:6827/629652] [v2:127.0.0.1:6828/629652,v1:127.0.0.1:6829/629652] exists,up f8e9f455-b9f3-4cfe-8bb9-33830abd1395 2020-11-16T22:08:54.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:08:54.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:08:54.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:08:54.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2020-11-16T22:08:54.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=629652 2020-11-16T22:08:54.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2020-11-16T22:08:54.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:08:54.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:08:54.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:08:54.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:08:54.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:08:54.257 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:08:54.257 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:08:54.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:08:54.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:08:54.258 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:08:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:08:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:08:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:08:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:08:54.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:08:54.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:08:54.811 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:08:54.811 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:08:54.811 INFO:tasks.workunit.client.0.smithi189.stderr:3' 2020-11-16T22:08:54.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:08:54.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:08:54.813 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:08:54.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822661 2020-11-16T22:08:54.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822661 2020-11-16T22:08:54.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822661' 2020-11-16T22:08:54.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:08:54.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:08:55.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757252 2020-11-16T22:08:55.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757252 2020-11-16T22:08:55.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822661 1-163208757252' 2020-11-16T22:08:55.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:08:55.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:08:55.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=176093659140 2020-11-16T22:08:55.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 176093659140 2020-11-16T22:08:55.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822661 1-163208757252 2-176093659140' 2020-11-16T22:08:55.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:08:55.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:08:55.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=188978561027 2020-11-16T22:08:55.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 188978561027 2020-11-16T22:08:55.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822661 1-163208757252 2-176093659140 3-188978561027' 2020-11-16T22:08:55.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:08:55.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-154618822661 2020-11-16T22:08:55.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:08:55.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:08:55.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-154618822661 2020-11-16T22:08:55.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:08:55.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822661 2020-11-16T22:08:55.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 154618822661' 2020-11-16T22:08:55.457 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 154618822661 2020-11-16T22:08:55.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:08:55.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822659 -lt 154618822661 2020-11-16T22:08:55.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:08:56.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:08:56.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:08:57.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822662 -lt 154618822661 2020-11-16T22:08:57.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:08:57.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-163208757252 2020-11-16T22:08:57.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:08:57.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:08:57.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-163208757252 2020-11-16T22:08:57.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:08:57.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757252 2020-11-16T22:08:57.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 163208757252' 2020-11-16T22:08:57.259 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 163208757252 2020-11-16T22:08:57.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:08:57.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757253 -lt 163208757252 2020-11-16T22:08:57.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:08:57.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-176093659140 2020-11-16T22:08:57.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:08:57.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:08:57.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-176093659140 2020-11-16T22:08:57.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:08:57.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=176093659140 2020-11-16T22:08:57.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 176093659140' 2020-11-16T22:08:57.676 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 176093659140 2020-11-16T22:08:57.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:08:58.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 176093659140 -lt 176093659140 2020-11-16T22:08:58.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:08:58.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-188978561027 2020-11-16T22:08:58.047 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:08:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:08:58.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-188978561027 2020-11-16T22:08:58.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:08:58.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=188978561027 2020-11-16T22:08:58.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 188978561027' 2020-11-16T22:08:58.052 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 188978561027 2020-11-16T22:08:58.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:08:58.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 188978561027 -lt 188978561027 2020-11-16T22:08:58.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:08:58.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:08:58.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:08:58.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 16 == 0 2020-11-16T22:08:58.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:08:58.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:08:58.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:08:58.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:08:58.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:08:58.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:08:58.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:08:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=16 2020-11-16T22:08:59.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:08:59.246 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:08:59.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:08:59.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 16 = 16 2020-11-16T22:08:59.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:08:59.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:08:59.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2020-11-16T22:08:59.736 INFO:tasks.workunit.client.0.smithi189.stdout:hints = 1 2020-11-16T22:08:59.737 INFO:tasks.workunit.client.0.smithi189.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2020-11-16T22:08:59.737 INFO:tasks.workunit.client.0.smithi189.stdout:Object prefix: benchmark_data_smithi189_630795 2020-11-16T22:08:59.737 INFO:tasks.workunit.client.0.smithi189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T22:08:59.737 INFO:tasks.workunit.client.0.smithi189.stdout: 0 0 0 0 0 0 - 0 2020-11-16T22:09:00.737 INFO:tasks.workunit.client.0.smithi189.stdout: 1 16 50 34 0.132795 0.132812 0.24617 0.259828 2020-11-16T22:09:01.737 INFO:tasks.workunit.client.0.smithi189.stdout: 2 16 56 40 0.0781143 0.0234375 0.100813 0.249432 2020-11-16T22:09:02.737 INFO:tasks.workunit.client.0.smithi189.stdout: 3 16 61 45 0.0585855 0.0195312 0.215334 0.424817 2020-11-16T22:09:03.737 INFO:tasks.workunit.client.0.smithi189.stdout: 4 16 71 55 0.053703 0.0390625 0.116679 0.375676 2020-11-16T22:09:04.737 INFO:tasks.workunit.client.0.smithi189.stdout: 5 16 80 64 0.0499927 0.0351562 0.109041 0.338699 2020-11-16T22:09:05.737 INFO:tasks.workunit.client.0.smithi189.stdout: 6 16 113 97 0.0631419 0.128906 0.238798 0.962155 2020-11-16T22:09:06.737 INFO:tasks.workunit.client.0.smithi189.stdout: 7 16 176 160 0.0892727 0.246094 0.357571 0.690921 2020-11-16T22:09:07.738 INFO:tasks.workunit.client.0.smithi189.stdout: 8 16 230 214 0.104478 0.210938 0.251329 0.585678 2020-11-16T22:09:08.738 INFO:tasks.workunit.client.0.smithi189.stdout: 9 16 291 275 0.119341 0.238281 0.254722 0.514404 2020-11-16T22:09:09.738 INFO:tasks.workunit.client.0.smithi189.stdout: 10 16 355 339 0.132404 0.25 0.28236 0.468237 2020-11-16T22:09:10.738 INFO:tasks.workunit.client.0.smithi189.stdout: 11 16 413 397 0.140961 0.226562 0.286035 0.438868 2020-11-16T22:09:11.738 INFO:tasks.workunit.client.0.smithi189.stdout: 12 16 468 452 0.147115 0.214844 0.212582 0.418605 2020-11-16T22:09:12.738 INFO:tasks.workunit.client.0.smithi189.stdout: 13 16 534 518 0.155627 0.257812 0.250304 0.396912 2020-11-16T22:09:13.739 INFO:tasks.workunit.client.0.smithi189.stdout: 14 16 598 582 0.162365 0.25 0.32245 0.380088 2020-11-16T22:09:14.739 INFO:tasks.workunit.client.0.smithi189.stdout: 15 16 661 645 0.167945 0.246094 0.241023 0.36863 2020-11-16T22:09:15.739 INFO:tasks.workunit.client.0.smithi189.stdout: 16 16 728 712 0.173804 0.261719 0.234503 0.3575 2020-11-16T22:09:16.739 INFO:tasks.workunit.client.0.smithi189.stdout: 17 16 780 764 0.175527 0.203125 0.42313 0.351504 2020-11-16T22:09:17.739 INFO:tasks.workunit.client.0.smithi189.stdout: 18 16 840 824 0.178794 0.234375 0.219634 0.345668 2020-11-16T22:09:18.739 INFO:tasks.workunit.client.0.smithi189.stdout: 19 16 888 872 0.179251 0.1875 0.261404 0.344913 2020-11-16T22:09:19.739 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T22:09:19.751047+0000 min lat: 0.0923536 max lat: 5.59469 avg lat: 0.339045 2020-11-16T22:09:19.740 INFO:tasks.workunit.client.0.smithi189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T22:09:19.740 INFO:tasks.workunit.client.0.smithi189.stdout: 20 16 953 937 0.182982 0.253906 0.26436 0.339045 2020-11-16T22:09:20.739 INFO:tasks.workunit.client.0.smithi189.stdout: 21 16 1016 1000 0.185986 0.246094 0.24315 0.334232 2020-11-16T22:09:21.739 INFO:tasks.workunit.client.0.smithi189.stdout: 22 16 1066 1050 0.186409 0.195312 0.28412 0.331225 2020-11-16T22:09:22.740 INFO:tasks.workunit.client.0.smithi189.stdout: 23 16 1129 1113 0.189002 0.246094 0.257185 0.329223 2020-11-16T22:09:23.740 INFO:tasks.workunit.client.0.smithi189.stdout: 24 16 1183 1167 0.189915 0.210938 0.301989 0.327064 2020-11-16T22:09:24.740 INFO:tasks.workunit.client.0.smithi189.stdout: 25 16 1245 1229 0.192005 0.242188 0.268857 0.324199 2020-11-16T22:09:25.740 INFO:tasks.workunit.client.0.smithi189.stdout: 26 16 1304 1288 0.193483 0.230469 0.246088 0.321679 2020-11-16T22:09:26.740 INFO:tasks.workunit.client.0.smithi189.stdout: 27 16 1359 1343 0.194273 0.214844 0.382231 0.320019 2020-11-16T22:09:27.740 INFO:tasks.workunit.client.0.smithi189.stdout: 28 16 1417 1401 0.195425 0.226562 0.267532 0.318568 2020-11-16T22:09:28.740 INFO:tasks.workunit.client.0.smithi189.stdout: 29 16 1472 1456 0.196093 0.214844 0.275678 0.316734 2020-11-16T22:09:29.741 INFO:tasks.workunit.client.0.smithi189.stdout: 30 16 1537 1521 0.198019 0.253906 0.262346 0.314602 2020-11-16T22:09:30.741 INFO:tasks.workunit.client.0.smithi189.stdout:Total time run: 30.2641 2020-11-16T22:09:30.741 INFO:tasks.workunit.client.0.smithi189.stdout:Total writes made: 1537 2020-11-16T22:09:30.741 INFO:tasks.workunit.client.0.smithi189.stdout:Write size: 4096 2020-11-16T22:09:30.742 INFO:tasks.workunit.client.0.smithi189.stdout:Object size: 4096 2020-11-16T22:09:30.742 INFO:tasks.workunit.client.0.smithi189.stdout:Bandwidth (MB/sec): 0.198383 2020-11-16T22:09:30.742 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Bandwidth: 0.0744145 2020-11-16T22:09:30.742 INFO:tasks.workunit.client.0.smithi189.stdout:Max bandwidth (MB/sec): 0.261719 2020-11-16T22:09:30.742 INFO:tasks.workunit.client.0.smithi189.stdout:Min bandwidth (MB/sec): 0.0195312 2020-11-16T22:09:30.742 INFO:tasks.workunit.client.0.smithi189.stdout:Average IOPS: 50 2020-11-16T22:09:30.742 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev IOPS: 19.0501 2020-11-16T22:09:30.743 INFO:tasks.workunit.client.0.smithi189.stdout:Max IOPS: 67 2020-11-16T22:09:30.743 INFO:tasks.workunit.client.0.smithi189.stdout:Min IOPS: 5 2020-11-16T22:09:30.743 INFO:tasks.workunit.client.0.smithi189.stdout:Average Latency(s): 0.31428 2020-11-16T22:09:30.743 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Latency(s): 0.438794 2020-11-16T22:09:30.743 INFO:tasks.workunit.client.0.smithi189.stdout:Max latency(s): 5.59469 2020-11-16T22:09:30.743 INFO:tasks.workunit.client.0.smithi189.stdout:Min latency(s): 0.0923536 2020-11-16T22:09:30.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 627581 2020-11-16T22:09:30.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2020-11-16T22:09:31.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 627581 2020-11-16T22:09:31.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (627581) - No such process 2020-11-16T22:09:31.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2020-11-16T22:09:32.256 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already down. 2020-11-16T22:09:32.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 628292 2020-11-16T22:09:32.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2020-11-16T22:09:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 628292 2020-11-16T22:09:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (628292) - No such process 2020-11-16T22:09:33.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2020-11-16T22:09:33.650 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already down. 2020-11-16T22:09:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 628977 2020-11-16T22:09:33.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2020-11-16T22:09:34.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 628977 2020-11-16T22:09:34.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (628977) - No such process 2020-11-16T22:09:34.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2020-11-16T22:09:35.063 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already down. 2020-11-16T22:09:35.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 629652 2020-11-16T22:09:35.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2020-11-16T22:09:36.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 629652 2020-11-16T22:09:36.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (629652) - No such process 2020-11-16T22:09:36.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2020-11-16T22:09:36.614 INFO:tasks.workunit.client.0.smithi189.stderr:osd.3 is already down. 2020-11-16T22:09:36.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2020-11-16T22:09:38.184 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:09:38.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2020-11-16T22:09:38.449 INFO:tasks.workunit.client.0.smithi189.stderr:512+0 records in 2020-11-16T22:09:38.449 INFO:tasks.workunit.client.0.smithi189.stderr:512+0 records out 2020-11-16T22:09:38.449 INFO:tasks.workunit.client.0.smithi189.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.256897 s, 2.1 GB/s 2020-11-16T22:09:38.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2020-11-16T22:09:38.464 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:09:42.752 INFO:tasks.workunit.client.0.smithi189.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2020-11-16T22:09:42.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2020-11-16T22:09:44.560 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:09:44.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2020-11-16T22:09:45.872 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:09:45.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2020-11-16T22:09:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records in 2020-11-16T22:09:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records out 2020-11-16T22:09:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.479436 s, 2.2 GB/s 2020-11-16T22:09:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2020-11-16T22:09:46.378 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:09:53.668 INFO:tasks.workunit.client.0.smithi189.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2020-11-16T22:09:53.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2020-11-16T22:09:53.689 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:09:55.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2020-11-16T22:09:57.536 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:09:57.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2020-11-16T22:09:58.344 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:09:58.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2020-11-16T22:09:58.366 INFO:tasks.workunit.client.0.smithi189.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2020-11-16T22:09:58.366 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:09:59.160 INFO:tasks.workunit.client.0.smithi189.stdout:DB device added 2020-11-16T22:09:59.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2020-11-16T22:09:59.181 INFO:tasks.workunit.client.0.smithi189.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2020-11-16T22:09:59.182 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:10:00.468 INFO:tasks.workunit.client.0.smithi189.stdout:WAL device added 2020-11-16T22:10:00.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2020-11-16T22:10:00.489 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:10:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2020-11-16T22:10:04.340 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:10:04.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2020-11-16T22:10:05.648 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:10:05.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2020-11-16T22:10:05.668 INFO:tasks.workunit.client.0.smithi189.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2020-11-16T22:10:05.669 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:10:06.968 INFO:tasks.workunit.client.0.smithi189.stdout:WAL device added 2020-11-16T22:10:06.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2020-11-16T22:10:06.989 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:10:09.028 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2020-11-16T22:10:09.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2020-11-16T22:10:10.337 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:10:10.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2020-11-16T22:10:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:10:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:10:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T22:10:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:10:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2020-11-16T22:10:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:10:10.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:10:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:10:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:10:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2020-11-16T22:10:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2020-11-16T22:10:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:10:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:10:10.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:10:10.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:10:10.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:10:10.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:10:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:10:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:10:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:10:10.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:10:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:10:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:10:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:10:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:10:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:10:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:10:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:10:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:10:10.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:10:10.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2020-11-16T22:10:10.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T22:10:10.349 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:10:10.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:10:10.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2020-11-16T22:10:10.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T22:10:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:10:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:10:10.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:10:10.375 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:10.382+0000 7f570b5ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:10.378 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:10.386+0000 7f570b5ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:10.379 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:10.386+0000 7f570b5ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T22:10:10.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:10:10.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:10:10.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:10:10.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:10:10.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:10.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:10:10.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:10.737 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:10:10.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:10:11.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:11.426 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:11.433+0000 7f570b5ccd80 -1 Falling back to public interface 2020-11-16T22:10:12.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:12.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:12.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:10:12.129 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:10:12.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:12.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:10:12.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:13.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:13.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:13.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:10:13.519 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:10:13.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:13.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:10:13.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:14.851 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:14.857+0000 7f570b5ccd80 -1 osd.0 46 log_to_monitors {default=true} 2020-11-16T22:10:14.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:14.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:14.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:10:14.891 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:10:14.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:14.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:10:15.308 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 56 up_thru 56 down_at 47 last_clean_interval [36,46) [v2:127.0.0.1:6802/635096,v1:127.0.0.1:6803/635096] [v2:127.0.0.1:6804/635096,v1:127.0.0.1:6805/635096] exists,up 1f6242ad-048f-45f5-818b-c23c7b6373fd 2020-11-16T22:10:15.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:10:15.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:10:15.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:10:15.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2020-11-16T22:10:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=635096 2020-11-16T22:10:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2020-11-16T22:10:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:10:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:10:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T22:10:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:10:15.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2020-11-16T22:10:15.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:10:15.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:10:15.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:10:15.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:10:15.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2020-11-16T22:10:15.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2020-11-16T22:10:15.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:10:15.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:10:15.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:10:15.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:10:15.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:10:15.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:10:15.316 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:10:15.316 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:10:15.316 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:10:15.317 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:10:15.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:10:15.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:10:15.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:10:15.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:10:15.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:10:15.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:10:15.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:10:15.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:10:15.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:10:15.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2020-11-16T22:10:15.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T22:10:15.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:10:15.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2020-11-16T22:10:15.320 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:10:15.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T22:10:15.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:10:15.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:10:15.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:10:15.340 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:15.349+0000 7f82a834ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:15.343 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:15.349+0000 7f82a834ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:15.345 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:15.353+0000 7f82a834ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:15.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T22:10:15.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:10:15.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:10:15.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:10:15.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:10:15.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:15.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:10:15.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:15.714 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:10:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:10:16.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:16.385 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:16.393+0000 7f82a834ed80 -1 Falling back to public interface 2020-11-16T22:10:17.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:17.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:17.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:10:17.101 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:10:17.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:17.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:10:17.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:18.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:18.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:18.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:10:18.494 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:10:18.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:18.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:10:18.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:19.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:19.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:19.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:10:19.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:19.877 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:10:19.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:10:19.932 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:19.941+0000 7f82a834ed80 -1 osd.1 49 log_to_monitors {default=true} 2020-11-16T22:10:20.303 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 58 up_thru 47 down_at 50 last_clean_interval [38,49) [v2:127.0.0.1:6810/635817,v1:127.0.0.1:6811/635817] [v2:127.0.0.1:6812/635817,v1:127.0.0.1:6813/635817] exists,up dabc47d8-e723-4845-87ab-b470967b9c5f 2020-11-16T22:10:20.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:10:20.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:10:20.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:10:20.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2020-11-16T22:10:20.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=635817 2020-11-16T22:10:20.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2020-11-16T22:10:20.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:10:20.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:10:20.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-11-16T22:10:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:10:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2020-11-16T22:10:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:10:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:10:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:10:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:10:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2020-11-16T22:10:20.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2020-11-16T22:10:20.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:10:20.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:10:20.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:10:20.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:10:20.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:10:20.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:10:20.312 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:10:20.312 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:10:20.312 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:10:20.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:10:20.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:10:20.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:10:20.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:10:20.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:10:20.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:10:20.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:10:20.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:10:20.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:10:20.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:10:20.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2020-11-16T22:10:20.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-11-16T22:10:20.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:10:20.315 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:10:20.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2020-11-16T22:10:20.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-11-16T22:10:20.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:10:20.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:10:20.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:10:20.335 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:20.341+0000 7fe8bd967d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:20.345+0000 7fe8bd967d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:20.345+0000 7fe8bd967d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-11-16T22:10:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:10:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:10:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:10:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:10:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:10:20.687 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:10:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:10:21.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:21.371 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:21.377+0000 7fe8bd967d80 -1 Falling back to public interface 2020-11-16T22:10:22.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:22.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:22.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:10:22.070 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:10:22.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:22.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:10:22.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:23.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:10:23.486 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:10:23.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:23.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:10:23.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:24.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:24.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:24.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:10:24.877 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:10:24.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:24.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:10:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:25.565+0000 7fe8bd967d80 -1 osd.2 52 log_to_monitors {default=true} 2020-11-16T22:10:26.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:26.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:26.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 4 2020-11-16T22:10:26.255 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T22:10:26.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:26.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:10:26.643 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 60 up_thru 60 down_at 53 last_clean_interval [41,52) [v2:127.0.0.1:6818/636538,v1:127.0.0.1:6819/636538] [v2:127.0.0.1:6820/636538,v1:127.0.0.1:6821/636538] exists,up 538b02af-9658-4dba-ac56-d474d8d8b6b0 2020-11-16T22:10:26.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:10:26.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:10:26.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:10:26.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2020-11-16T22:10:26.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=636538 2020-11-16T22:10:26.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2020-11-16T22:10:26.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:10:26.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:10:26.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2020-11-16T22:10:26.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:10:26.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2020-11-16T22:10:26.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:10:26.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:10:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:10:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:10:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2020-11-16T22:10:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2020-11-16T22:10:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:10:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:10:26.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:10:26.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:10:26.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:10:26.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:10:26.653 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:10:26.653 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:10:26.653 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:10:26.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:10:26.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:10:26.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:10:26.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:10:26.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:10:26.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:10:26.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:10:26.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:10:26.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:10:26.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:10:26.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2020-11-16T22:10:26.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2020-11-16T22:10:26.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:10:26.655 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T22:10:26.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2020-11-16T22:10:26.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2020-11-16T22:10:26.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:10:26.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:10:26.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:10:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:26.677+0000 7f51ce92bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:26.673 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:26.681+0000 7f51ce92bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:26.674 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:26.681+0000 7f51ce92bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:10:27.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2020-11-16T22:10:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:10:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T22:10:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:10:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:10:27.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:27.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:10:27.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:27.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:10:27.035 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:10:27.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:27.471 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:27.477+0000 7f51ce92bd80 -1 Falling back to public interface 2020-11-16T22:10:28.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:28.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:28.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:10:28.401 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:10:28.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:28.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:10:28.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:29.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:10:29.790 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:10:29.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:29.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:10:30.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:10:30.821 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:10:30.829+0000 7f51ce92bd80 -1 osd.3 54 log_to_monitors {default=true} 2020-11-16T22:10:31.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:10:31.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:10:31.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:10:31.189 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:10:31.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:10:31.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:10:31.775 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 62 up_thru 53 down_at 55 last_clean_interval [44,54) [v2:127.0.0.1:6826/637285,v1:127.0.0.1:6827/637285] [v2:127.0.0.1:6828/637285,v1:127.0.0.1:6829/637285] exists,up f8e9f455-b9f3-4cfe-8bb9-33830abd1395 2020-11-16T22:10:31.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:10:31.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:10:31.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:10:31.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2020-11-16T22:10:31.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=637285 2020-11-16T22:10:31.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2020-11-16T22:10:31.815 INFO:tasks.workunit.client.0.smithi189.stdout:hints = 1 2020-11-16T22:10:31.815 INFO:tasks.workunit.client.0.smithi189.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2020-11-16T22:10:31.815 INFO:tasks.workunit.client.0.smithi189.stdout:Object prefix: benchmark_data_smithi189_637981 2020-11-16T22:10:31.816 INFO:tasks.workunit.client.0.smithi189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T22:10:31.816 INFO:tasks.workunit.client.0.smithi189.stdout: 0 11 11 0 0 0 - 0 2020-11-16T22:10:32.816 INFO:tasks.workunit.client.0.smithi189.stdout: 1 16 35 19 0.0741816 0.0742188 0.354453 0.407578 2020-11-16T22:10:33.816 INFO:tasks.workunit.client.0.smithi189.stdout: 2 16 43 27 0.0527174 0.03125 0.137905 0.353819 2020-11-16T22:10:34.816 INFO:tasks.workunit.client.0.smithi189.stdout: 3 16 43 27 0.0351466 0 - 0.353819 2020-11-16T22:10:35.816 INFO:tasks.workunit.client.0.smithi189.stdout: 4 16 43 27 0.0263609 0 - 0.353819 2020-11-16T22:10:36.816 INFO:tasks.workunit.client.0.smithi189.stdout: 5 16 43 27 0.0210891 0 - 0.353819 2020-11-16T22:10:37.816 INFO:tasks.workunit.client.0.smithi189.stdout: 6 16 43 27 0.0175745 0 - 0.353819 2020-11-16T22:10:38.817 INFO:tasks.workunit.client.0.smithi189.stdout: 7 16 43 27 0.015064 0 - 0.353819 2020-11-16T22:10:39.817 INFO:tasks.workunit.client.0.smithi189.stdout: 8 16 43 27 0.0131811 0 - 0.353819 2020-11-16T22:10:40.817 INFO:tasks.workunit.client.0.smithi189.stdout: 9 16 43 27 0.0117166 0 - 0.353819 2020-11-16T22:10:41.817 INFO:tasks.workunit.client.0.smithi189.stdout: 10 16 43 27 0.010545 0 - 0.353819 2020-11-16T22:10:42.817 INFO:tasks.workunit.client.0.smithi189.stdout: 11 16 54 38 0.013492 0.00477431 10.137 3.22656 2020-11-16T22:10:43.817 INFO:tasks.workunit.client.0.smithi189.stdout: 12 16 85 69 0.0224571 0.121094 0.157396 1.89744 2020-11-16T22:10:44.817 INFO:tasks.workunit.client.0.smithi189.stdout: 13 16 118 102 0.0306438 0.128906 0.275283 2.01584 2020-11-16T22:10:45.818 INFO:tasks.workunit.client.0.smithi189.stdout: 14 16 171 155 0.0432405 0.207031 0.211977 1.42357 2020-11-16T22:10:46.818 INFO:tasks.workunit.client.0.smithi189.stdout: 15 16 230 214 0.0557201 0.230469 0.189414 1.1145 2020-11-16T22:10:47.818 INFO:tasks.workunit.client.0.smithi189.stdout: 16 16 287 271 0.0661515 0.222656 0.292573 0.938342 2020-11-16T22:10:48.818 INFO:tasks.workunit.client.0.smithi189.stdout: 17 16 338 322 0.0739771 0.199219 0.285415 0.834642 2020-11-16T22:10:49.818 INFO:tasks.workunit.client.0.smithi189.stdout: 18 16 396 380 0.0824523 0.226562 0.281805 0.752266 2020-11-16T22:10:50.818 INFO:tasks.workunit.client.0.smithi189.stdout: 19 16 449 433 0.0890074 0.207031 0.31101 0.695035 2020-11-16T22:10:51.818 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T22:10:51.830342+0000 min lat: 0.125721 max lat: 12.8363 avg lat: 0.640621 2020-11-16T22:10:51.819 INFO:tasks.workunit.client.0.smithi189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T22:10:51.819 INFO:tasks.workunit.client.0.smithi189.stdout: 20 16 513 497 0.097055 0.25 0.300286 0.640621 2020-11-16T22:10:52.819 INFO:tasks.workunit.client.0.smithi189.stdout: 21 16 564 548 0.101918 0.199219 0.300979 0.609087 2020-11-16T22:10:53.819 INFO:tasks.workunit.client.0.smithi189.stdout: 22 16 618 602 0.106872 0.210938 0.305475 0.581983 2020-11-16T22:10:54.819 INFO:tasks.workunit.client.0.smithi189.stdout: 23 16 668 652 0.110716 0.195312 0.311378 0.562267 2020-11-16T22:10:55.819 INFO:tasks.workunit.client.0.smithi189.stdout: 24 16 722 706 0.114891 0.210938 0.31704 0.54 2020-11-16T22:10:56.819 INFO:tasks.workunit.client.0.smithi189.stdout: 25 16 778 762 0.119044 0.21875 0.28625 0.522262 2020-11-16T22:10:57.819 INFO:tasks.workunit.client.0.smithi189.stdout: 26 16 831 815 0.122427 0.207031 0.323415 0.508335 2020-11-16T22:10:58.819 INFO:tasks.workunit.client.0.smithi189.stdout: 27 16 885 869 0.125704 0.210938 0.253482 0.495077 2020-11-16T22:10:59.820 INFO:tasks.workunit.client.0.smithi189.stdout: 28 16 929 913 0.127352 0.171875 0.308635 0.487905 2020-11-16T22:11:00.820 INFO:tasks.workunit.client.0.smithi189.stdout: 29 16 988 972 0.130907 0.230469 0.170167 0.475905 2020-11-16T22:11:01.820 INFO:tasks.workunit.client.0.smithi189.stdout: 30 16 1043 1027 0.133704 0.214844 0.301941 0.465218 2020-11-16T22:11:02.820 INFO:tasks.workunit.client.0.smithi189.stdout:Total time run: 30.2298 2020-11-16T22:11:02.821 INFO:tasks.workunit.client.0.smithi189.stdout:Total writes made: 1043 2020-11-16T22:11:02.821 INFO:tasks.workunit.client.0.smithi189.stdout:Write size: 4096 2020-11-16T22:11:02.821 INFO:tasks.workunit.client.0.smithi189.stdout:Object size: 4096 2020-11-16T22:11:02.821 INFO:tasks.workunit.client.0.smithi189.stdout:Bandwidth (MB/sec): 0.134775 2020-11-16T22:11:02.821 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Bandwidth: 0.0990077 2020-11-16T22:11:02.821 INFO:tasks.workunit.client.0.smithi189.stdout:Max bandwidth (MB/sec): 0.25 2020-11-16T22:11:02.821 INFO:tasks.workunit.client.0.smithi189.stdout:Min bandwidth (MB/sec): 0 2020-11-16T22:11:02.822 INFO:tasks.workunit.client.0.smithi189.stdout:Average IOPS: 34 2020-11-16T22:11:02.822 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev IOPS: 25.3559 2020-11-16T22:11:02.822 INFO:tasks.workunit.client.0.smithi189.stdout:Max IOPS: 64 2020-11-16T22:11:02.822 INFO:tasks.workunit.client.0.smithi189.stdout:Min IOPS: 0 2020-11-16T22:11:02.822 INFO:tasks.workunit.client.0.smithi189.stdout:Average Latency(s): 0.462322 2020-11-16T22:11:02.822 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Latency(s): 1.3205 2020-11-16T22:11:02.822 INFO:tasks.workunit.client.0.smithi189.stdout:Max latency(s): 12.8363 2020-11-16T22:11:02.823 INFO:tasks.workunit.client.0.smithi189.stdout:Min latency(s): 0.125721 2020-11-16T22:11:02.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 635096 2020-11-16T22:11:02.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2020-11-16T22:11:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 635096 2020-11-16T22:11:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (635096) - No such process 2020-11-16T22:11:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2020-11-16T22:11:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already down. 2020-11-16T22:11:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 635817 2020-11-16T22:11:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2020-11-16T22:11:05.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 635817 2020-11-16T22:11:05.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (635817) - No such process 2020-11-16T22:11:05.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2020-11-16T22:11:05.842 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 is already down. 2020-11-16T22:11:05.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 636538 2020-11-16T22:11:05.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2020-11-16T22:11:06.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 636538 2020-11-16T22:11:06.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (636538) - No such process 2020-11-16T22:11:06.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2020-11-16T22:11:07.292 INFO:tasks.workunit.client.0.smithi189.stderr:osd.2 is already down. 2020-11-16T22:11:07.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 637285 2020-11-16T22:11:07.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2020-11-16T22:11:08.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 637285 2020-11-16T22:11:08.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (637285) - No such process 2020-11-16T22:11:08.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2020-11-16T22:11:08.855 INFO:tasks.workunit.client.0.smithi189.stderr:osd.3 is already down. 2020-11-16T22:11:08.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2020-11-16T22:11:10.924 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:11:10.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2020-11-16T22:11:11.417 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records in 2020-11-16T22:11:11.417 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records out 2020-11-16T22:11:11.418 INFO:tasks.workunit.client.0.smithi189.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.488121 s, 2.2 GB/s 2020-11-16T22:11:11.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2020-11-16T22:11:11.434 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:11:19.476 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2020-11-16T22:11:19.477 INFO:tasks.workunit.client.0.smithi189.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2020-11-16T22:11:19.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2020-11-16T22:11:21.288 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:11:21.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2020-11-16T22:11:21.550 INFO:tasks.workunit.client.0.smithi189.stderr:512+0 records in 2020-11-16T22:11:21.550 INFO:tasks.workunit.client.0.smithi189.stderr:512+0 records out 2020-11-16T22:11:21.550 INFO:tasks.workunit.client.0.smithi189.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.25433 s, 2.1 GB/s 2020-11-16T22:11:21.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2020-11-16T22:11:21.566 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:11:26.608 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2020-11-16T22:11:26.609 INFO:tasks.workunit.client.0.smithi189.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2020-11-16T22:11:26.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2020-11-16T22:11:26.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2020-11-16T22:11:28.728 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:11:28.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2020-11-16T22:11:30.796 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:11:30.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2020-11-16T22:11:31.282 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records in 2020-11-16T22:11:31.282 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records out 2020-11-16T22:11:31.283 INFO:tasks.workunit.client.0.smithi189.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.478661 s, 2.2 GB/s 2020-11-16T22:11:31.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2020-11-16T22:11:31.299 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:11:39.588 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2020-11-16T22:11:39.589 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2020-11-16T22:11:39.589 INFO:tasks.workunit.client.0.smithi189.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2020-11-16T22:11:39.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2020-11-16T22:11:39.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2020-11-16T22:11:41.004 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:11:41.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2020-11-16T22:11:41.025 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:11:42.564 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2020-11-16T22:11:42.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2020-11-16T22:11:42.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2020-11-16T22:11:43.480 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:11:43.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2020-11-16T22:11:45.308 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:11:45.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2020-11-16T22:11:45.806 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records in 2020-11-16T22:11:45.806 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records out 2020-11-16T22:11:45.806 INFO:tasks.workunit.client.0.smithi189.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.490455 s, 2.2 GB/s 2020-11-16T22:11:45.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2020-11-16T22:11:45.821 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:11:45.826 INFO:tasks.workunit.client.0.smithi189.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2020-11-16T22:11:45.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2020-11-16T22:11:45.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2020-11-16T22:11:47.628 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:11:47.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2020-11-16T22:11:53.261 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records in 2020-11-16T22:11:53.262 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records out 2020-11-16T22:11:53.262 INFO:tasks.workunit.client.0.smithi189.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.4997 s, 195 MB/s 2020-11-16T22:11:53.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2020-11-16T22:11:53.280 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:11:56.324 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2020-11-16T22:11:56.324 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2020-11-16T22:11:56.325 INFO:tasks.workunit.client.0.smithi189.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2020-11-16T22:11:56.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2020-11-16T22:11:57.896 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:11:57.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2020-11-16T22:11:58.394 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records in 2020-11-16T22:11:58.394 INFO:tasks.workunit.client.0.smithi189.stderr:1024+0 records out 2020-11-16T22:11:58.394 INFO:tasks.workunit.client.0.smithi189.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.48993 s, 2.2 GB/s 2020-11-16T22:11:58.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2020-11-16T22:11:58.411 INFO:tasks.workunit.client.0.smithi189.stdout:inferring bluefs devices from bluestore path 2020-11-16T22:12:06.200 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2020-11-16T22:12:06.201 INFO:tasks.workunit.client.0.smithi189.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2020-11-16T22:12:06.201 INFO:tasks.workunit.client.0.smithi189.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2020-11-16T22:12:06.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2020-11-16T22:12:07.520 INFO:tasks.workunit.client.0.smithi189.stdout:fsck success 2020-11-16T22:12:07.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2020-11-16T22:12:07.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:12:07.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:12:07.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T22:12:07.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:12:07.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2020-11-16T22:12:07.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:12:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:12:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:12:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:12:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2020-11-16T22:12:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2020-11-16T22:12:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:12:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:12:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:12:07.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:12:07.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:12:07.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:12:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:12:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:12:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:12:07.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:12:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:12:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:12:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:12:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:12:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:12:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:12:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:12:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:12:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:12:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2020-11-16T22:12:07.533 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:12:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T22:12:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:12:07.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2020-11-16T22:12:07.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T22:12:07.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:12:07.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:12:07.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:12:07.562 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:07.571+0000 7fd68b67ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:07.565 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:07.575+0000 7fd68b67ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:07.567 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:07.575+0000 7fd68b67ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:07.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T22:12:07.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:12:07.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:12:07.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:12:07.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:12:07.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:07.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:12:07.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:12:07.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:07.921 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:12:08.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:08.602 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:08.611+0000 7fd68b67ed80 -1 Falling back to public interface 2020-11-16T22:12:09.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:09.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:09.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:12:09.323 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:12:09.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:09.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:12:09.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:10.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:10.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:10.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:12:10.723 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:12:10.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:10.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:12:11.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:12.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:12.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:12.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:12:12.137 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:12:12.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:12.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:12:12.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:12.564 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:12.575+0000 7fd68b67ed80 -1 osd.0 64 log_to_monitors {default=true} 2020-11-16T22:12:13.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:13.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:13.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 4 2020-11-16T22:12:13.550 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T22:12:13.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:13.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:12:13.931 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 74 up_thru 74 down_at 65 last_clean_interval [56,64) [v2:127.0.0.1:6802/642316,v1:127.0.0.1:6803/642316] [v2:127.0.0.1:6804/642316,v1:127.0.0.1:6805/642316] exists,up 1f6242ad-048f-45f5-818b-c23c7b6373fd 2020-11-16T22:12:13.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:12:13.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:12:13.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:12:13.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2020-11-16T22:12:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=642316 2020-11-16T22:12:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2020-11-16T22:12:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:12:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:12:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T22:12:13.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:12:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2020-11-16T22:12:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:12:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:12:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:12:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:12:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2020-11-16T22:12:13.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2020-11-16T22:12:13.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:12:13.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:12:13.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:12:13.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:12:13.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:12:13.937 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:12:13.937 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:12:13.937 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:12:13.937 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:12:13.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:12:13.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:12:13.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:12:13.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:12:13.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:12:13.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:12:13.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:12:13.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:12:13.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:12:13.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:12:13.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2020-11-16T22:12:13.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T22:12:13.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:12:13.941 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:12:13.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2020-11-16T22:12:13.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T22:12:13.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:12:13.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:12:13.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:12:13.971 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:13.979+0000 7f359070ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:13.974 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:13.983+0000 7f359070ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:13.977 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:13.987+0000 7f359070ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:14.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T22:12:14.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:12:14.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:12:14.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:12:14.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:12:14.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:14.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:12:14.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:14.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:12:14.358 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:12:14.526 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:14.535+0000 7f359070ad80 -1 Falling back to public interface 2020-11-16T22:12:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:12:15.718 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:12:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:12:16.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:17.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:17.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:17.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:12:17.259 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:12:17.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:17.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:12:17.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:17.926 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:17.935+0000 7f359070ad80 -1 osd.1 67 log_to_monitors {default=true} 2020-11-16T22:12:18.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:18.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:18.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:12:18.659 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:12:18.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:18.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:12:19.110 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 76 up_thru 76 down_at 68 last_clean_interval [58,67) [v2:127.0.0.1:6810/643063,v1:127.0.0.1:6811/643063] [v2:127.0.0.1:6812/643063,v1:127.0.0.1:6813/643063] exists,up dabc47d8-e723-4845-87ab-b470967b9c5f 2020-11-16T22:12:19.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:12:19.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:12:19.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:12:19.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2020-11-16T22:12:19.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=643063 2020-11-16T22:12:19.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2020-11-16T22:12:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:12:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:12:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-11-16T22:12:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:12:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2020-11-16T22:12:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:12:19.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:12:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:12:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:12:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2020-11-16T22:12:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2020-11-16T22:12:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:12:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:12:19.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:12:19.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:12:19.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:12:19.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:12:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:12:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:12:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:12:19.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:12:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:12:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:12:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:12:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:12:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:12:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:12:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:12:19.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:12:19.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:12:19.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2020-11-16T22:12:19.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-11-16T22:12:19.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:12:19.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2020-11-16T22:12:19.119 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:12:19.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-11-16T22:12:19.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:12:19.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:12:19.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:12:19.143 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:19.151+0000 7fb759cb7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:19.150 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:19.155+0000 7fb759cb7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:19.152 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:19.163+0000 7fb759cb7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:19.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-11-16T22:12:19.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:12:19.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:12:19.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:12:19.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:12:19.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:19.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:12:19.506 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:12:19.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:19.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:12:19.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:19.950 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:19.959+0000 7fb759cb7d80 -1 Falling back to public interface 2020-11-16T22:12:20.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:20.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:20.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:12:20.897 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:12:20.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:20.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:12:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:12:22.276 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:12:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:12:22.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:22.791 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:22.799+0000 7fb759cb7d80 -1 osd.2 70 log_to_monitors {default=true} 2020-11-16T22:12:23.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:23.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:23.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:12:23.670 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:12:23.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:23.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:12:24.148 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 79 up_thru 79 down_at 71 last_clean_interval [60,70) [v2:127.0.0.1:6818/643765,v1:127.0.0.1:6819/643765] [v2:127.0.0.1:6820/643765,v1:127.0.0.1:6821/643765] exists,up 538b02af-9658-4dba-ac56-d474d8d8b6b0 2020-11-16T22:12:24.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:12:24.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:12:24.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:12:24.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2020-11-16T22:12:24.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=643765 2020-11-16T22:12:24.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2020-11-16T22:12:24.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:12:24.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:12:24.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2020-11-16T22:12:24.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:12:24.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2020-11-16T22:12:24.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2020-11-16T22:12:24.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:12:24.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:12:24.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:12:24.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2020-11-16T22:12:24.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2020-11-16T22:12:24.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:12:24.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:12:24.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2020-11-16T22:12:24.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:12:24.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:12:24.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:12:24.155 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:12:24.155 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:12:24.155 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:12:24.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:12:24.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' 2020-11-16T22:12:24.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:12:24.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2020-11-16T22:12:24.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2020-11-16T22:12:24.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:12:24.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:12:24.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:12:24.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:12:24.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:12:24.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2020-11-16T22:12:24.158 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T22:12:24.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2020-11-16T22:12:24.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=29ae00e8-84f5-4fe6-a91d-3f1cafd59649 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.612441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:12:24.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2020-11-16T22:12:24.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2020-11-16T22:12:24.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:12:24.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:12:24.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:12:24.182 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:24.191+0000 7f2970f3fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:24.184 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:24.195+0000 7f2970f3fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:24.185 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:24.195+0000 7f2970f3fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:12:24.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2020-11-16T22:12:24.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:12:24.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T22:12:24.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:12:24.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:12:24.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:24.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:12:24.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:24.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:12:24.573 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:12:24.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:24.990 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:24.999+0000 7f2970f3fd80 -1 Falling back to public interface 2020-11-16T22:12:25.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:25.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:25.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:12:25.943 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:12:25.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:25.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:12:26.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:27.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:27.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:27.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-11-16T22:12:27.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:27.333 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:12:27.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:12:27.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:12:28.031 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:12:28.039+0000 7f2970f3fd80 -1 osd.3 72 log_to_monitors {default=true} 2020-11-16T22:12:28.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:12:28.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:12:28.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2020-11-16T22:12:28.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:12:28.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:12:28.756 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:12:29.437 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 82 up_thru 82 down_at 73 last_clean_interval [62,72) [v2:127.0.0.1:6826/644476,v1:127.0.0.1:6827/644476] [v2:127.0.0.1:6828/644476,v1:127.0.0.1:6829/644476] exists,up f8e9f455-b9f3-4cfe-8bb9-33830abd1395 2020-11-16T22:12:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:12:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:12:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:12:29.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2020-11-16T22:12:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=644476 2020-11-16T22:12:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2020-11-16T22:12:29.511 INFO:tasks.workunit.client.0.smithi189.stdout:hints = 1 2020-11-16T22:12:29.511 INFO:tasks.workunit.client.0.smithi189.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2020-11-16T22:12:29.511 INFO:tasks.workunit.client.0.smithi189.stdout:Object prefix: benchmark_data_smithi189_645172 2020-11-16T22:12:29.511 INFO:tasks.workunit.client.0.smithi189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T22:12:29.512 INFO:tasks.workunit.client.0.smithi189.stdout: 0 0 0 0 0 0 - 0 2020-11-16T22:12:30.511 INFO:tasks.workunit.client.0.smithi189.stdout: 1 16 38 22 0.0859327 0.0859375 0.150528 0.260373 2020-11-16T22:12:31.511 INFO:tasks.workunit.client.0.smithi189.stdout: 2 16 38 22 0.0429637 0 - 0.260373 2020-11-16T22:12:32.511 INFO:tasks.workunit.client.0.smithi189.stdout: 3 16 38 22 0.0286421 0 - 0.260373 2020-11-16T22:12:33.511 INFO:tasks.workunit.client.0.smithi189.stdout: 4 16 38 22 0.0214815 0 - 0.260373 2020-11-16T22:12:34.512 INFO:tasks.workunit.client.0.smithi189.stdout: 5 16 38 22 0.0171852 0 - 0.260373 2020-11-16T22:12:35.512 INFO:tasks.workunit.client.0.smithi189.stdout: 6 16 38 22 0.014321 0 - 0.260373 2020-11-16T22:12:36.512 INFO:tasks.workunit.client.0.smithi189.stdout: 7 16 38 22 0.0122751 0 - 0.260373 2020-11-16T22:12:37.512 INFO:tasks.workunit.client.0.smithi189.stdout: 8 16 38 22 0.0107407 0 - 0.260373 2020-11-16T22:12:38.512 INFO:tasks.workunit.client.0.smithi189.stdout: 9 16 38 22 0.00954727 0 - 0.260373 2020-11-16T22:12:39.513 INFO:tasks.workunit.client.0.smithi189.stdout: 10 16 38 22 0.00859255 0 - 0.260373 2020-11-16T22:12:40.512 INFO:tasks.workunit.client.0.smithi189.stdout: 11 16 81 65 0.0230792 0.0167969 0.0882737 1.99747 2020-11-16T22:12:41.513 INFO:tasks.workunit.client.0.smithi189.stdout: 12 16 101 85 0.0276654 0.078125 0.205137 1.57744 2020-11-16T22:12:42.513 INFO:tasks.workunit.client.0.smithi189.stdout: 13 16 115 99 0.0297435 0.0546875 0.0757335 1.37142 2020-11-16T22:12:43.513 INFO:tasks.workunit.client.0.smithi189.stdout: 14 16 172 156 0.0435209 0.222656 0.158885 1.42393 2020-11-16T22:12:44.513 INFO:tasks.workunit.client.0.smithi189.stdout: 15 16 245 229 0.0596273 0.285156 0.141467 1.03917 2020-11-16T22:12:45.513 INFO:tasks.workunit.client.0.smithi189.stdout: 16 16 321 305 0.0744529 0.296875 0.297025 0.834901 2020-11-16T22:12:46.513 INFO:tasks.workunit.client.0.smithi189.stdout: 17 15 386 371 0.0852368 0.257812 0.222283 0.727763 2020-11-16T22:12:47.513 INFO:tasks.workunit.client.0.smithi189.stdout: 18 16 434 418 0.0906996 0.183594 0.160216 0.683018 2020-11-16T22:12:48.514 INFO:tasks.workunit.client.0.smithi189.stdout: 19 16 502 486 0.0999043 0.265625 0.271011 0.622303 2020-11-16T22:12:49.514 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T22:12:49.525852+0000 min lat: 0.0488917 max lat: 13.3403 avg lat: 0.578284 2020-11-16T22:12:49.514 INFO:tasks.workunit.client.0.smithi189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T22:12:49.514 INFO:tasks.workunit.client.0.smithi189.stdout: 20 16 568 552 0.107798 0.257812 0.291884 0.578284 2020-11-16T22:12:50.514 INFO:tasks.workunit.client.0.smithi189.stdout: 21 16 644 628 0.1168 0.296875 0.156865 0.532549 2020-11-16T22:12:51.514 INFO:tasks.workunit.client.0.smithi189.stdout: 22 16 709 693 0.12303 0.253906 0.26371 0.504311 2020-11-16T22:12:52.514 INFO:tasks.workunit.client.0.smithi189.stdout: 23 16 774 758 0.128719 0.253906 0.183575 0.482233 2020-11-16T22:12:53.514 INFO:tasks.workunit.client.0.smithi189.stdout: 24 16 838 822 0.133771 0.25 0.186408 0.465061 2020-11-16T22:12:54.514 INFO:tasks.workunit.client.0.smithi189.stdout: 25 16 918 902 0.140918 0.3125 0.148555 0.441899 2020-11-16T22:12:55.515 INFO:tasks.workunit.client.0.smithi189.stdout: 26 16 981 965 0.144962 0.246094 0.321386 0.427999 2020-11-16T22:12:56.515 INFO:tasks.workunit.client.0.smithi189.stdout: 27 16 1056 1040 0.150442 0.292969 0.158957 0.412929 2020-11-16T22:12:57.515 INFO:tasks.workunit.client.0.smithi189.stdout: 28 16 1121 1105 0.154136 0.253906 0.274502 0.401491 2020-11-16T22:12:58.515 INFO:tasks.workunit.client.0.smithi189.stdout: 29 16 1183 1167 0.157171 0.242188 0.125113 0.395401 2020-11-16T22:12:59.515 INFO:tasks.workunit.client.0.smithi189.stdout: 30 16 1252 1236 0.160916 0.269531 0.146982 0.385868 2020-11-16T22:13:00.515 INFO:tasks.workunit.client.0.smithi189.stdout:Total time run: 30.1505 2020-11-16T22:13:00.516 INFO:tasks.workunit.client.0.smithi189.stdout:Total writes made: 1252 2020-11-16T22:13:00.516 INFO:tasks.workunit.client.0.smithi189.stdout:Write size: 4096 2020-11-16T22:13:00.516 INFO:tasks.workunit.client.0.smithi189.stdout:Object size: 4096 2020-11-16T22:13:00.516 INFO:tasks.workunit.client.0.smithi189.stdout:Bandwidth (MB/sec): 0.162207 2020-11-16T22:13:00.516 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Bandwidth: 0.126388 2020-11-16T22:13:00.516 INFO:tasks.workunit.client.0.smithi189.stdout:Max bandwidth (MB/sec): 0.3125 2020-11-16T22:13:00.517 INFO:tasks.workunit.client.0.smithi189.stdout:Min bandwidth (MB/sec): 0 2020-11-16T22:13:00.517 INFO:tasks.workunit.client.0.smithi189.stdout:Average IOPS: 41 2020-11-16T22:13:00.517 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev IOPS: 32.3668 2020-11-16T22:13:00.517 INFO:tasks.workunit.client.0.smithi189.stdout:Max IOPS: 80 2020-11-16T22:13:00.517 INFO:tasks.workunit.client.0.smithi189.stdout:Min IOPS: 0 2020-11-16T22:13:00.517 INFO:tasks.workunit.client.0.smithi189.stdout:Average Latency(s): 0.384543 2020-11-16T22:13:00.517 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Latency(s): 1.19515 2020-11-16T22:13:00.518 INFO:tasks.workunit.client.0.smithi189.stdout:Max latency(s): 13.3403 2020-11-16T22:13:00.518 INFO:tasks.workunit.client.0.smithi189.stdout:Min latency(s): 0.0488917 2020-11-16T22:13:00.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2020-11-16T22:13:00.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:13:00.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:13:00.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:13:00.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:13:00.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:13:00.645 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:13:00.645 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:13:00.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:13:00.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:13:00.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:13:00.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:13:00.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:13:00.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:13:00.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:13:00.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:13:01.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:13:01.248 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:13:01.248 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:13:01.249 INFO:tasks.workunit.client.0.smithi189.stderr:3' 2020-11-16T22:13:01.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:13:01.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:13:01.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:13:01.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=317827579916 2020-11-16T22:13:01.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 317827579916 2020-11-16T22:13:01.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-317827579916' 2020-11-16T22:13:01.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:13:01.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:13:01.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=326417514507 2020-11-16T22:13:01.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 326417514507 2020-11-16T22:13:01.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-317827579916 1-326417514507' 2020-11-16T22:13:01.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:13:01.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:13:01.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=339302416394 2020-11-16T22:13:01.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 339302416394 2020-11-16T22:13:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-317827579916 1-326417514507 2-339302416394' 2020-11-16T22:13:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:13:01.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:13:01.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=352187318281 2020-11-16T22:13:01.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 352187318281 2020-11-16T22:13:01.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-317827579916 1-326417514507 2-339302416394 3-352187318281' 2020-11-16T22:13:01.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:13:01.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-317827579916 2020-11-16T22:13:01.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:13:01.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:13:01.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-317827579916 2020-11-16T22:13:01.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:13:01.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=317827579916 2020-11-16T22:13:01.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 317827579916' 2020-11-16T22:13:01.891 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 317827579916 2020-11-16T22:13:01.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:13:02.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 317827579914 -lt 317827579916 2020-11-16T22:13:02.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:13:03.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:13:03.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:13:03.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 317827579917 -lt 317827579916 2020-11-16T22:13:03.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:13:03.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-326417514507 2020-11-16T22:13:03.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:13:03.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:13:03.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-326417514507 2020-11-16T22:13:03.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:13:03.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=326417514507 2020-11-16T22:13:03.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 326417514507' 2020-11-16T22:13:03.671 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 326417514507 2020-11-16T22:13:03.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:13:04.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 326417514507 -lt 326417514507 2020-11-16T22:13:04.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:13:04.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-339302416394 2020-11-16T22:13:04.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:13:04.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:13:04.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-339302416394 2020-11-16T22:13:04.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:13:04.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=339302416394 2020-11-16T22:13:04.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 339302416394' 2020-11-16T22:13:04.079 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 339302416394 2020-11-16T22:13:04.080 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:13:04.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 339302416395 -lt 339302416394 2020-11-16T22:13:04.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:13:04.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-352187318281 2020-11-16T22:13:04.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:13:04.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:13:04.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-352187318281 2020-11-16T22:13:04.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:13:04.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=352187318281 2020-11-16T22:13:04.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 352187318281' 2020-11-16T22:13:04.456 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 352187318281 2020-11-16T22:13:04.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:13:04.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 352187318281 -lt 352187318281 2020-11-16T22:13:04.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:13:04.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:13:04.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:13:05.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 16 == 0 2020-11-16T22:13:05.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:13:05.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:13:05.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:13:05.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:13:05.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:13:05.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:13:05.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:13:05.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=16 2020-11-16T22:13:05.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:13:05.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:13:05.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:13:06.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 16 = 16 2020-11-16T22:13:06.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:13:06.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:13:06.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:25: run: teardown td/osd-bluefs-volume-ops 2020-11-16T22:13:06.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:13:06.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:13:06.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2020-11-16T22:13:06.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:06.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:06.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:06.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:06.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:06.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:06.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:06.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:06.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:06.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:06.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:06.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:06.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:06.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:06.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:06.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:06.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:06.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:13:06.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2020-11-16T22:13:06.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:06.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:06.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:13:06.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.612441 2020-11-16T22:13:06.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:06.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:06.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T22:13:06.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-bluefs-volume-ops 0 2020-11-16T22:13:06.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2020-11-16T22:13:06.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T22:13:06.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2020-11-16T22:13:06.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:06.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:06.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:06.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:06.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:06.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:06.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:06.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:06.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:06.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:06.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:06.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:06.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:06.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:06.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:06.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T22:13:06.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2020-11-16T22:13:06.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:06.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:06.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.612441 2020-11-16T22:13:06.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.612441 2020-11-16T22:13:06.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:06.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:06.678 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T22:13:06.680 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T22:13:06.680 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T22:13:06.734 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2020-11-16T22:13:06.735 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2020-11-16T22:13:06.793 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T22:13:06.805 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T22:13:06.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:13:06.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:13:06.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T22:13:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T22:13:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T22:13:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T22:13:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T22:13:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T22:13:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-config 2020-11-16T22:13:06.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2020-11-16T22:13:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2020-11-16T22:13:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2020-11-16T22:13:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2020-11-16T22:13:06.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2020-11-16T22:13:06.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2020-11-16T22:13:06.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=246780c7-40b1-41d9-b326-8d3ae2cb8441 --auth-supported=none ' 2020-11-16T22:13:06.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2020-11-16T22:13:06.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2020-11-16T22:13:06.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T22:13:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2020-11-16T22:13:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_config_track 2020-11-16T22:13:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_default_adjustment' 2020-11-16T22:13:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2020-11-16T22:13:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2020-11-16T22:13:06.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2020-11-16T22:13:06.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2020-11-16T22:13:06.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2020-11-16T22:13:06.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:13:06.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2020-11-16T22:13:06.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:06.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:06.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:06.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:06.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:06.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:06.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:06.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:06.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:06.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:06.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:06.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:06.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:06.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:13:06.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2020-11-16T22:13:06.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:06.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:06.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:06.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.645705 2020-11-16T22:13:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:06.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2020-11-16T22:13:06.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:13:06.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:06.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:06.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.645705 2020-11-16T22:13:06.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:13:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T22:13:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T22:13:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:13:06.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2020-11-16T22:13:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2020-11-16T22:13:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2020-11-16T22:13:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2020-11-16T22:13:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2020-11-16T22:13:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:13:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:13:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:13:06.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2020-11-16T22:13:06.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2020-11-16T22:13:07.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:13:07.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:07.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:07.136 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:07.136 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:07.136 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:07.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.645705/$cluster-$name.asok' 2020-11-16T22:13:07.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.645705/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:13:07.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:13:07.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:13:07.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:13:07.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:13:07.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:13:07.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:13:07.326 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:13:07.326 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:13:07.327 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:13:07.327 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:13:07.328 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:07.328 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:07.328 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.645705/ceph-mon.a.asok 2020-11-16T22:13:07.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:13:07.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.645705/ceph-mon.a.asok config get fsid 2020-11-16T22:13:07.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:13:07.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:13:07.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:13:07.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:13:07.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:13:07.429 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:13:07.429 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:13:07.430 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:13:07.430 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:13:07.431 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:07.431 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:07.431 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.645705/ceph-mon.a.asok 2020-11-16T22:13:07.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:13:07.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.645705/ceph-mon.a.asok config get mon_host 2020-11-16T22:13:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2020-11-16T22:13:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-config 2020-11-16T22:13:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:13:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:13:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:13:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-config/x 2020-11-16T22:13:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:13:08.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:13:08.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:08.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:08.055 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:08.055 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:08.055 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:08.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.645705/$cluster-$name.asok' 2020-11-16T22:13:08.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:13:08.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.645705/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:13:08.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2020-11-16T22:13:08.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2020-11-16T22:13:08.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2020-11-16T22:13:08.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-config 2020-11-16T22:13:08.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:13:08.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:13:08.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:13:08.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-config/0 2020-11-16T22:13:08.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=246780c7-40b1-41d9-b326-8d3ae2cb8441 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2020-11-16T22:13:08.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:13:08.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:13:08.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:13:08.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2020-11-16T22:13:08.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2020-11-16T22:13:08.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:13:08.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:13:08.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-config' 2020-11-16T22:13:08.084 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:13:08.084 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:08.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:08.086 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:08.086 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:08.086 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:08.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.645705/$cluster-$name.asok' 2020-11-16T22:13:08.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.645705/$cluster-$name.asok' 2020-11-16T22:13:08.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:13:08.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:13:08.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:13:08.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2020-11-16T22:13:08.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2020-11-16T22:13:08.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:13:08.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:13:08.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:08.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:13:08.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2020-11-16T22:13:08.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-config/0 2020-11-16T22:13:08.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:13:08.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=20db1803-e9d8-48dd-a453-1c343e456140 2020-11-16T22:13:08.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 20db1803-e9d8-48dd-a453-1c343e456140' 2020-11-16T22:13:08.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:13:08.090 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 20db1803-e9d8-48dd-a453-1c343e456140 2020-11-16T22:13:08.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB0+bJfCOfFBhAAXeyNRGQlWJjMh46IeX2viw== 2020-11-16T22:13:08.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB0+bJfCOfFBhAAXeyNRGQlWJjMh46IeX2viw=="}' 2020-11-16T22:13:08.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 20db1803-e9d8-48dd-a453-1c343e456140 -i td/osd-config/0/new.json 2020-11-16T22:13:08.455 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:13:08.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-config/0/new.json 2020-11-16T22:13:08.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=246780c7-40b1-41d9-b326-8d3ae2cb8441 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.645705/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQB0+bJfCOfFBhAAXeyNRGQlWJjMh46IeX2viw== --osd-uuid 20db1803-e9d8-48dd-a453-1c343e456140 2020-11-16T22:13:08.500 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:08.506+0000 7f9903e7fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:08.503 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:08.510+0000 7f9903e7fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:08.504 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:08.514+0000 7f9903e7fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:08.505 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:08.514+0000 7f9903e7fd80 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2020-11-16T22:13:08.505 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:08.514+0000 7f9903e7fd80 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2020-11-16T22:13:08.505 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:08.514+0000 7f9903e7fd80 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2020-11-16T22:13:08.573 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:08.582+0000 7f9903e7fd80 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2020-11-16T22:13:12.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-config/0/keyring 2020-11-16T22:13:12.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:13:12.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:13:12.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:13:12.943 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:13:13.442 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:13:13.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:13:13.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=246780c7-40b1-41d9-b326-8d3ae2cb8441 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.645705/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2020-11-16T22:13:13.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:13:13.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:13:13.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:13:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:13.478+0000 7fd9e3c72d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:13.472 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:13.482+0000 7fd9e3c72d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:13.474 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:13.482+0000 7fd9e3c72d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:13.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:13:13.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:13:13.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:13:13.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:13:13.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:13:13.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:13.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:13:13.810 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:13:13.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:13.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:13:14.018 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:14.026+0000 7fd9e3c72d80 -1 Falling back to public interface 2020-11-16T22:13:14.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:13:14.854 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:14.862+0000 7fd9e3c72d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:13:15.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:13:15.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:15.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:13:15.204 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:13:15.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:15.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:13:15.619 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/646558,v1:127.0.0.1:6803/646558] [v2:127.0.0.1:6804/646558,v1:127.0.0.1:6805/646558] exists,up 20db1803-e9d8-48dd-a453-1c343e456140 2020-11-16T22:13:15.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:13:15.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:13:15.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:13:15.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2020-11-16T22:13:15.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T22:13:15.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T22:13:15.621 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:13:15.622 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:15.622 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:15.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.645705/ceph-osd.0.asok 2020-11-16T22:13:15.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2020-11-16T22:13:15.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.645705/ceph-osd.0.asok log flush 2020-11-16T22:13:15.726 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2020-11-16T22:13:15.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2020-11-16T22:13:15.737 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T22:13:15.150+0000 7fd9e3c72d80 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2020-11-16T22:13:15.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2020-11-16T22:13:15.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2020-11-16T22:13:15.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:13:15.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2020-11-16T22:13:15.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:15.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:15.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:15.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:15.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:15.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:15.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:15.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:15.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:15.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:15.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:15.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:15.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:15.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:15.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:15.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:15.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:15.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:13:15.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2020-11-16T22:13:15.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:15.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:15.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:15.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.645705 2020-11-16T22:13:15.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:15.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:15.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2020-11-16T22:13:15.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2020-11-16T22:13:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2020-11-16T22:13:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2020-11-16T22:13:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2020-11-16T22:13:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:13:15.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2020-11-16T22:13:15.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:15.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:15.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:15.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:15.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:15.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:15.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:15.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:15.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:15.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:15.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:15.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:15.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:15.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:15.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:15.878 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:15.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:15.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:13:15.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2020-11-16T22:13:15.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:15.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:15.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:15.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.645705 2020-11-16T22:13:15.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:15.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:15.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2020-11-16T22:13:15.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:13:15.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:15.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:15.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.645705 2020-11-16T22:13:15.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:13:15.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T22:13:15.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:13:15.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2020-11-16T22:13:15.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2020-11-16T22:13:15.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2020-11-16T22:13:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2020-11-16T22:13:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2020-11-16T22:13:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:13:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:13:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:13:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2020-11-16T22:13:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2020-11-16T22:13:16.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:13:16.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:16.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:16.187 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:16.187 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:16.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.645705/$cluster-$name.asok' 2020-11-16T22:13:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.645705/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:13:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:13:16.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:13:16.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:13:16.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:13:16.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:13:16.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:13:16.409 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:13:16.409 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:13:16.409 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:13:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:13:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.645705/ceph-mon.a.asok 2020-11-16T22:13:16.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:13:16.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.645705/ceph-mon.a.asok config get fsid 2020-11-16T22:13:16.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:13:16.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:13:16.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:13:16.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:13:16.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:13:16.517 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:13:16.517 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:13:16.517 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:13:16.517 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:13:16.518 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:16.518 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:16.518 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.645705/ceph-mon.a.asok 2020-11-16T22:13:16.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:13:16.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.645705/ceph-mon.a.asok config get mon_host 2020-11-16T22:13:16.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2020-11-16T22:13:16.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-config 2020-11-16T22:13:16.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:13:16.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:13:16.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:13:16.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-config/x 2020-11-16T22:13:16.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:13:17.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:13:17.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:17.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:17.170 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:17.171 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:17.171 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:17.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.645705/$cluster-$name.asok' 2020-11-16T22:13:17.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:13:17.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.645705/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:13:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2020-11-16T22:13:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-config 2020-11-16T22:13:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:13:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:13:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:13:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-config/0 2020-11-16T22:13:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=246780c7-40b1-41d9-b326-8d3ae2cb8441 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2020-11-16T22:13:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:13:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:13:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:13:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2020-11-16T22:13:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2020-11-16T22:13:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:13:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:13:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-config' 2020-11-16T22:13:17.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:13:17.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:17.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:17.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.645705/$cluster-$name.asok' 2020-11-16T22:13:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.645705/$cluster-$name.asok' 2020-11-16T22:13:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:13:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:13:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:13:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2020-11-16T22:13:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2020-11-16T22:13:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:13:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:13:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:13:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:13:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-config/0 2020-11-16T22:13:17.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:13:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c99f154d-25d4-41e8-acd2-1d5c3213a5c7 2020-11-16T22:13:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 c99f154d-25d4-41e8-acd2-1d5c3213a5c7' 2020-11-16T22:13:17.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:13:17.209 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 c99f154d-25d4-41e8-acd2-1d5c3213a5c7 2020-11-16T22:13:17.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB9+bJfikEEDhAAqpgD7nNdUCjQ74aU4ydrnw== 2020-11-16T22:13:17.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB9+bJfikEEDhAAqpgD7nNdUCjQ74aU4ydrnw=="}' 2020-11-16T22:13:17.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c99f154d-25d4-41e8-acd2-1d5c3213a5c7 -i td/osd-config/0/new.json 2020-11-16T22:13:17.550 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:13:17.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-config/0/new.json 2020-11-16T22:13:17.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=246780c7-40b1-41d9-b326-8d3ae2cb8441 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.645705/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB9+bJfikEEDhAAqpgD7nNdUCjQ74aU4ydrnw== --osd-uuid c99f154d-25d4-41e8-acd2-1d5c3213a5c7 2020-11-16T22:13:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:17.598+0000 7fa63a8e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:17.594 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:17.602+0000 7fa63a8e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:17.602+0000 7fa63a8e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:17.606+0000 7fa63a8e9d80 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2020-11-16T22:13:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:17.606+0000 7fa63a8e9d80 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2020-11-16T22:13:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:17.606+0000 7fa63a8e9d80 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2020-11-16T22:13:17.632 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:17.642+0000 7fa63a8e9d80 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2020-11-16T22:13:22.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-config/0/keyring 2020-11-16T22:13:22.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:13:22.197 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:13:22.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:13:22.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:13:22.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:13:22.693 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:13:22.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=246780c7-40b1-41d9-b326-8d3ae2cb8441 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.645705/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:22.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:13:22.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:13:22.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:13:22.717 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:22.726+0000 7f5a41bd9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:22.719 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:22.726+0000 7f5a41bd9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:22.720 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:22.730+0000 7f5a41bd9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:13:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:13:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:13:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:13:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:13:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:13:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:23.097 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:13:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:13:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:23.274+0000 7f5a41bd9d80 -1 Falling back to public interface 2020-11-16T22:13:23.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:13:24.122 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:24.130+0000 7f5a41bd9d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:13:24.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:13:24.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:24.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:13:24.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:24.452 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:13:24.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:13:24.848 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/647816,v1:127.0.0.1:6803/647816] [v2:127.0.0.1:6804/647816,v1:127.0.0.1:6805/647816] exists,up c99f154d-25d4-41e8-acd2-1d5c3213a5c7 2020-11-16T22:13:24.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:13:24.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:13:24.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:13:24.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2020-11-16T22:13:24.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2020-11-16T22:13:24.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2020-11-16T22:13:24.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2020-11-16T22:13:24.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2020-11-16T22:13:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2020-11-16T22:13:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2020-11-16T22:13:24.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2020-11-16T22:13:24.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2020-11-16T22:13:25.023 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2020-11-16T22:13:25.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2020-11-16T22:13:25.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T22:13:25.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T22:13:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:13:25.035 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:25.036 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:25.036 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.645705/ceph-osd.0.asok 2020-11-16T22:13:25.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2020-11-16T22:13:25.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.645705/ceph-osd.0.asok log flush 2020-11-16T22:13:25.137 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2020-11-16T22:13:25.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2020-11-16T22:13:25.148 INFO:tasks.workunit.client.0.smithi189.stdout:2020-11-16T22:13:25.018+0000 7f5a3c0a9700 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2020-11-16T22:13:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2020-11-16T22:13:25.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2020-11-16T22:13:25.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T22:13:25.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T22:13:25.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:13:25.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:25.151 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:25.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.645705/ceph-osd.0.asok 2020-11-16T22:13:25.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2020-11-16T22:13:25.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.645705/ceph-osd.0.asok log reopen 2020-11-16T22:13:25.252 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2020-11-16T22:13:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2020-11-16T22:13:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2020-11-16T22:13:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:13:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2020-11-16T22:13:25.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:25.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:25.375 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:25.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:25.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:25.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:25.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:25.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:25.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:25.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:25.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:13:25.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2020-11-16T22:13:25.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:25.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:25.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:25.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.645705 2020-11-16T22:13:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2020-11-16T22:13:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2020-11-16T22:13:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2020-11-16T22:13:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2020-11-16T22:13:25.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2020-11-16T22:13:25.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:13:25.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2020-11-16T22:13:25.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:25.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:25.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:25.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:25.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:25.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:25.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:25.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:25.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:25.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:25.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:25.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:25.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:25.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:25.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:13:25.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2020-11-16T22:13:25.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:25.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:25.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:25.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.645705 2020-11-16T22:13:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:25.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2020-11-16T22:13:25.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:13:25.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:25.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:25.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.645705 2020-11-16T22:13:25.419 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:13:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T22:13:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:13:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2020-11-16T22:13:25.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2020-11-16T22:13:25.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2020-11-16T22:13:25.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2020-11-16T22:13:25.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2020-11-16T22:13:25.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2020-11-16T22:13:25.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2020-11-16T22:13:25.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2020-11-16T22:13:25.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2020-11-16T22:13:25.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2020-11-16T22:13:25.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2020-11-16T22:13:25.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2020-11-16T22:13:25.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2020-11-16T22:13:25.503 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2020-11-16T22:13:25.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2020-11-16T22:13:25.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2020-11-16T22:13:25.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2020-11-16T22:13:25.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2020-11-16T22:13:25.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2020-11-16T22:13:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2020-11-16T22:13:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2020-11-16T22:13:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2020-11-16T22:13:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:13:25.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2020-11-16T22:13:25.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:25.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:25.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:25.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:25.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:25.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:25.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:25.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:25.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:25.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:13:25.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2020-11-16T22:13:25.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:25.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:25.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:25.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.645705 2020-11-16T22:13:25.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:25.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:25.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T22:13:25.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-config 0 2020-11-16T22:13:25.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2020-11-16T22:13:25.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T22:13:25.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2020-11-16T22:13:25.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:25.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:25.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:25.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:25.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:25.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:25.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:25.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:25.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:25.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:25.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:25.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:25.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:25.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:25.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:25.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:25.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:25.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T22:13:25.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2020-11-16T22:13:25.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:25.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:25.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.645705 2020-11-16T22:13:25.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.645705 2020-11-16T22:13:25.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:25.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:25.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T22:13:25.580 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T22:13:25.580 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T22:13:25.636 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2020-11-16T22:13:25.636 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2020-11-16T22:13:25.697 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T22:13:25.710 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T22:13:25.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:13:25.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:13:25.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T22:13:25.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T22:13:25.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T22:13:25.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T22:13:25.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T22:13:25.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T22:13:25.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-copy-from 2020-11-16T22:13:25.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2020-11-16T22:13:25.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2020-11-16T22:13:25.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2020-11-16T22:13:25.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2020-11-16T22:13:25.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2020-11-16T22:13:25.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2020-11-16T22:13:25.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=ec0ae14c-3be4-49c0-8878-7e6ab0524798 --auth-supported=none ' 2020-11-16T22:13:25.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2020-11-16T22:13:25.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2020-11-16T22:13:25.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T22:13:25.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2020-11-16T22:13:25.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2020-11-16T22:13:25.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2020-11-16T22:13:25.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2020-11-16T22:13:25.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2020-11-16T22:13:25.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2020-11-16T22:13:25.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:13:25.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2020-11-16T22:13:25.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:25.718 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:25.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:25.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:25.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:25.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:25.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:25.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:25.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:25.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:25.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:25.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:25.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:25.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:25.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:25.727 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:25.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:25.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:13:25.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2020-11-16T22:13:25.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:25.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:25.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.648326 2020-11-16T22:13:25.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.648326 2020-11-16T22:13:25.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:25.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:25.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2020-11-16T22:13:25.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:13:25.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:25.736 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.648326 2020-11-16T22:13:25.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.648326 2020-11-16T22:13:25.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:13:25.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T22:13:25.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T22:13:25.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:13:25.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2020-11-16T22:13:25.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2020-11-16T22:13:25.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2020-11-16T22:13:25.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2020-11-16T22:13:25.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2020-11-16T22:13:25.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:13:25.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:13:25.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:13:25.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2020-11-16T22:13:25.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2020-11-16T22:13:26.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:13:26.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:26.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:26.055 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:26.055 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:26.056 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.648326 2020-11-16T22:13:26.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.648326/$cluster-$name.asok' 2020-11-16T22:13:26.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.648326/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:13:26.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:13:26.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:13:26.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:13:26.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:13:26.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:13:26.261 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:13:26.262 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:13:26.262 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:13:26.262 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:13:26.263 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:13:26.263 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:26.263 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.648326 2020-11-16T22:13:26.263 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.648326/ceph-mon.a.asok 2020-11-16T22:13:26.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:13:26.264 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.648326/ceph-mon.a.asok config get fsid 2020-11-16T22:13:26.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:13:26.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:13:26.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:13:26.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:13:26.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:13:26.372 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:13:26.373 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:13:26.373 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:13:26.373 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:13:26.373 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:26.373 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.648326 2020-11-16T22:13:26.373 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.648326/ceph-mon.a.asok 2020-11-16T22:13:26.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:13:26.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.648326/ceph-mon.a.asok config get mon_host 2020-11-16T22:13:26.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2020-11-16T22:13:26.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-copy-from 2020-11-16T22:13:26.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:13:26.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:13:26.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:13:26.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-copy-from/x 2020-11-16T22:13:26.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:13:27.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:13:27.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:27.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:27.019 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:27.020 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:27.020 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.648326 2020-11-16T22:13:27.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.648326/$cluster-$name.asok' 2020-11-16T22:13:27.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:13:27.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.648326/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:13:27.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2020-11-16T22:13:27.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-copy-from 2020-11-16T22:13:27.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:13:27.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:13:27.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:13:27.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-copy-from/0 2020-11-16T22:13:27.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=ec0ae14c-3be4-49c0-8878-7e6ab0524798 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2020-11-16T22:13:27.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:13:27.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:13:27.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:13:27.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2020-11-16T22:13:27.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2020-11-16T22:13:27.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:13:27.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:13:27.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2020-11-16T22:13:27.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:13:27.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:27.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:27.049 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:27.049 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:27.049 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.648326 2020-11-16T22:13:27.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.648326/$cluster-$name.asok' 2020-11-16T22:13:27.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.648326/$cluster-$name.asok' 2020-11-16T22:13:27.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:13:27.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:13:27.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:13:27.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2020-11-16T22:13:27.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2020-11-16T22:13:27.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:13:27.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:13:27.051 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:27.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:13:27.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:13:27.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-copy-from/0 2020-11-16T22:13:27.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:13:27.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2474b1e6-5c00-42ff-904c-e9ac368a1367 2020-11-16T22:13:27.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 2474b1e6-5c00-42ff-904c-e9ac368a1367' 2020-11-16T22:13:27.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:13:27.053 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 2474b1e6-5c00-42ff-904c-e9ac368a1367 2020-11-16T22:13:27.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCH+bJfrVGgBBAAe0MqP15+1kH/UFzpzEWVOw== 2020-11-16T22:13:27.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCH+bJfrVGgBBAAe0MqP15+1kH/UFzpzEWVOw=="}' 2020-11-16T22:13:27.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2474b1e6-5c00-42ff-904c-e9ac368a1367 -i td/osd-copy-from/0/new.json 2020-11-16T22:13:27.386 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:13:27.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-copy-from/0/new.json 2020-11-16T22:13:27.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=ec0ae14c-3be4-49c0-8878-7e6ab0524798 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.648326/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCH+bJfrVGgBBAAe0MqP15+1kH/UFzpzEWVOw== --osd-uuid 2474b1e6-5c00-42ff-904c-e9ac368a1367 2020-11-16T22:13:27.424 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:27.434+0000 7fa7f4d5dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:27.427 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:27.434+0000 7fa7f4d5dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:27.429 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:27.438+0000 7fa7f4d5dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:27.430 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:27.438+0000 7fa7f4d5dd80 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2020-11-16T22:13:27.430 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:27.438+0000 7fa7f4d5dd80 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2020-11-16T22:13:27.430 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:27.438+0000 7fa7f4d5dd80 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2020-11-16T22:13:27.469 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:27.478+0000 7fa7f4d5dd80 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2020-11-16T22:13:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-copy-from/0/keyring 2020-11-16T22:13:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:13:31.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:13:31.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:13:31.739 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:13:32.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:13:32.211 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:13:32.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=ec0ae14c-3be4-49c0-8878-7e6ab0524798 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.648326/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:32.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:13:32.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:13:32.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:13:32.237 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:32.246+0000 7fb410422d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:32.239 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:32.246+0000 7fb410422d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:32.240 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:32.250+0000 7fb410422d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:32.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:13:32.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:13:32.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:13:32.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:13:32.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:13:32.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:32.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:13:32.582 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:13:32.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:32.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:13:32.794 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:32.802+0000 7fb410422d80 -1 Falling back to public interface 2020-11-16T22:13:32.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:13:33.643 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:33.650+0000 7fb410422d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:13:33.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:13:33.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:33.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:13:33.989 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:13:33.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:33.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:13:34.400 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/649179,v1:127.0.0.1:6803/649179] [v2:127.0.0.1:6804/649179,v1:127.0.0.1:6805/649179] exists,up 2474b1e6-5c00-42ff-904c-e9ac368a1367 2020-11-16T22:13:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:13:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:13:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:13:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2020-11-16T22:13:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-copy-from 2020-11-16T22:13:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:13:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:13:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:13:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-copy-from/1 2020-11-16T22:13:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=ec0ae14c-3be4-49c0-8878-7e6ab0524798 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2020-11-16T22:13:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:13:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:13:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:13:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2020-11-16T22:13:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2020-11-16T22:13:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:13:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:13:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2020-11-16T22:13:34.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:13:34.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:34.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:34.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:34.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.648326 2020-11-16T22:13:34.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.648326/$cluster-$name.asok' 2020-11-16T22:13:34.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.648326/$cluster-$name.asok' 2020-11-16T22:13:34.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:13:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:13:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:13:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2020-11-16T22:13:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2020-11-16T22:13:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:13:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:13:34.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:13:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:13:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-copy-from/1 2020-11-16T22:13:34.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:13:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=192c31c6-3160-4182-95f5-b564ae377edf 2020-11-16T22:13:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 192c31c6-3160-4182-95f5-b564ae377edf' 2020-11-16T22:13:34.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:13:34.409 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 192c31c6-3160-4182-95f5-b564ae377edf 2020-11-16T22:13:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCO+bJfFy56GRAAr/X7ImL4OQyF8HSKktWc8Q== 2020-11-16T22:13:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCO+bJfFy56GRAAr/X7ImL4OQyF8HSKktWc8Q=="}' 2020-11-16T22:13:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 192c31c6-3160-4182-95f5-b564ae377edf -i td/osd-copy-from/1/new.json 2020-11-16T22:13:34.892 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:13:34.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-copy-from/1/new.json 2020-11-16T22:13:34.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=ec0ae14c-3be4-49c0-8878-7e6ab0524798 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.648326/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCO+bJfFy56GRAAr/X7ImL4OQyF8HSKktWc8Q== --osd-uuid 192c31c6-3160-4182-95f5-b564ae377edf 2020-11-16T22:13:34.934 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:34.942+0000 7f756bee8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:34.936 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:34.946+0000 7f756bee8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:34.938 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:34.946+0000 7f756bee8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:34.938 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:34.946+0000 7f756bee8d80 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2020-11-16T22:13:34.938 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:34.946+0000 7f756bee8d80 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2020-11-16T22:13:34.938 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:34.946+0000 7f756bee8d80 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2020-11-16T22:13:35.014 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:35.026+0000 7f756bee8d80 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2020-11-16T22:13:39.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-copy-from/1/keyring 2020-11-16T22:13:39.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:13:39.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:13:39.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:13:39.448 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:13:39.938 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:13:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:13:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:13:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:13:39.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:13:39.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=ec0ae14c-3be4-49c0-8878-7e6ab0524798 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.648326/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:39.964 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:39.970+0000 7fc3fc5e1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:39.966 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:39.974+0000 7fc3fc5e1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:39.968 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:39.978+0000 7fc3fc5e1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:40.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:13:40.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:13:40.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:13:40.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:13:40.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:13:40.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:40.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:13:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:13:40.326 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:13:40.502 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:40.510+0000 7fc3fc5e1d80 -1 Falling back to public interface 2020-11-16T22:13:40.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:13:41.433 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:41.442+0000 7fc3fc5e1d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:13:41.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:13:41.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:41.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:13:41.722 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:13:41.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:41.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:13:42.194 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/650173,v1:127.0.0.1:6811/650173] [v2:127.0.0.1:6812/650173,v1:127.0.0.1:6813/650173] exists,up 192c31c6-3160-4182-95f5-b564ae377edf 2020-11-16T22:13:42.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:13:42.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:13:42.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:13:42.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2020-11-16T22:13:42.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2020-11-16T22:13:42.658 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'rbd' does not exist 2020-11-16T22:13:42.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2020-11-16T22:13:42.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-11-16T22:13:43.187 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'rbd' already exists 2020-11-16T22:13:43.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:13:44.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2020-11-16T22:13:45.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2020-11-16T22:13:45.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2020-11-16T22:13:45.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2020-11-16T22:13:45.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2020-11-16T22:13:45.690 INFO:tasks.workunit.client.0.smithi189.stdout:rbd/foo2 mtime 2020-11-16T22:13:45.000000+0000, size 657688 2020-11-16T22:13:45.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2020-11-16T22:13:45.846 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0: {} 2020-11-16T22:13:45.879 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1: {} 2020-11-16T22:13:45.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2020-11-16T22:13:45.926 INFO:tasks.workunit.client.0.smithi189.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2020-11-16T22:13:45.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2020-11-16T22:13:45.956 INFO:tasks.workunit.client.0.smithi189.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2020-11-16T22:13:45.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2020-11-16T22:13:46.105 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0: {} 2020-11-16T22:13:46.139 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1: {} 2020-11-16T22:13:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2020-11-16T22:13:46.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2020-11-16T22:13:46.337 INFO:tasks.workunit.client.0.smithi189.stdout:rbd/foo3 mtime 2020-11-16T22:13:45.000000+0000, size 657688 2020-11-16T22:13:46.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2020-11-16T22:13:46.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2020-11-16T22:13:46.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:13:46.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2020-11-16T22:13:46.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:46.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:46.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:46.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:46.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:46.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:46.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:46.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:46.460 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:46.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:46.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:46.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:46.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:46.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:46.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:46.464 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:46.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:46.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:13:46.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2020-11-16T22:13:46.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:46.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:46.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.648326 2020-11-16T22:13:46.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.648326 2020-11-16T22:13:46.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:46.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:46.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T22:13:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-copy-from 0 2020-11-16T22:13:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2020-11-16T22:13:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T22:13:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2020-11-16T22:13:46.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:46.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:46.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:46.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:46.484 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:46.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:46.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:46.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:46.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T22:13:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2020-11-16T22:13:46.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:46.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:46.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.648326 2020-11-16T22:13:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.648326 2020-11-16T22:13:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T22:13:46.498 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T22:13:46.498 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T22:13:46.558 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2020-11-16T22:13:46.558 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2020-11-16T22:13:46.621 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T22:13:46.635 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T22:13:46.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:13:46.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:13:46.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T22:13:46.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T22:13:46.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T22:13:46.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T22:13:46.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T22:13:46.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T22:13:46.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-dup 2020-11-16T22:13:46.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2020-11-16T22:13:46.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2020-11-16T22:13:46.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2020-11-16T22:13:46.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2020-11-16T22:13:46.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2020-11-16T22:13:46.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2020-11-16T22:13:46.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none ' 2020-11-16T22:13:46.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2020-11-16T22:13:46.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2020-11-16T22:13:46.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2020-11-16T22:13:46.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2020-11-16T22:13:46.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T22:13:46.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs=TEST_filestore_to_bluestore 2020-11-16T22:13:46.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:19: run: for func in $funcs 2020-11-16T22:13:46.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:20: run: setup td/osd-dup 2020-11-16T22:13:46.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-dup 2020-11-16T22:13:46.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-dup 2020-11-16T22:13:46.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2020-11-16T22:13:46.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:13:46.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2020-11-16T22:13:46.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:13:46.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:13:46.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:13:46.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:13:46.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:13:46.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:13:46.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:13:46.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:13:46.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:13:46.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:13:46.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:13:46.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:13:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:13:46.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:13:46.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:13:46.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:13:46.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:13:46.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2020-11-16T22:13:46.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:13:46.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:46.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:13:46.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.650871 2020-11-16T22:13:46.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:13:46.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:13:46.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-dup 2020-11-16T22:13:46.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:13:46.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:46.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:13:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.650871 2020-11-16T22:13:46.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:13:46.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T22:13:46.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T22:13:46.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:13:46.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-dup 1' TERM HUP INT 2020-11-16T22:13:46.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:21: run: TEST_filestore_to_bluestore td/osd-dup 2020-11-16T22:13:46.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:27: TEST_filestore_to_bluestore: local dir=td/osd-dup 2020-11-16T22:13:46.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: ulimit -n 2020-11-16T22:13:46.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: local flimit=4096 2020-11-16T22:13:46.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:30: TEST_filestore_to_bluestore: '[' 4096 -lt 1536 ']' 2020-11-16T22:13:46.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:34: TEST_filestore_to_bluestore: run_mon td/osd-dup a 2020-11-16T22:13:46.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-dup 2020-11-16T22:13:46.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:13:46.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:13:46.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:13:46.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-dup/a 2020-11-16T22:13:46.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-dup/a --run-dir=td/osd-dup 2020-11-16T22:13:46.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:13:46.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:46.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:46.985 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:46.986 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:46.986 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:13:46.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.650871/$cluster-$name.asok' 2020-11-16T22:13:46.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-dup/a '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' --mon-cluster-log-file=td/osd-dup/log --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:13:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:13:47.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:13:47.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:13:47.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:13:47.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:13:47.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:13:47.184 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:13:47.184 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:13:47.185 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:13:47.185 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:13:47.185 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:47.185 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:13:47.186 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.650871/ceph-mon.a.asok 2020-11-16T22:13:47.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:13:47.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.650871/ceph-mon.a.asok config get fsid 2020-11-16T22:13:47.290 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:13:47.290 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:13:47.290 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:13:47.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:13:47.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:13:47.292 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:13:47.292 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:13:47.292 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:13:47.292 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:13:47.292 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:47.292 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:13:47.293 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.650871/ceph-mon.a.asok 2020-11-16T22:13:47.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:13:47.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.650871/ceph-mon.a.asok config get mon_host 2020-11-16T22:13:47.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:35: TEST_filestore_to_bluestore: run_mgr td/osd-dup x 2020-11-16T22:13:47.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-dup 2020-11-16T22:13:47.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:13:47.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:13:47.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:13:47.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-dup/x 2020-11-16T22:13:47.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:13:47.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:13:47.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:47.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:47.947 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:47.947 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:47.948 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:13:47.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.650871/$cluster-$name.asok' 2020-11-16T22:13:47.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:13:47.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-dup/x '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:13:47.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:36: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 0 2020-11-16T22:13:47.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local dir=td/osd-dup 2020-11-16T22:13:47.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2020-11-16T22:13:47.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local id=0 2020-11-16T22:13:47.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2020-11-16T22:13:47.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local osd_data=td/osd-dup/0 2020-11-16T22:13:47.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local 'ceph_args=--fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2020-11-16T22:13:47.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:13:47.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:13:47.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:13:47.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/0' 2020-11-16T22:13:47.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2020-11-16T22:13:47.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --chdir=' 2020-11-16T22:13:47.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+= 2020-11-16T22:13:47.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2020-11-16T22:13:47.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: get_asok_path 2020-11-16T22:13:47.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:47.981 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:47.981 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:47.982 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:47.982 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:13:47.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.650871/$cluster-$name.asok' 2020-11-16T22:13:47.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' 2020-11-16T22:13:47.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-osd=20' 2020-11-16T22:13:47.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-ms=1' 2020-11-16T22:13:47.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-monc=20' 2020-11-16T22:13:47.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2020-11-16T22:13:47.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2020-11-16T22:13:47.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:13:47.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:13:47.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:47.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' ' 2020-11-16T22:13:47.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+= 2020-11-16T22:13:47.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: mkdir -p td/osd-dup/0 2020-11-16T22:13:47.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: uuidgen 2020-11-16T22:13:47.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: local uuid=b21a904e-88b4-47c7-874c-4491821f1c9d 2020-11-16T22:13:47.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo 'add osd b21a904e-88b4-47c7-874c-4491821f1c9d' 2020-11-16T22:13:47.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph-authtool --gen-print-key 2020-11-16T22:13:47.985 INFO:tasks.workunit.client.0.smithi189.stdout:add osd b21a904e-88b4-47c7-874c-4491821f1c9d 2020-11-16T22:13:48.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: OSD_SECRET=AQCc+bJf6R68ABAAPIOMe39SmpWEytkfaR7F7Q== 2020-11-16T22:13:48.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo '{"cephx_secret": "AQCc+bJf6R68ABAAPIOMe39SmpWEytkfaR7F7Q=="}' 2020-11-16T22:13:48.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph osd new b21a904e-88b4-47c7-874c-4491821f1c9d -i td/osd-dup/0/new.json 2020-11-16T22:13:48.370 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:13:48.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: rm td/osd-dup/0/new.json 2020-11-16T22:13:48.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph-osd -i 0 --fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCc+bJf6R68ABAAPIOMe39SmpWEytkfaR7F7Q== --osd-uuid b21a904e-88b4-47c7-874c-4491821f1c9d --osd-objectstore=filestore 2020-11-16T22:13:48.412 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:48.417+0000 7fad5f377d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:48.414 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:48.421+0000 7fad5f377d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:48.415 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:48.421+0000 7fad5f377d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:48.891 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:48.897+0000 7fad5f377d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-16T22:13:49.345 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:49.353+0000 7fad5f377d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-16T22:13:49.346 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:49.353+0000 7fad5f377d80 -1 journal do_read_entry(4096): bad header magic 2020-11-16T22:13:49.346 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:49.353+0000 7fad5f377d80 -1 journal do_read_entry(4096): bad header magic 2020-11-16T22:13:49.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: local key_fn=td/osd-dup/0/keyring 2020-11-16T22:13:49.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: cat 2020-11-16T22:13:49.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo adding osd0 key to auth repository 2020-11-16T22:13:49.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph -i td/osd-dup/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:13:49.796 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:13:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2020-11-16T22:13:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo start osd.0 2020-11-16T22:13:50.002 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:13:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph osd dump --format=json 2020-11-16T22:13:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph-osd -i 0 --fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: grep -q '"noup"' 2020-11-16T22:13:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: jq '.flags_set[]' 2020-11-16T22:13:50.029 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:50.037+0000 7f4ce21f4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:50.033 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:50.041+0000 7f4ce21f4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:50.041+0000 7f4ce21f4d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:50.045+0000 7f4ce21f4d80 -1 Falling back to public interface 2020-11-16T22:13:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: wait_for_osd up 0 2020-11-16T22:13:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:13:50.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:13:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:13:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:13:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:13:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:50.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:13:50.342 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:13:50.375 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:50.381+0000 7f4ce21f4d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-16T22:13:50.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:50.389+0000 7f4ce21f4d80 -1 journal do_read_entry(8192): bad header magic 2020-11-16T22:13:50.381 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:50.389+0000 7f4ce21f4d80 -1 journal do_read_entry(8192): bad header magic 2020-11-16T22:13:50.388 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:50.397+0000 7f4ce21f4d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:13:50.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:13:51.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:13:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:13:51.624 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:13:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:13:51.897 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/651189,v1:127.0.0.1:6801/651189] [v2:127.0.0.1:6802/651189,v1:127.0.0.1:6803/651189] exists,up b21a904e-88b4-47c7-874c-4491821f1c9d 2020-11-16T22:13:51.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:13:51.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:13:51.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:13:51.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: cat td/osd-dup/osd.0.pid 2020-11-16T22:13:51.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: osd_pid=651189 2020-11-16T22:13:51.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:38: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 1 2020-11-16T22:13:51.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local dir=td/osd-dup 2020-11-16T22:13:51.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2020-11-16T22:13:51.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local id=1 2020-11-16T22:13:51.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2020-11-16T22:13:51.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local osd_data=td/osd-dup/1 2020-11-16T22:13:51.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local 'ceph_args=--fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2020-11-16T22:13:51.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:13:51.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:13:51.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:13:51.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/1' 2020-11-16T22:13:51.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/1/journal' 2020-11-16T22:13:51.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --chdir=' 2020-11-16T22:13:51.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+= 2020-11-16T22:13:51.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2020-11-16T22:13:51.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: get_asok_path 2020-11-16T22:13:51.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:51.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:51.903 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:51.903 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:51.903 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:13:51.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.650871/$cluster-$name.asok' 2020-11-16T22:13:51.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' 2020-11-16T22:13:51.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-osd=20' 2020-11-16T22:13:51.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-ms=1' 2020-11-16T22:13:51.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-monc=20' 2020-11-16T22:13:51.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2020-11-16T22:13:51.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2020-11-16T22:13:51.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:13:51.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:13:51.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:51.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' ' 2020-11-16T22:13:51.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+= 2020-11-16T22:13:51.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: mkdir -p td/osd-dup/1 2020-11-16T22:13:51.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: uuidgen 2020-11-16T22:13:51.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: local uuid=6fd29aef-b604-438a-bd56-e5e93dcc897c 2020-11-16T22:13:51.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo 'add osd 6fd29aef-b604-438a-bd56-e5e93dcc897c' 2020-11-16T22:13:51.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph-authtool --gen-print-key 2020-11-16T22:13:51.907 INFO:tasks.workunit.client.0.smithi189.stdout:add osd 6fd29aef-b604-438a-bd56-e5e93dcc897c 2020-11-16T22:13:51.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: OSD_SECRET=AQCf+bJfFkGtNxAAgY1/lh2j+oNs0o0ImWDEBA== 2020-11-16T22:13:51.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo '{"cephx_secret": "AQCf+bJfFkGtNxAAgY1/lh2j+oNs0o0ImWDEBA=="}' 2020-11-16T22:13:51.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph osd new 6fd29aef-b604-438a-bd56-e5e93dcc897c -i td/osd-dup/1/new.json 2020-11-16T22:13:52.410 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:13:52.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: rm td/osd-dup/1/new.json 2020-11-16T22:13:52.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph-osd -i 1 --fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCf+bJfFkGtNxAAgY1/lh2j+oNs0o0ImWDEBA== --osd-uuid 6fd29aef-b604-438a-bd56-e5e93dcc897c --osd-objectstore=filestore 2020-11-16T22:13:52.460 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:52.469+0000 7fd07383dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:52.463 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:52.469+0000 7fd07383dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:52.464 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:52.473+0000 7fd07383dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:52.797 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:52.805+0000 7fd07383dd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-16T22:13:53.210 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:53.217+0000 7fd07383dd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-16T22:13:53.210 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:53.217+0000 7fd07383dd80 -1 journal do_read_entry(4096): bad header magic 2020-11-16T22:13:53.210 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:53.217+0000 7fd07383dd80 -1 journal do_read_entry(4096): bad header magic 2020-11-16T22:13:53.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: local key_fn=td/osd-dup/1/keyring 2020-11-16T22:13:53.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: cat 2020-11-16T22:13:53.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo adding osd1 key to auth repository 2020-11-16T22:13:53.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph -i td/osd-dup/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:13:53.776 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:13:54.097 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2020-11-16T22:13:54.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo start osd.1 2020-11-16T22:13:54.100 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:13:54.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph-osd -i 1 --fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:54.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph osd dump --format=json 2020-11-16T22:13:54.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: jq '.flags_set[]' 2020-11-16T22:13:54.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: grep -q '"noup"' 2020-11-16T22:13:54.127 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:54.133+0000 7f092501dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:54.129 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:54.137+0000 7f092501dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:54.131 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:54.137+0000 7f092501dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:54.133 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:54.141+0000 7f092501dd80 -1 Falling back to public interface 2020-11-16T22:13:54.499 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:54.505+0000 7f092501dd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-16T22:13:54.499 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:54.505+0000 7f092501dd80 -1 journal do_read_entry(8192): bad header magic 2020-11-16T22:13:54.499 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:54.505+0000 7f092501dd80 -1 journal do_read_entry(8192): bad header magic 2020-11-16T22:13:54.504 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:54.513+0000 7f092501dd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:13:54.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: wait_for_osd up 1 2020-11-16T22:13:54.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:13:54.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:13:54.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:13:54.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:13:54.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:54.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:13:54.623 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:13:54.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:54.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:13:55.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:13:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:13:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:13:56.030 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:13:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:13:56.419 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/651545,v1:127.0.0.1:6811/651545] [v2:127.0.0.1:6812/651545,v1:127.0.0.1:6813/651545] exists,up 6fd29aef-b604-438a-bd56-e5e93dcc897c 2020-11-16T22:13:56.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:13:56.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:13:56.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:13:56.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:39: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 2 2020-11-16T22:13:56.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local dir=td/osd-dup 2020-11-16T22:13:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2020-11-16T22:13:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local id=2 2020-11-16T22:13:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2020-11-16T22:13:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local osd_data=td/osd-dup/2 2020-11-16T22:13:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local 'ceph_args=--fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2020-11-16T22:13:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:13:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:13:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:13:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/2' 2020-11-16T22:13:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/2/journal' 2020-11-16T22:13:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --chdir=' 2020-11-16T22:13:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+= 2020-11-16T22:13:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2020-11-16T22:13:56.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: get_asok_path 2020-11-16T22:13:56.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:13:56.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:13:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:13:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:13:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:13:56.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.650871/$cluster-$name.asok' 2020-11-16T22:13:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' 2020-11-16T22:13:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-osd=20' 2020-11-16T22:13:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-ms=1' 2020-11-16T22:13:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-monc=20' 2020-11-16T22:13:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2020-11-16T22:13:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2020-11-16T22:13:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:13:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:13:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' ' 2020-11-16T22:13:56.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+= 2020-11-16T22:13:56.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: mkdir -p td/osd-dup/2 2020-11-16T22:13:56.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: uuidgen 2020-11-16T22:13:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: local uuid=6f5e8d3f-8dbb-494a-8702-1e1ae8146d97 2020-11-16T22:13:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo 'add osd 6f5e8d3f-8dbb-494a-8702-1e1ae8146d97' 2020-11-16T22:13:56.430 INFO:tasks.workunit.client.0.smithi189.stdout:add osd 6f5e8d3f-8dbb-494a-8702-1e1ae8146d97 2020-11-16T22:13:56.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph-authtool --gen-print-key 2020-11-16T22:13:56.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: OSD_SECRET=AQCk+bJf26BzGxAArIHPZildCvGJrUsiYzPuyg== 2020-11-16T22:13:56.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo '{"cephx_secret": "AQCk+bJf26BzGxAArIHPZildCvGJrUsiYzPuyg=="}' 2020-11-16T22:13:56.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph osd new 6f5e8d3f-8dbb-494a-8702-1e1ae8146d97 -i td/osd-dup/2/new.json 2020-11-16T22:13:56.934 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:13:56.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: rm td/osd-dup/2/new.json 2020-11-16T22:13:56.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph-osd -i 2 --fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCk+bJf26BzGxAArIHPZildCvGJrUsiYzPuyg== --osd-uuid 6f5e8d3f-8dbb-494a-8702-1e1ae8146d97 --osd-objectstore=filestore 2020-11-16T22:13:56.980 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:56.985+0000 7f95be267d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:56.982 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:56.989+0000 7f95be267d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:56.983 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:56.989+0000 7f95be267d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:57.439 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:57.445+0000 7f95be267d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-16T22:13:57.927 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:57.933+0000 7f95be267d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-16T22:13:57.940 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:57.949+0000 7f95be267d80 -1 journal do_read_entry(4096): bad header magic 2020-11-16T22:13:57.940 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:57.949+0000 7f95be267d80 -1 journal do_read_entry(4096): bad header magic 2020-11-16T22:13:58.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: local key_fn=td/osd-dup/2/keyring 2020-11-16T22:13:58.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: cat 2020-11-16T22:13:58.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo adding osd2 key to auth repository 2020-11-16T22:13:58.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph -i td/osd-dup/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:13:58.484 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T22:13:58.832 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2020-11-16T22:13:58.835 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:13:58.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo start osd.2 2020-11-16T22:13:58.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph osd dump --format=json 2020-11-16T22:13:58.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: jq '.flags_set[]' 2020-11-16T22:13:58.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph-osd -i 2 --fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:13:58.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: grep -q '"noup"' 2020-11-16T22:13:58.862 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:58.869+0000 7f0244085d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:58.865 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:58.873+0000 7f0244085d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:58.866 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:58.873+0000 7f0244085d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:13:58.868 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:58.877+0000 7f0244085d80 -1 Falling back to public interface 2020-11-16T22:13:59.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: wait_for_osd up 2 2020-11-16T22:13:59.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:13:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:13:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:13:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:13:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:13:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:13:59.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:13:59.229 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:13:59.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:13:59.234 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:59.241+0000 7f0244085d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-16T22:13:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:59.245+0000 7f0244085d80 -1 journal do_read_entry(8192): bad header magic 2020-11-16T22:13:59.238 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:59.245+0000 7f0244085d80 -1 journal do_read_entry(8192): bad header magic 2020-11-16T22:13:59.243 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:13:59.249+0000 7f0244085d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T22:13:59.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:14:00.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:14:00.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:14:00.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:14:00.821 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:14:00.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:14:00.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:14:01.198 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/651838,v1:127.0.0.1:6819/651838] [v2:127.0.0.1:6820/651838,v1:127.0.0.1:6821/651838] exists,up 6f5e8d3f-8dbb-494a-8702-1e1ae8146d97 2020-11-16T22:14:01.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:14:01.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:14:01.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:14:01.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:41: TEST_filestore_to_bluestore: sleep 5 2020-11-16T22:14:06.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:43: TEST_filestore_to_bluestore: create_pool foo 16 2020-11-16T22:14:06.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create foo 16 2020-11-16T22:14:06.957 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' already exists 2020-11-16T22:14:06.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:14:07.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:46: TEST_filestore_to_bluestore: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2020-11-16T22:14:08.009 INFO:tasks.workunit.client.0.smithi189.stdout:hints = 1 2020-11-16T22:14:08.010 INFO:tasks.workunit.client.0.smithi189.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2020-11-16T22:14:08.010 INFO:tasks.workunit.client.0.smithi189.stdout:Object prefix: benchmark_data_smithi189_652024 2020-11-16T22:14:08.010 INFO:tasks.workunit.client.0.smithi189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T22:14:08.010 INFO:tasks.workunit.client.0.smithi189.stdout: 0 0 0 0 0 0 - 0 2020-11-16T22:14:09.010 INFO:tasks.workunit.client.0.smithi189.stdout: 1 16 16 0 0 0 - 0 2020-11-16T22:14:10.010 INFO:tasks.workunit.client.0.smithi189.stdout: 2 16 16 0 0 0 - 0 2020-11-16T22:14:11.010 INFO:tasks.workunit.client.0.smithi189.stdout: 3 16 17 1 0.00130195 0.00130208 2.4272 2.4272 2020-11-16T22:14:12.010 INFO:tasks.workunit.client.0.smithi189.stdout: 4 16 19 3 0.00292937 0.0078125 3.12686 2.86379 2020-11-16T22:14:13.010 INFO:tasks.workunit.client.0.smithi189.stdout: 5 16 19 3 0.00234349 0 - 2.86379 2020-11-16T22:14:14.010 INFO:tasks.workunit.client.0.smithi189.stdout: 6 16 115 99 0.0644458 0.1875 0.121125 0.960009 2020-11-16T22:14:15.010 INFO:tasks.workunit.client.0.smithi189.stdout: 7 16 227 211 0.117733 0.4375 0.121743 0.517423 2020-11-16T22:14:16.011 INFO:tasks.workunit.client.0.smithi189.stdout: 8 16 307 291 0.142074 0.3125 0.28243 0.436357 2020-11-16T22:14:17.011 INFO:tasks.workunit.client.0.smithi189.stdout: 9 16 387 371 0.161006 0.3125 0.189842 0.386877 2020-11-16T22:14:18.011 INFO:tasks.workunit.client.0.smithi189.stdout: 10 16 516 500 0.19529 0.503906 0.1202 0.316836 2020-11-16T22:14:19.011 INFO:tasks.workunit.client.0.smithi189.stdout:Total time run: 10.0619 2020-11-16T22:14:19.012 INFO:tasks.workunit.client.0.smithi189.stdout:Total writes made: 516 2020-11-16T22:14:19.012 INFO:tasks.workunit.client.0.smithi189.stdout:Write size: 4096 2020-11-16T22:14:19.012 INFO:tasks.workunit.client.0.smithi189.stdout:Object size: 4096 2020-11-16T22:14:19.012 INFO:tasks.workunit.client.0.smithi189.stdout:Bandwidth (MB/sec): 0.200322 2020-11-16T22:14:19.012 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Bandwidth: 0.201394 2020-11-16T22:14:19.012 INFO:tasks.workunit.client.0.smithi189.stdout:Max bandwidth (MB/sec): 0.503906 2020-11-16T22:14:19.013 INFO:tasks.workunit.client.0.smithi189.stdout:Min bandwidth (MB/sec): 0 2020-11-16T22:14:19.013 INFO:tasks.workunit.client.0.smithi189.stdout:Average IOPS: 51 2020-11-16T22:14:19.013 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev IOPS: 51.5891 2020-11-16T22:14:19.013 INFO:tasks.workunit.client.0.smithi189.stdout:Max IOPS: 129 2020-11-16T22:14:19.013 INFO:tasks.workunit.client.0.smithi189.stdout:Min IOPS: 0 2020-11-16T22:14:19.013 INFO:tasks.workunit.client.0.smithi189.stdout:Average Latency(s): 0.310516 2020-11-16T22:14:19.013 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Latency(s): 0.850447 2020-11-16T22:14:19.014 INFO:tasks.workunit.client.0.smithi189.stdout:Max latency(s): 5.30779 2020-11-16T22:14:19.014 INFO:tasks.workunit.client.0.smithi189.stdout:Min latency(s): 0.0930961 2020-11-16T22:14:19.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 651189 2020-11-16T22:14:19.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: sleep 1 2020-11-16T22:14:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 651189 2020-11-16T22:14:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh: line 49: kill: (651189) - No such process 2020-11-16T22:14:20.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:50: TEST_filestore_to_bluestore: ceph osd down 0 2020-11-16T22:14:20.617 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already down. 2020-11-16T22:14:20.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:52: TEST_filestore_to_bluestore: mv td/osd-dup/0 td/osd-dup/0.old 2020-11-16T22:14:20.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:53: TEST_filestore_to_bluestore: mkdir td/osd-dup/0 2020-11-16T22:14:20.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: cat td/osd-dup/0.old/fsid 2020-11-16T22:14:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: ofsid=b21a904e-88b4-47c7-874c-4491821f1c9d 2020-11-16T22:14:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:55: TEST_filestore_to_bluestore: echo 'osd fsid b21a904e-88b4-47c7-874c-4491821f1c9d' 2020-11-16T22:14:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:56: TEST_filestore_to_bluestore: O='--fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2020-11-16T22:14:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:57: TEST_filestore_to_bluestore: CEPH_ARGS+='--log-file td/osd-dup/cot.log --log-max-recent 0 ' 2020-11-16T22:14:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:58: TEST_filestore_to_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-dup/0 --fsid b21a904e-88b4-47c7-874c-4491821f1c9d --op mkfs --no-mon-config 2020-11-16T22:14:20.640 INFO:tasks.workunit.client.0.smithi189.stdout:osd fsid b21a904e-88b4-47c7-874c-4491821f1c9d 2020-11-16T22:14:23.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:60: TEST_filestore_to_bluestore: ceph-objectstore-tool --data-path td/osd-dup/0.old --target-data-path td/osd-dup/0 --op dup 2020-11-16T22:14:23.934 INFO:tasks.workunit.client.0.smithi189.stdout:dup from filestore: td/osd-dup/0.old 2020-11-16T22:14:23.934 INFO:tasks.workunit.client.0.smithi189.stdout: to bluestore: td/osd-dup/0 2020-11-16T22:14:24.822 INFO:tasks.workunit.client.0.smithi189.stdout:fsid b21a904e-88b4-47c7-874c-4491821f1c9d 2020-11-16T22:14:24.823 INFO:tasks.workunit.client.0.smithi189.stdout:17 collections 2020-11-16T22:14:24.823 INFO:tasks.workunit.client.0.smithi189.stdout:1/17 1.1_head 2020-11-16T22:14:25.129 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 34 objects, 135168 bytes, 39 keys 2020-11-16T22:14:25.130 INFO:tasks.workunit.client.0.smithi189.stdout:2/17 1.b_head 2020-11-16T22:14:25.339 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 27 objects, 106496 bytes, 32 keys 2020-11-16T22:14:25.339 INFO:tasks.workunit.client.0.smithi189.stdout:3/17 1.5_head 2020-11-16T22:14:25.548 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 38 objects, 151552 bytes, 43 keys 2020-11-16T22:14:25.549 INFO:tasks.workunit.client.0.smithi189.stdout:4/17 1.8_head 2020-11-16T22:14:25.624 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 110592 bytes, 33 keys 2020-11-16T22:14:25.625 INFO:tasks.workunit.client.0.smithi189.stdout:5/17 1.3_head 2020-11-16T22:14:25.717 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 31 objects, 122880 bytes, 36 keys 2020-11-16T22:14:25.717 INFO:tasks.workunit.client.0.smithi189.stdout:6/17 1.0_head 2020-11-16T22:14:25.927 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 47 objects, 188416 bytes, 52 keys 2020-11-16T22:14:25.927 INFO:tasks.workunit.client.0.smithi189.stdout:7/17 1.2_head 2020-11-16T22:14:25.931 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 110592 bytes, 33 keys 2020-11-16T22:14:25.931 INFO:tasks.workunit.client.0.smithi189.stdout:8/17 1.4_head 2020-11-16T22:14:26.032 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 38 objects, 151552 bytes, 43 keys 2020-11-16T22:14:26.032 INFO:tasks.workunit.client.0.smithi189.stdout:9/17 1.e_head 2020-11-16T22:14:26.272 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 38 objects, 151552 bytes, 43 keys 2020-11-16T22:14:26.273 INFO:tasks.workunit.client.0.smithi189.stdout:10/17 1.f_head 2020-11-16T22:14:26.423 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 27 objects, 106496 bytes, 32 keys 2020-11-16T22:14:26.424 INFO:tasks.workunit.client.0.smithi189.stdout:11/17 meta 2020-11-16T22:14:26.428 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 46 objects, 47104 bytes, 0 keys 2020-11-16T22:14:26.428 INFO:tasks.workunit.client.0.smithi189.stdout:12/17 1.a_head 2020-11-16T22:14:26.575 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 43 objects, 172032 bytes, 48 keys 2020-11-16T22:14:26.575 INFO:tasks.workunit.client.0.smithi189.stdout:13/17 1.7_head 2020-11-16T22:14:26.668 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 110592 bytes, 33 keys 2020-11-16T22:14:26.669 INFO:tasks.workunit.client.0.smithi189.stdout:14/17 1.c_head 2020-11-16T22:14:26.761 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 29 objects, 114688 bytes, 34 keys 2020-11-16T22:14:26.762 INFO:tasks.workunit.client.0.smithi189.stdout:15/17 1.d_head 2020-11-16T22:14:26.850 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 106520 bytes, 33 keys 2020-11-16T22:14:26.850 INFO:tasks.workunit.client.0.smithi189.stdout:16/17 1.9_head 2020-11-16T22:14:26.914 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 33 objects, 131072 bytes, 38 keys 2020-11-16T22:14:26.915 INFO:tasks.workunit.client.0.smithi189.stdout:17/17 1.6_head 2020-11-16T22:14:27.073 INFO:tasks.workunit.client.0.smithi189.stdout: 0 objects, 0 bytes, 0 keys 36 objects, 143360 bytes, 41 keys 2020-11-16T22:14:27.073 INFO:tasks.workunit.client.0.smithi189.stdout:keyring 2020-11-16T22:14:27.073 INFO:tasks.workunit.client.0.smithi189.stdout:duping osd metadata 2020-11-16T22:14:27.457 INFO:tasks.workunit.client.0.smithi189.stdout:done. 2020-11-16T22:14:28.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:62: TEST_filestore_to_bluestore: CEPH_ARGS='--fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2020-11-16T22:14:28.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:64: TEST_filestore_to_bluestore: activate_osd td/osd-dup 0 2020-11-16T22:14:28.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-dup 2020-11-16T22:14:28.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:14:28.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T22:14:28.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:14:28.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-dup/0 2020-11-16T22:14:28.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2020-11-16T22:14:28.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:14:28.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:14:28.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:14:28.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-dup/0' 2020-11-16T22:14:28.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2020-11-16T22:14:28.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:14:28.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:14:28.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-dup' 2020-11-16T22:14:28.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:14:28.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:14:28.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:14:28.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:14:28.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:28.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:14:28.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.650871/$cluster-$name.asok' 2020-11-16T22:14:28.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' 2020-11-16T22:14:28.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:14:28.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-dup/$name.log' 2020-11-16T22:14:28.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2020-11-16T22:14:28.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:14:28.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:14:28.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:14:28.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:14:28.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:14:28.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-dup/0 2020-11-16T22:14:28.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T22:14:28.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=650bee77-fc19-4721-9659-72908c7c6b7b --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.650871/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:14:28.306 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:14:28.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-dup/0/whoami 2020-11-16T22:14:28.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T22:14:28.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:14:28.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:14:28.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:14:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:28.337+0000 7f6f6c850d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:14:28.333 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:28.341+0000 7f6f6c850d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:14:28.335 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:28.341+0000 7f6f6c850d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:14:28.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T22:14:28.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:14:28.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:14:28.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:14:28.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:14:28.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:14:28.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:14:28.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:14:28.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:14:28.703 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:14:28.877 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:28.885+0000 7f6f6c850d80 -1 Falling back to public interface 2020-11-16T22:14:29.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:14:30.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:14:30.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:14:30.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:14:30.112 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:14:30.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:14:30.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:14:30.298 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:30.305+0000 7f6f6c850d80 -1 osd.0 21 log_to_monitors {default=true} 2020-11-16T22:14:30.503 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 25 up_thru 17 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6800/652667,v1:127.0.0.1:6801/652667] [v2:127.0.0.1:6802/652667,v1:127.0.0.1:6803/652667] exists,up b21a904e-88b4-47c7-874c-4491821f1c9d 2020-11-16T22:14:30.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:14:30.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:14:30.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:14:30.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: ceph osd stat 2020-11-16T22:14:30.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: grep '3 up' 2020-11-16T22:14:30.997 INFO:tasks.workunit.client.0.smithi189.stdout:3 osds: 3 up (since 0.594714s), 3 in (since 30s); epoch: e26 2020-11-16T22:14:30.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: ceph osd metadata 0 2020-11-16T22:14:30.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: grep bluestore 2020-11-16T22:14:31.473 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_access_mode": "file", 2020-11-16T22:14:31.474 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_block_size": "4096", 2020-11-16T22:14:31.474 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_devices": "sda", 2020-11-16T22:14:31.474 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-11-16T22:14:31.474 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_path": "td/osd-dup/0/block", 2020-11-16T22:14:31.474 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_rotational": "1", 2020-11-16T22:14:31.475 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_size": "107374182400", 2020-11-16T22:14:31.475 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_support_discard": "0", 2020-11-16T22:14:31.475 INFO:tasks.workunit.client.0.smithi189.stdout: "bluestore_bdev_type": "hdd", 2020-11-16T22:14:31.475 INFO:tasks.workunit.client.0.smithi189.stdout: "osd_objectstore": "bluestore", 2020-11-16T22:14:31.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:69: TEST_filestore_to_bluestore: ceph osd scrub 0 2020-11-16T22:14:31.868 INFO:tasks.workunit.client.0.smithi189.stderr:instructed osd(s) 0 to scrub 2020-11-16T22:14:31.876 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:72: TEST_filestore_to_bluestore: sleep 15 2020-11-16T22:14:46.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:74: TEST_filestore_to_bluestore: flush_pg_stats 2020-11-16T22:14:46.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:14:46.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:14:47.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:14:47.275 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:14:47.275 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T22:14:47.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:14:47.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:14:47.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:14:47.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182406 2020-11-16T22:14:47.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182406 2020-11-16T22:14:47.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182406' 2020-11-16T22:14:47.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:14:47.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:14:47.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672973 2020-11-16T22:14:47.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672973 2020-11-16T22:14:47.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182406 1-42949672973' 2020-11-16T22:14:47.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:14:47.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:14:47.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2020-11-16T22:14:47.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2020-11-16T22:14:47.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182406 1-42949672973 2-64424509452' 2020-11-16T22:14:47.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:14:47.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-107374182406 2020-11-16T22:14:47.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:14:47.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:14:47.778 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-107374182406 2020-11-16T22:14:47.779 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:14:47.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182406 2020-11-16T22:14:47.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2020-11-16T22:14:47.781 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 107374182406 2020-11-16T22:14:47.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:14:48.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182406 -lt 107374182406 2020-11-16T22:14:48.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:14:48.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672973 2020-11-16T22:14:48.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:14:48.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:14:48.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672973 2020-11-16T22:14:48.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:14:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672973 2020-11-16T22:14:48.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2020-11-16T22:14:48.180 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672973 2020-11-16T22:14:48.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:14:48.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672973 -lt 42949672973 2020-11-16T22:14:48.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:14:48.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2020-11-16T22:14:48.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:14:48.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:14:48.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:14:48.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2020-11-16T22:14:48.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2020-11-16T22:14:48.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2020-11-16T22:14:48.568 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509452 2020-11-16T22:14:48.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:14:48.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509450 -lt 64424509452 2020-11-16T22:14:48.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:14:49.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:14:49.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:14:50.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509452 -lt 64424509452 2020-11-16T22:14:50.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:76: TEST_filestore_to_bluestore: wait_for_clean 2020-11-16T22:14:50.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:14:50.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:14:50.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:14:50.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:14:50.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:14:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:14:50.339 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:14:50.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:14:50.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:14:50.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:14:50.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:14:50.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:14:50.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:14:50.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:14:50.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:14:50.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:14:50.920 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:14:50.920 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T22:14:50.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:14:50.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:14:50.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:14:51.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182409 2020-11-16T22:14:51.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182409 2020-11-16T22:14:51.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182409' 2020-11-16T22:14:51.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:14:51.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:14:51.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672976 2020-11-16T22:14:51.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672976 2020-11-16T22:14:51.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182409 1-42949672976' 2020-11-16T22:14:51.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:14:51.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:14:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509455 2020-11-16T22:14:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509455 2020-11-16T22:14:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182409 1-42949672976 2-64424509455' 2020-11-16T22:14:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:14:51.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-107374182409 2020-11-16T22:14:51.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:14:51.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:14:51.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-107374182409 2020-11-16T22:14:51.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:14:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182409 2020-11-16T22:14:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 107374182409' 2020-11-16T22:14:51.404 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 107374182409 2020-11-16T22:14:51.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:14:51.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182409 -lt 107374182409 2020-11-16T22:14:51.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:14:51.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672976 2020-11-16T22:14:51.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:14:51.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:14:51.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672976 2020-11-16T22:14:51.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:14:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672976 2020-11-16T22:14:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2020-11-16T22:14:51.813 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672976 2020-11-16T22:14:51.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:14:52.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672976 -lt 42949672976 2020-11-16T22:14:52.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:14:52.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509455 2020-11-16T22:14:52.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:14:52.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:14:52.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509455 2020-11-16T22:14:52.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:14:52.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509455 2020-11-16T22:14:52.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2020-11-16T22:14:52.214 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509455 2020-11-16T22:14:52.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:14:52.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509455 -lt 64424509455 2020-11-16T22:14:52.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:14:52.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:14:52.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:14:53.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 16 == 0 2020-11-16T22:14:53.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:14:53.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:14:53.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:14:53.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:14:53.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:14:53.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:14:53.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:14:53.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=16 2020-11-16T22:14:53.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:14:53.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:14:53.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:14:53.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 16 = 16 2020-11-16T22:14:53.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:14:53.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:14:53.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:22: run: teardown td/osd-dup 2020-11-16T22:14:53.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2020-11-16T22:14:53.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:14:53.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2020-11-16T22:14:53.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:14:53.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:14:53.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:14:53.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:14:53.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:14:54.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:14:54.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:14:54.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:14:54.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:14:54.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:14:54.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:14:54.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:14:54.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:14:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:14:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:14:54.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:14:54.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:14:54.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:14:54.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2020-11-16T22:14:54.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:14:54.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:54.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:14:54.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.650871 2020-11-16T22:14:54.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:14:54.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:14:54.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T22:14:54.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-dup 0 2020-11-16T22:14:54.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2020-11-16T22:14:54.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T22:14:54.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2020-11-16T22:14:54.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:14:54.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:14:54.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:14:54.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:14:54.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:14:54.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:14:54.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:14:54.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:14:54.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:14:54.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:14:54.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:14:54.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:14:54.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:14:54.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:14:54.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:14:54.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:14:54.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:14:54.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T22:14:54.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2020-11-16T22:14:54.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:14:54.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:54.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.650871 2020-11-16T22:14:54.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.650871 2020-11-16T22:14:54.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:14:54.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:14:54.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T22:14:54.165 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T22:14:54.166 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T22:14:54.221 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2020-11-16T22:14:54.221 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2020-11-16T22:14:54.281 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T22:14:54.293 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T22:14:54.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:14:54.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:14:54.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T22:14:54.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T22:14:54.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T22:14:54.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T22:14:54.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T22:14:54.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T22:14:54.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-fast-mark-down 2020-11-16T22:14:54.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2020-11-16T22:14:54.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2020-11-16T22:14:54.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2020-11-16T22:14:54.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2020-11-16T22:14:54.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2020-11-16T22:14:54.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2020-11-16T22:14:54.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2020-11-16T22:14:54.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none ' 2020-11-16T22:14:54.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none ' 2020-11-16T22:14:54.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2020-11-16T22:14:54.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2020-11-16T22:14:54.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2020-11-16T22:14:54.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2020-11-16T22:14:54.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2020-11-16T22:14:54.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2020-11-16T22:14:54.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2020-11-16T22:14:54.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:14:54.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2020-11-16T22:14:54.298 INFO:tasks.workunit.client.0.smithi189.stdout:Ensuring old behavior is there... 2020-11-16T22:14:54.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:14:54.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:14:54.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:14:54.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:14:54.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:14:54.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:14:54.301 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:14:54.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:14:54.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:14:54.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:14:54.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:14:54.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:14:54.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:14:54.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:14:54.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:14:54.307 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:14:54.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:14:54.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:14:54.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2020-11-16T22:14:54.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:14:54.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:54.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:14:54.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.653760 2020-11-16T22:14:54.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:14:54.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:14:54.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2020-11-16T22:14:54.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:14:54.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:54.317 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:14:54.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.653760 2020-11-16T22:14:54.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:14:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T22:14:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T22:14:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:14:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2020-11-16T22:14:54.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2020-11-16T22:14:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2020-11-16T22:14:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:14:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:14:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:14:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2020-11-16T22:14:54.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2020-11-16T22:14:54.346 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:54.352+0000 7f82909c2540 -1 failed for service _ceph-mon._tcp 2020-11-16T22:14:54.346 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:54.352+0000 7f82909c2540 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2020-11-16T22:14:54.346 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T22:14:54.347 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:54.352+0000 7f82909c2540 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2020-11-16T22:14:54.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:14:54.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:14:54.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:14:54.637 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:14:54.637 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:54.637 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:14:54.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.653760/$cluster-$name.asok' 2020-11-16T22:14:54.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_pool_default_size=3 2020-11-16T22:14:54.838 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:54.844+0000 7f0277c3c540 -1 failed for service _ceph-mon._tcp 2020-11-16T22:14:54.839 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:54.844+0000 7f0277c3c540 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2020-11-16T22:14:54.839 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T22:14:54.839 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:54.844+0000 7f0277c3c540 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2020-11-16T22:14:54.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:484: run_mon: return 1 2020-11-16T22:14:54.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2020-11-16T22:14:54.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2020-11-16T22:14:54.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2020-11-16T22:14:54.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2020-11-16T22:14:54.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2020-11-16T22:14:54.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2020-11-16T22:14:54.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2020-11-16T22:14:54.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2020-11-16T22:14:54.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2020-11-16T22:14:54.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2020-11-16T22:14:54.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:14:54.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2020-11-16T22:14:54.847 INFO:tasks.workunit.client.0.smithi189.stdout:Testing async msgr... 2020-11-16T22:14:54.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:14:54.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:14:54.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:14:54.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:14:54.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:14:54.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:14:54.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:14:54.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:14:54.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:14:54.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:14:54.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:14:54.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:14:54.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:14:54.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:14:54.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:14:54.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:14:54.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:14:54.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:14:54.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2020-11-16T22:14:54.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:14:54.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:54.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:14:54.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.653760 2020-11-16T22:14:54.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:14:54.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:14:54.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2020-11-16T22:14:54.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:14:54.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:54.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:14:54.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.653760 2020-11-16T22:14:54.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:14:54.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T22:14:54.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:14:54.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2020-11-16T22:14:54.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2020-11-16T22:14:54.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2020-11-16T22:14:54.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:14:54.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:14:54.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:14:54.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2020-11-16T22:14:54.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2020-11-16T22:14:55.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:14:55.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:14:55.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:14:55.172 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:14:55.172 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:55.172 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:14:55.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.653760/$cluster-$name.asok' 2020-11-16T22:14:55.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_pool_default_size=3 2020-11-16T22:14:55.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:14:55.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:14:55.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:14:55.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:14:55.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:14:55.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:14:55.378 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:14:55.379 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:14:55.379 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:14:55.379 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:14:55.380 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:55.380 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:14:55.380 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.653760/ceph-mon.a.asok 2020-11-16T22:14:55.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:14:55.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.653760/ceph-mon.a.asok config get fsid 2020-11-16T22:14:55.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:14:55.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:14:55.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:14:55.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:14:55.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:14:55.487 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:14:55.487 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:14:55.487 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:14:55.487 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:14:55.487 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:55.487 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:14:55.488 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.653760/ceph-mon.a.asok 2020-11-16T22:14:55.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:14:55.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.653760/ceph-mon.a.asok config get mon_host 2020-11-16T22:14:55.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2020-11-16T22:14:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-fast-mark-down 2020-11-16T22:14:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:14:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:14:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:14:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-fast-mark-down/x 2020-11-16T22:14:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:14:56.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:14:56.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:14:56.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:14:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:14:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:14:56.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.653760/$cluster-$name.asok' 2020-11-16T22:14:56.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:14:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:14:56.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2020-11-16T22:14:56.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2020-11-16T22:14:56.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-fast-mark-down 2020-11-16T22:14:56.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:14:56.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:14:56.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:14:56.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-fast-mark-down/0 2020-11-16T22:14:56.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2020-11-16T22:14:56.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:14:56.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:14:56.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:14:56.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2020-11-16T22:14:56.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2020-11-16T22:14:56.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:14:56.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:14:56.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2020-11-16T22:14:56.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:14:56.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:14:56.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:14:56.152 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:14:56.153 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:14:56.153 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:14:56.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.653760/$cluster-$name.asok' 2020-11-16T22:14:56.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' 2020-11-16T22:14:56.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:14:56.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:14:56.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:14:56.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2020-11-16T22:14:56.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2020-11-16T22:14:56.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:14:56.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:14:56.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:14:56.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:14:56.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:14:56.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-fast-mark-down/0 2020-11-16T22:14:56.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:14:56.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6bc10086-9d47-483b-9989-52f56b4afef5 2020-11-16T22:14:56.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 6bc10086-9d47-483b-9989-52f56b4afef5' 2020-11-16T22:14:56.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:14:56.156 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 6bc10086-9d47-483b-9989-52f56b4afef5 2020-11-16T22:14:56.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDg+bJfxPnsChAAlO6DyGa9ueA39JQeUjB74w== 2020-11-16T22:14:56.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDg+bJfxPnsChAAlO6DyGa9ueA39JQeUjB74w=="}' 2020-11-16T22:14:56.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6bc10086-9d47-483b-9989-52f56b4afef5 -i td/osd-fast-mark-down/0/new.json 2020-11-16T22:14:56.515 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:14:56.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-fast-mark-down/0/new.json 2020-11-16T22:14:56.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDg+bJfxPnsChAAlO6DyGa9ueA39JQeUjB74w== --osd-uuid 6bc10086-9d47-483b-9989-52f56b4afef5 2020-11-16T22:14:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:56.560+0000 7fb6d553bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:14:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:56.564+0000 7fb6d553bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:14:56.559 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:56.564+0000 7fb6d553bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:14:56.560 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:56.568+0000 7fb6d553bd80 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2020-11-16T22:14:56.560 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:56.568+0000 7fb6d553bd80 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2020-11-16T22:14:56.560 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:56.568+0000 7fb6d553bd80 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2020-11-16T22:14:56.613 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:14:56.620+0000 7fb6d553bd80 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2020-11-16T22:15:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2020-11-16T22:15:00.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:15:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:15:00.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:15:00.983 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:15:01.485 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:15:01.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:15:01.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:15:01.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:15:01.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:15:01.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:15:01.515 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:01.520+0000 7fca8d154d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:01.518 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:01.524+0000 7fca8d154d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:01.519 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:01.524+0000 7fca8d154d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:01.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:15:01.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:15:01.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:15:01.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:15:01.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:15:01.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:01.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:15:01.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:01.873 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:15:01.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:15:02.057 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:02.064+0000 7fca8d154d80 -1 Falling back to public interface 2020-11-16T22:15:02.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:15:02.946 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:02.944+0000 7fca8d154d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:15:03.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:15:03.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:03.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:15:03.240 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:15:03.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:03.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:15:03.657 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/654685,v1:127.0.0.1:6803/654685] [v2:127.0.0.1:6804/654685,v1:127.0.0.1:6805/654685] exists,up 6bc10086-9d47-483b-9989-52f56b4afef5 2020-11-16T22:15:03.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:15:03.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:15:03.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:15:03.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2020-11-16T22:15:03.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=654685 2020-11-16T22:15:03.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2020-11-16T22:15:03.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2020-11-16T22:15:03.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-fast-mark-down 2020-11-16T22:15:03.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:15:03.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:15:03.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:15:03.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-fast-mark-down/1 2020-11-16T22:15:03.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2020-11-16T22:15:03.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:15:03.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:15:03.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:15:03.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2020-11-16T22:15:03.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2020-11-16T22:15:03.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:15:03.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:15:03.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2020-11-16T22:15:03.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:15:03.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:15:03.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:15:03.664 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:15:03.664 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:03.664 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:15:03.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.653760/$cluster-$name.asok' 2020-11-16T22:15:03.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' 2020-11-16T22:15:03.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:15:03.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:15:03.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:15:03.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2020-11-16T22:15:03.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2020-11-16T22:15:03.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:15:03.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:15:03.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:15:03.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:15:03.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:15:03.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-fast-mark-down/1 2020-11-16T22:15:03.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:15:03.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0967752f-0fcf-4521-8470-bcb191fb696e 2020-11-16T22:15:03.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 0967752f-0fcf-4521-8470-bcb191fb696e' 2020-11-16T22:15:03.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:15:03.667 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 0967752f-0fcf-4521-8470-bcb191fb696e 2020-11-16T22:15:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDn+bJfXoOhKRAAyQqQVJkrs5SHsJSpIZ7Akw== 2020-11-16T22:15:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDn+bJfXoOhKRAAyQqQVJkrs5SHsJSpIZ7Akw=="}' 2020-11-16T22:15:03.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0967752f-0fcf-4521-8470-bcb191fb696e -i td/osd-fast-mark-down/1/new.json 2020-11-16T22:15:04.146 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:15:04.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-fast-mark-down/1/new.json 2020-11-16T22:15:04.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDn+bJfXoOhKRAAyQqQVJkrs5SHsJSpIZ7Akw== --osd-uuid 0967752f-0fcf-4521-8470-bcb191fb696e 2020-11-16T22:15:04.193 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:04.200+0000 7f686e831d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:04.197 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:04.204+0000 7f686e831d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:04.199 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:04.204+0000 7f686e831d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:04.200 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:04.208+0000 7f686e831d80 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2020-11-16T22:15:04.200 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:04.208+0000 7f686e831d80 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2020-11-16T22:15:04.200 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:04.208+0000 7f686e831d80 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2020-11-16T22:15:04.279 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:04.284+0000 7f686e831d80 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2020-11-16T22:15:08.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2020-11-16T22:15:08.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:15:08.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:15:08.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:15:08.645 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:15:09.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:15:09.127 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:15:09.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:15:09.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:15:09.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:15:09.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:15:09.156 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:09.164+0000 7fa7c2d04d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:09.158 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:09.164+0000 7fa7c2d04d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:09.159 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:09.164+0000 7fa7c2d04d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:09.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:15:09.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:15:09.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:15:09.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:15:09.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:15:09.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:09.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:15:09.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:09.514 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:15:09.514 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:15:09.702 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:09.708+0000 7fa7c2d04d80 -1 Falling back to public interface 2020-11-16T22:15:09.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:15:10.594 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:10.600+0000 7fa7c2d04d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:15:10.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:15:10.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:10.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:15:10.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:10.889 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:15:10.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:15:11.290 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/655680,v1:127.0.0.1:6811/655680] [v2:127.0.0.1:6812/655680,v1:127.0.0.1:6813/655680] exists,up 0967752f-0fcf-4521-8470-bcb191fb696e 2020-11-16T22:15:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:15:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:15:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:15:11.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2020-11-16T22:15:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=655680 2020-11-16T22:15:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2020-11-16T22:15:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2020-11-16T22:15:11.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-fast-mark-down 2020-11-16T22:15:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:15:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T22:15:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:15:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-fast-mark-down/2 2020-11-16T22:15:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2020-11-16T22:15:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:15:11.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:15:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:15:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2020-11-16T22:15:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2020-11-16T22:15:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:15:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:15:11.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2020-11-16T22:15:11.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:15:11.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:15:11.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:15:11.299 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:15:11.299 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:11.299 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:15:11.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.653760/$cluster-$name.asok' 2020-11-16T22:15:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' 2020-11-16T22:15:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:15:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:15:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:15:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2020-11-16T22:15:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2020-11-16T22:15:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:15:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:15:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:15:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:15:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:15:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-fast-mark-down/2 2020-11-16T22:15:11.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:15:11.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=11fc13fb-000d-4eb5-81e5-571ac58689b0 2020-11-16T22:15:11.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 11fc13fb-000d-4eb5-81e5-571ac58689b0' 2020-11-16T22:15:11.305 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 11fc13fb-000d-4eb5-81e5-571ac58689b0 2020-11-16T22:15:11.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:15:11.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDv+bJf3rgbFBAAvrBj/Q/g4ScdYGwebul6IA== 2020-11-16T22:15:11.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDv+bJf3rgbFBAAvrBj/Q/g4ScdYGwebul6IA=="}' 2020-11-16T22:15:11.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 11fc13fb-000d-4eb5-81e5-571ac58689b0 -i td/osd-fast-mark-down/2/new.json 2020-11-16T22:15:11.791 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:15:11.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-fast-mark-down/2/new.json 2020-11-16T22:15:11.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDv+bJf3rgbFBAAvrBj/Q/g4ScdYGwebul6IA== --osd-uuid 11fc13fb-000d-4eb5-81e5-571ac58689b0 2020-11-16T22:15:11.827 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:11.832+0000 7f72bd5b7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:11.830 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:11.836+0000 7f72bd5b7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:11.831 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:11.840+0000 7f72bd5b7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:11.832 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:11.840+0000 7f72bd5b7d80 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2020-11-16T22:15:11.833 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:11.840+0000 7f72bd5b7d80 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2020-11-16T22:15:11.833 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:11.840+0000 7f72bd5b7d80 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2020-11-16T22:15:11.884 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:11.892+0000 7f72bd5b7d80 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2020-11-16T22:15:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2020-11-16T22:15:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:15:16.211 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T22:15:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T22:15:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:15:16.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T22:15:16.718 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:15:16.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=6e9f5e93-226d-4b2d-a955-1e504e34c1cf --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.653760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:15:16.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:15:16.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:15:16.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:15:16.745 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:16.752+0000 7fbc2f441d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:16.748 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:16.756+0000 7fbc2f441d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:16.750 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:16.756+0000 7fbc2f441d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:17.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T22:15:17.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:15:17.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:15:17.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:15:17.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:15:17.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:17.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:15:17.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:15:17.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:17.110 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:15:17.294 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:17.300+0000 7fbc2f441d80 -1 Falling back to public interface 2020-11-16T22:15:17.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:15:18.158 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:18.164+0000 7fbc2f441d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T22:15:18.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:15:18.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:18.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:15:18.509 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:15:18.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:18.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:15:18.962 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/656675,v1:127.0.0.1:6819/656675] [v2:127.0.0.1:6820/656675,v1:127.0.0.1:6821/656675] exists,up 11fc13fb-000d-4eb5-81e5-571ac58689b0 2020-11-16T22:15:18.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:15:18.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:15:18.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:15:18.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2020-11-16T22:15:18.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=656675 2020-11-16T22:15:18.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2020-11-16T22:15:18.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2020-11-16T22:15:19.450 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'rbd' does not exist 2020-11-16T22:15:19.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2020-11-16T22:15:19.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-11-16T22:15:20.065 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'rbd' already exists 2020-11-16T22:15:20.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:15:21.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2020-11-16T22:15:22.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2020-11-16T22:15:22.864 INFO:tasks.workunit.client.0.smithi189.stdout:hints = 1 2020-11-16T22:15:22.864 INFO:tasks.workunit.client.0.smithi189.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2020-11-16T22:15:22.864 INFO:tasks.workunit.client.0.smithi189.stdout:Object prefix: benchmark_data_smithi189_657127 2020-11-16T22:15:22.864 INFO:tasks.workunit.client.0.smithi189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2020-11-16T22:15:22.865 INFO:tasks.workunit.client.0.smithi189.stdout: 0 0 0 0 0 0 - 0 2020-11-16T22:15:23.864 INFO:tasks.workunit.client.0.smithi189.stdout: 1 16 81 65 0.253884 0.253906 0.258613 0.206837 2020-11-16T22:15:24.865 INFO:tasks.workunit.client.0.smithi189.stdout:Total time run: 1.72556 2020-11-16T22:15:24.865 INFO:tasks.workunit.client.0.smithi189.stdout:Total writes made: 128 2020-11-16T22:15:24.865 INFO:tasks.workunit.client.0.smithi189.stdout:Write size: 4096 2020-11-16T22:15:24.865 INFO:tasks.workunit.client.0.smithi189.stdout:Object size: 4096 2020-11-16T22:15:24.865 INFO:tasks.workunit.client.0.smithi189.stdout:Bandwidth (MB/sec): 0.289761 2020-11-16T22:15:24.866 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Bandwidth: 0 2020-11-16T22:15:24.866 INFO:tasks.workunit.client.0.smithi189.stdout:Max bandwidth (MB/sec): 0.253906 2020-11-16T22:15:24.866 INFO:tasks.workunit.client.0.smithi189.stdout:Min bandwidth (MB/sec): 0.253906 2020-11-16T22:15:24.866 INFO:tasks.workunit.client.0.smithi189.stdout:Average IOPS: 74 2020-11-16T22:15:24.866 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev IOPS: 0 2020-11-16T22:15:24.866 INFO:tasks.workunit.client.0.smithi189.stdout:Max IOPS: 65 2020-11-16T22:15:24.866 INFO:tasks.workunit.client.0.smithi189.stdout:Min IOPS: 65 2020-11-16T22:15:24.867 INFO:tasks.workunit.client.0.smithi189.stdout:Average Latency(s): 0.214564 2020-11-16T22:15:24.867 INFO:tasks.workunit.client.0.smithi189.stdout:Stddev Latency(s): 0.0249123 2020-11-16T22:15:24.867 INFO:tasks.workunit.client.0.smithi189.stdout:Max latency(s): 0.258813 2020-11-16T22:15:24.867 INFO:tasks.workunit.client.0.smithi189.stdout:Min latency(s): 0.116361 2020-11-16T22:15:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2020-11-16T22:15:25.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2020-11-16T22:15:25.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2020-11-16T22:15:25.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2020-11-16T22:15:25.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2020-11-16T22:15:25.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=654685 2020-11-16T22:15:25.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 654685 -eq 0 ']' 2020-11-16T22:15:25.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=654685 2020-11-16T22:15:25.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 654685 2020-11-16T22:15:25.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2020-11-16T22:15:25.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2020-11-16T22:15:25.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2020-11-16T22:15:25.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2020-11-16T22:15:26.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2020-11-16T22:15:26.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2020-11-16T22:15:26.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2020-11-16T22:15:26.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2020-11-16T22:15:26.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2020-11-16T22:15:27.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2020-11-16T22:15:27.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2020-11-16T22:15:27.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2020-11-16T22:15:27.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2020-11-16T22:15:27.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2020-11-16T22:15:27.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2020-11-16T22:15:27.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 654685 -eq 654685 ']' 2020-11-16T22:15:27.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=656675 2020-11-16T22:15:27.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 656675 -eq 654685 ']' 2020-11-16T22:15:27.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=656675 2020-11-16T22:15:27.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 656675 2020-11-16T22:15:27.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2020-11-16T22:15:27.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2020-11-16T22:15:27.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2020-11-16T22:15:27.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2020-11-16T22:15:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2020-11-16T22:15:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2020-11-16T22:15:28.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2020-11-16T22:15:28.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2020-11-16T22:15:28.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2020-11-16T22:15:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2020-11-16T22:15:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2020-11-16T22:15:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2020-11-16T22:15:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2020-11-16T22:15:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2020-11-16T22:15:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2020-11-16T22:15:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2020-11-16T22:15:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2020-11-16T22:15:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:15:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2020-11-16T22:15:28.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:15:28.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:15:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:15:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:15:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:15:28.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:15:28.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:15:28.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:15:28.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:15:28.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:15:28.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:15:28.926 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:15:28.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:15:28.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:15:28.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:15:28.929 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:15:28.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:15:28.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:15:28.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2020-11-16T22:15:28.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:15:28.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:28.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:15:28.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.653760 2020-11-16T22:15:28.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:15:28.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:15:28.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2020-11-16T22:15:28.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T22:15:28.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-fast-mark-down 0 2020-11-16T22:15:28.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2020-11-16T22:15:28.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T22:15:28.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2020-11-16T22:15:28.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:15:28.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:15:28.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:15:28.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:15:28.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:15:28.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:15:28.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:15:28.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:15:28.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:15:28.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:15:28.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:15:28.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:15:28.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:15:28.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:15:28.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:15:28.955 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:15:28.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:15:28.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T22:15:28.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2020-11-16T22:15:28.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:15:28.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:28.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.653760 2020-11-16T22:15:28.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.653760 2020-11-16T22:15:28.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:15:28.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:15:28.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T22:15:28.964 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T22:15:28.964 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T22:15:29.019 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2020-11-16T22:15:29.019 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2020-11-16T22:15:29.076 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T22:15:29.088 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T22:15:29.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:15:29.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:15:29.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T22:15:29.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T22:15:29.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T22:15:29.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T22:15:29.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T22:15:29.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T22:15:29.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-force-create-pg 2020-11-16T22:15:29.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2020-11-16T22:15:29.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2020-11-16T22:15:29.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2020-11-16T22:15:29.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2020-11-16T22:15:29.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2020-11-16T22:15:29.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2020-11-16T22:15:29.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none ' 2020-11-16T22:15:29.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2020-11-16T22:15:29.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2020-11-16T22:15:29.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T22:15:29.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2020-11-16T22:15:29.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2020-11-16T22:15:29.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: TEST_reuse_id td/osd-force-create-pg 2020-11-16T22:15:29.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:20: TEST_reuse_id: local dir=td/osd-force-create-pg 2020-11-16T22:15:29.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: setup td/osd-force-create-pg 2020-11-16T22:15:29.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2020-11-16T22:15:29.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2020-11-16T22:15:29.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2020-11-16T22:15:29.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:15:29.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2020-11-16T22:15:29.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:15:29.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:15:29.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:15:29.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:15:29.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:15:29.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:15:29.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:15:29.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:15:29.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:15:29.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:15:29.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:15:29.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:15:29.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:15:29.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:15:29.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:15:29.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:15:29.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:15:29.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:15:29.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2020-11-16T22:15:29.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:15:29.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:29.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:15:29.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.657260 2020-11-16T22:15:29.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:15:29.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:15:29.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2020-11-16T22:15:29.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:15:29.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:29.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:15:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.657260 2020-11-16T22:15:29.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:15:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T22:15:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T22:15:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:15:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2020-11-16T22:15:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:23: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2020-11-16T22:15:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2020-11-16T22:15:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:15:29.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:15:29.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:15:29.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2020-11-16T22:15:29.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2020-11-16T22:15:29.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:15:29.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:15:29.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:15:29.405 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:15:29.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:29.406 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:15:29.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:15:29.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2020-11-16T22:15:29.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:15:29.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:15:29.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:15:29.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:15:29.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:15:29.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:15:29.619 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:15:29.619 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:15:29.619 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:15:29.620 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:15:29.620 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:29.620 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:15:29.620 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.657260/ceph-mon.a.asok 2020-11-16T22:15:29.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:15:29.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.657260/ceph-mon.a.asok config get fsid 2020-11-16T22:15:29.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:15:29.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:15:29.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:15:29.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:15:29.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:15:29.730 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:15:29.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:15:29.731 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:15:29.731 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:15:29.732 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:29.732 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:15:29.732 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.657260/ceph-mon.a.asok 2020-11-16T22:15:29.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:15:29.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.657260/ceph-mon.a.asok config get mon_host 2020-11-16T22:15:29.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2020-11-16T22:15:29.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-force-create-pg 2020-11-16T22:15:29.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:15:29.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:15:29.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:15:29.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-force-create-pg/x 2020-11-16T22:15:29.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:15:30.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:15:30.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:15:30.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:15:30.355 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:15:30.355 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:30.355 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:15:30.356 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:15:30.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:15:30.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:15:30.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2020-11-16T22:15:30.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-force-create-pg 2020-11-16T22:15:30.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:15:30.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:15:30.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:15:30.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-force-create-pg/0 2020-11-16T22:15:30.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2020-11-16T22:15:30.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:15:30.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:15:30.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:15:30.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2020-11-16T22:15:30.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2020-11-16T22:15:30.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:15:30.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:15:30.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2020-11-16T22:15:30.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:15:30.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:15:30.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:15:30.388 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:15:30.389 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:30.389 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:15:30.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:15:30.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:15:30.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:15:30.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:15:30.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:15:30.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2020-11-16T22:15:30.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2020-11-16T22:15:30.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:15:30.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:15:30.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:15:30.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:15:30.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:15:30.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-force-create-pg/0 2020-11-16T22:15:30.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:15:30.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3ba4d662-7196-4cf0-9064-746ae9a8b55c 2020-11-16T22:15:30.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 3ba4d662-7196-4cf0-9064-746ae9a8b55c' 2020-11-16T22:15:30.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:15:30.392 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 3ba4d662-7196-4cf0-9064-746ae9a8b55c 2020-11-16T22:15:30.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAC+rJfAxPHGBAAkzkbXsN0UYCSLLkpkci8jw== 2020-11-16T22:15:30.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAC+rJfAxPHGBAAkzkbXsN0UYCSLLkpkci8jw=="}' 2020-11-16T22:15:30.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3ba4d662-7196-4cf0-9064-746ae9a8b55c -i td/osd-force-create-pg/0/new.json 2020-11-16T22:15:30.738 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:15:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-force-create-pg/0/new.json 2020-11-16T22:15:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAC+rJfAxPHGBAAkzkbXsN0UYCSLLkpkci8jw== --osd-uuid 3ba4d662-7196-4cf0-9064-746ae9a8b55c 2020-11-16T22:15:30.777 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:30.787+0000 7f3a3e920d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:30.779 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:30.787+0000 7f3a3e920d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:30.780 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:30.791+0000 7f3a3e920d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:30.781 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:30.791+0000 7f3a3e920d80 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2020-11-16T22:15:30.781 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:30.791+0000 7f3a3e920d80 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2020-11-16T22:15:30.781 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:30.791+0000 7f3a3e920d80 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2020-11-16T22:15:30.825 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:30.835+0000 7f3a3e920d80 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2020-11-16T22:15:35.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2020-11-16T22:15:35.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:15:35.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:15:35.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:15:35.172 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:15:35.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:15:35.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:15:35.660 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:15:35.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:15:35.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:15:35.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:15:35.677 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:35.687+0000 7f4791ca1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:35.680 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:35.691+0000 7f4791ca1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:35.681 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:35.691+0000 7f4791ca1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:36.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:15:36.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:15:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:15:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:15:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:15:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:15:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:36.025 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:15:36.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:15:36.225 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:36.235+0000 7f4791ca1d80 -1 Falling back to public interface 2020-11-16T22:15:36.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:15:37.054 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:37.063+0000 7f4791ca1d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:15:37.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:15:37.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:37.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:15:37.397 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:15:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:15:37.808 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/658113,v1:127.0.0.1:6803/658113] [v2:127.0.0.1:6804/658113,v1:127.0.0.1:6805/658113] exists,up 3ba4d662-7196-4cf0-9064-746ae9a8b55c 2020-11-16T22:15:37.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:15:37.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:15:37.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:15:37.809 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2020-11-16T22:15:37.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-force-create-pg 2020-11-16T22:15:37.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:15:37.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:15:37.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:15:37.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-force-create-pg/1 2020-11-16T22:15:37.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2020-11-16T22:15:37.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:15:37.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:15:37.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:15:37.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2020-11-16T22:15:37.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2020-11-16T22:15:37.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:15:37.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:15:37.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2020-11-16T22:15:37.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:15:37.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:15:37.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:15:37.813 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:15:37.814 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:37.814 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:15:37.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:15:37.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:15:37.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:15:37.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:15:37.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:15:37.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2020-11-16T22:15:37.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2020-11-16T22:15:37.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:15:37.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:15:37.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:15:37.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:15:37.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:15:37.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-force-create-pg/1 2020-11-16T22:15:37.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:15:37.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=51aa1129-4aa1-4913-b30e-b94e51fcfce7 2020-11-16T22:15:37.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 51aa1129-4aa1-4913-b30e-b94e51fcfce7' 2020-11-16T22:15:37.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:15:37.816 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 51aa1129-4aa1-4913-b30e-b94e51fcfce7 2020-11-16T22:15:37.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAJ+rJfZQAvMhAAN9yhibMmDUAOMfJlyTtSAw== 2020-11-16T22:15:37.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAJ+rJfZQAvMhAAN9yhibMmDUAOMfJlyTtSAw=="}' 2020-11-16T22:15:37.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 51aa1129-4aa1-4913-b30e-b94e51fcfce7 -i td/osd-force-create-pg/1/new.json 2020-11-16T22:15:38.405 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:15:38.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-force-create-pg/1/new.json 2020-11-16T22:15:38.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAJ+rJfZQAvMhAAN9yhibMmDUAOMfJlyTtSAw== --osd-uuid 51aa1129-4aa1-4913-b30e-b94e51fcfce7 2020-11-16T22:15:38.446 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:38.455+0000 7f12df51fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:38.448 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:38.459+0000 7f12df51fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:38.450 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:38.459+0000 7f12df51fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:38.451 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:38.459+0000 7f12df51fd80 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2020-11-16T22:15:38.451 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:38.459+0000 7f12df51fd80 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2020-11-16T22:15:38.451 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:38.459+0000 7f12df51fd80 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2020-11-16T22:15:38.527 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:38.535+0000 7f12df51fd80 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2020-11-16T22:15:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2020-11-16T22:15:42.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:15:42.858 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:15:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:15:42.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:15:43.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:15:43.356 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:15:43.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:15:43.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:15:43.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:15:43.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:15:43.383 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:43.391+0000 7fbe0486bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:43.385 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:43.395+0000 7fbe0486bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:43.387 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:43.395+0000 7fbe0486bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:43.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:15:43.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:15:43.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:15:43.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:15:43.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:15:43.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:43.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:15:43.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:43.726 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:15:43.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:15:43.929 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:43.939+0000 7fbe0486bd80 -1 Falling back to public interface 2020-11-16T22:15:44.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:15:44.839 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:44.847+0000 7fbe0486bd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:15:45.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:15:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:15:45.117 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:15:45.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:45.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:15:45.565 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/659107,v1:127.0.0.1:6811/659107] [v2:127.0.0.1:6812/659107,v1:127.0.0.1:6813/659107] exists,up 51aa1129-4aa1-4913-b30e-b94e51fcfce7 2020-11-16T22:15:45.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:15:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:15:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:15:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2020-11-16T22:15:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-force-create-pg 2020-11-16T22:15:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:15:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T22:15:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:15:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-force-create-pg/2 2020-11-16T22:15:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2020-11-16T22:15:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:15:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:15:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:15:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2020-11-16T22:15:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2020-11-16T22:15:45.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:15:45.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:15:45.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2020-11-16T22:15:45.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:15:45.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:15:45.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:15:45.570 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:15:45.570 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:15:45.570 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:15:45.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:15:45.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:15:45.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:15:45.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:15:45.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:15:45.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2020-11-16T22:15:45.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2020-11-16T22:15:45.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:15:45.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:15:45.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:15:45.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:15:45.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:15:45.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-force-create-pg/2 2020-11-16T22:15:45.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:15:45.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5e683df2-8938-449d-b117-008db50f4f6c 2020-11-16T22:15:45.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 5e683df2-8938-449d-b117-008db50f4f6c' 2020-11-16T22:15:45.573 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 5e683df2-8938-449d-b117-008db50f4f6c 2020-11-16T22:15:45.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:15:45.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAR+rJfyPgXJBAAyZO3HsqHQTbrsByNiHwEJA== 2020-11-16T22:15:45.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAR+rJfyPgXJBAAyZO3HsqHQTbrsByNiHwEJA=="}' 2020-11-16T22:15:45.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5e683df2-8938-449d-b117-008db50f4f6c -i td/osd-force-create-pg/2/new.json 2020-11-16T22:15:46.182 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:15:46.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-force-create-pg/2/new.json 2020-11-16T22:15:46.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAR+rJfyPgXJBAAyZO3HsqHQTbrsByNiHwEJA== --osd-uuid 5e683df2-8938-449d-b117-008db50f4f6c 2020-11-16T22:15:46.218 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:46.227+0000 7f9a093f1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:46.221 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:46.231+0000 7f9a093f1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:46.231+0000 7f9a093f1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:46.231+0000 7f9a093f1d80 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2020-11-16T22:15:46.223 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:46.231+0000 7f9a093f1d80 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2020-11-16T22:15:46.223 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:46.231+0000 7f9a093f1d80 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2020-11-16T22:15:46.277 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:46.287+0000 7f9a093f1d80 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2020-11-16T22:15:50.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2020-11-16T22:15:50.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:15:50.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T22:15:50.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:15:50.957 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T22:15:51.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T22:15:51.457 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:15:51.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:15:51.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:15:51.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:15:51.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:15:51.484 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:51.495+0000 7fb7a5115d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:51.486 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:51.495+0000 7fb7a5115d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:51.487 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:51.499+0000 7fb7a5115d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:15:51.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T22:15:51.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:15:51.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:15:51.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:15:51.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:15:51.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:51.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:15:51.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:51.843 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:15:51.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:15:52.042 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:52.051+0000 7fb7a5115d80 -1 Falling back to public interface 2020-11-16T22:15:52.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:15:52.876 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:15:52.887+0000 7fb7a5115d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T22:15:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:15:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:15:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:15:53.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:15:53.226 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:15:53.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:15:53.619 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/660101,v1:127.0.0.1:6819/660101] [v2:127.0.0.1:6820/660101,v1:127.0.0.1:6821/660101] exists,up 5e683df2-8938-449d-b117-008db50f4f6c 2020-11-16T22:15:53.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:15:53.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:15:53.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:15:53.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:29: TEST_reuse_id: ceph osd pool create foo 50 2020-11-16T22:15:54.103 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' already exists 2020-11-16T22:15:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: wait_for_clean 2020-11-16T22:15:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:15:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:15:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:15:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:15:54.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:15:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:15:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:15:54.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:15:54.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:15:54.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:15:54.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:15:54.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:15:54.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:15:54.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:15:54.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:15:54.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:15:54.779 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:15:54.779 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T22:15:54.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:15:54.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:15:54.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:15:54.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836486 2020-11-16T22:15:54.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836486 2020-11-16T22:15:54.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486' 2020-11-16T22:15:54.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:15:54.949 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:15:55.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672964 2020-11-16T22:15:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672964 2020-11-16T22:15:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486 1-42949672964' 2020-11-16T22:15:55.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:15:55.122 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:15:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509443 2020-11-16T22:15:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509443 2020-11-16T22:15:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486 1-42949672964 2-64424509443' 2020-11-16T22:15:55.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:15:55.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836486 2020-11-16T22:15:55.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:15:55.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:15:55.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836486 2020-11-16T22:15:55.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:15:55.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836486 2020-11-16T22:15:55.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-11-16T22:15:55.287 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836486 2020-11-16T22:15:55.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:15:55.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836484 -lt 21474836486 2020-11-16T22:15:55.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:15:56.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:15:56.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:15:57.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836486 -lt 21474836486 2020-11-16T22:15:57.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:15:57.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672964 2020-11-16T22:15:57.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:15:57.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:15:57.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672964 2020-11-16T22:15:57.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:15:57.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672964 2020-11-16T22:15:57.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2020-11-16T22:15:57.074 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672964 2020-11-16T22:15:57.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:15:57.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672964 -lt 42949672964 2020-11-16T22:15:57.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:15:57.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509443 2020-11-16T22:15:57.452 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:15:57.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:15:57.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509443 2020-11-16T22:15:57.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:15:57.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509443 2020-11-16T22:15:57.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2020-11-16T22:15:57.457 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509443 2020-11-16T22:15:57.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:15:57.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509443 -lt 64424509443 2020-11-16T22:15:57.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:15:57.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:15:57.841 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:15:58.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 50 == 0 2020-11-16T22:15:58.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:15:58.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:15:58.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:15:58.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:15:58.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:15:58.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:15:58.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:15:58.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=36 2020-11-16T22:15:58.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:15:58.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:15:58.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:15:59.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 36 = 50 2020-11-16T22:15:59.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 36 '!=' -1 2020-11-16T22:15:59.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T22:15:59.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=36 2020-11-16T22:15:59.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:15:59.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:15:59.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:15:59.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:15:59.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:15:59.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:15:59.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:15:59.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:15:59.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:15:59.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:15:59.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=36 2020-11-16T22:15:59.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:15:59.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:15:59.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:16:00.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 36 = 50 2020-11-16T22:16:00.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 36 '!=' 36 2020-11-16T22:16:00.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:16:00.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:16:00.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:16:00.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:16:00.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:16:00.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:16:00.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:16:00.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-11-16T22:16:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-11-16T22:16:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2020-11-16T22:16:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:16:00.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-11-16T22:16:00.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:16:00.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:16:00.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:16:00.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:16:00.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:16:00.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:16:00.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:16:00.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:16:01.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=50 2020-11-16T22:16:01.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:16:01.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:16:01.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:16:01.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 50 = 50 2020-11-16T22:16:01.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:16:01.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:16:01.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:32: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2020-11-16T22:16:01.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:16:01.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:16:01.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:16:01.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:16:01.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:16:01.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:16:01.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2020-11-16T22:16:01.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:16:01.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:16:01.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:16:01.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:16:01.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:16:01.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:16:01.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2020-11-16T22:16:01.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:16:01.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:16:01.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:16:01.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:16:01.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:16:01.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:16:01.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2020-11-16T22:16:02.960 INFO:tasks.workunit.client.0.smithi189.stderr:PG '1.0' not found 2020-11-16T22:16:03.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2020-11-16T22:16:04.544 INFO:tasks.workunit.client.0.smithi189.stdout: marking collection for removal 2020-11-16T22:16:04.545 INFO:tasks.workunit.client.0.smithi189.stdout:setting '_remove' omap key 2020-11-16T22:16:04.545 INFO:tasks.workunit.client.0.smithi189.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-11-16T22:16:04.652 INFO:tasks.workunit.client.0.smithi189.stdout:Remove successful 2020-11-16T22:16:05.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2020-11-16T22:16:06.031 INFO:tasks.workunit.client.0.smithi189.stderr:PG '1.0' not found 2020-11-16T22:16:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2020-11-16T22:16:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-force-create-pg 2020-11-16T22:16:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:16:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T22:16:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:16:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-force-create-pg/0 2020-11-16T22:16:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2020-11-16T22:16:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:16:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:16:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:16:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2020-11-16T22:16:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2020-11-16T22:16:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:16:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:16:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2020-11-16T22:16:06.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:16:06.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:16:06.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:16:06.577 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:16:06.577 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:06.578 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:16:06.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:16:06.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:16:06.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:16:06.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2020-11-16T22:16:06.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2020-11-16T22:16:06.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:16:06.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:16:06.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:06.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:16:06.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:16:06.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-force-create-pg/0 2020-11-16T22:16:06.580 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:16:06.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T22:16:06.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:06.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-force-create-pg/0/whoami 2020-11-16T22:16:06.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T22:16:06.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:16:06.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:16:06.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:16:06.605 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:06.615+0000 7fd604861d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:06.608 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:06.619+0000 7fd604861d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:06.610 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:06.619+0000 7fd604861d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:06.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T22:16:06.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:16:06.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:16:06.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:16:06.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:16:06.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:06.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:16:06.972 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:16:06.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:16:06.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:07.154 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:07.163+0000 7fd604861d80 -1 Falling back to public interface 2020-11-16T22:16:07.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:16:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:08.147+0000 7fd604861d80 -1 osd.0 21 log_to_monitors {default=true} 2020-11-16T22:16:08.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:16:08.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:08.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:16:08.373 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:16:08.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:08.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:16:08.770 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 24 up_thru 24 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/661879,v1:127.0.0.1:6803/661879] [v2:127.0.0.1:6804/661879,v1:127.0.0.1:6805/661879] exists,up 3ba4d662-7196-4cf0-9064-746ae9a8b55c 2020-11-16T22:16:08.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:16:08.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:16:08.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:16:08.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2020-11-16T22:16:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-force-create-pg 2020-11-16T22:16:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:16:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-11-16T22:16:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:16:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-force-create-pg/1 2020-11-16T22:16:08.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2020-11-16T22:16:08.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:16:08.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:16:08.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:16:08.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2020-11-16T22:16:08.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2020-11-16T22:16:08.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:16:08.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:16:08.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2020-11-16T22:16:08.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:16:08.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:16:08.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:16:08.776 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:16:08.776 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:08.776 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:16:08.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:16:08.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:16:08.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:16:08.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2020-11-16T22:16:08.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2020-11-16T22:16:08.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:16:08.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:16:08.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:08.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:16:08.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:16:08.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-force-create-pg/1 2020-11-16T22:16:08.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-11-16T22:16:08.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:08.778 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-force-create-pg/1/whoami 2020-11-16T22:16:08.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-11-16T22:16:08.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:16:08.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:16:08.779 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:16:08.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:16:08.794 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:08.803+0000 7fb2b9d07d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:08.796 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:08.807+0000 7fb2b9d07d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:08.798 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:08.807+0000 7fb2b9d07d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:09.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-11-16T22:16:09.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:16:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:16:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:16:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:16:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:16:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:09.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:16:09.201 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:16:09.338 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:09.347+0000 7fb2b9d07d80 -1 Falling back to public interface 2020-11-16T22:16:09.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:16:10.373 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:10.383+0000 7fb2b9d07d80 -1 osd.1 21 log_to_monitors {default=true} 2020-11-16T22:16:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:16:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:16:10.604 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:16:10.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:10.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:16:11.005 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [10,21) [v2:127.0.0.1:6810/662278,v1:127.0.0.1:6811/662278] [v2:127.0.0.1:6812/662278,v1:127.0.0.1:6813/662278] exists,up 51aa1129-4aa1-4913-b30e-b94e51fcfce7 2020-11-16T22:16:11.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:16:11.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:16:11.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:16:11.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2020-11-16T22:16:11.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-force-create-pg 2020-11-16T22:16:11.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:16:11.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-11-16T22:16:11.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:16:11.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-force-create-pg/2 2020-11-16T22:16:11.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2020-11-16T22:16:11.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:16:11.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:16:11.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:16:11.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2020-11-16T22:16:11.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2020-11-16T22:16:11.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:16:11.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:16:11.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2020-11-16T22:16:11.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:16:11.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:16:11.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:16:11.011 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:16:11.011 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:11.011 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:16:11.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:16:11.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' 2020-11-16T22:16:11.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:16:11.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2020-11-16T22:16:11.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2020-11-16T22:16:11.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:16:11.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:16:11.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:11.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:16:11.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:16:11.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-force-create-pg/2 2020-11-16T22:16:11.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-11-16T22:16:11.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=b0ef020b-ae19-4583-bb73-9caf66c7aac5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.657260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:11.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-force-create-pg/2/whoami 2020-11-16T22:16:11.014 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:16:11.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-11-16T22:16:11.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:16:11.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:16:11.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:16:11.035 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:11.043+0000 7fdc90c15d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:11.037 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:11.047+0000 7fdc90c15d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:11.038 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:11.047+0000 7fdc90c15d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:11.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-11-16T22:16:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:16:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:16:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:16:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:16:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:16:11.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:11.392 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:16:11.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:16:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:11.595+0000 7fdc90c15d80 -1 Falling back to public interface 2020-11-16T22:16:11.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:16:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:12.655+0000 7fdc90c15d80 -1 osd.2 21 log_to_monitors {default=true} 2020-11-16T22:16:12.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:16:12.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:12.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:16:12.773 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:16:12.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:12.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:16:13.277 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 28 up_thru 28 down_at 25 last_clean_interval [15,21) [v2:127.0.0.1:6818/662677,v1:127.0.0.1:6819/662677] [v2:127.0.0.1:6820/662677,v1:127.0.0.1:6821/662677] exists,up 5e683df2-8938-449d-b117-008db50f4f6c 2020-11-16T22:16:13.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:16:13.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:16:13.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:16:13.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: sleep 10 2020-11-16T22:16:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: ceph pg ls 2020-11-16T22:16:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: grep 1.0 2020-11-16T22:16:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: grep stale 2020-11-16T22:16:23.651 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 0 0 0 0 0 0 0 0 stale+active+clean 28s 0'0 20:10 [1]p1 [1]p1 2020-11-16T22:15:53.918448+0000 2020-11-16T22:15:53.918448+0000 2020-11-16T22:16:23.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:44: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2020-11-16T22:16:24.196 INFO:tasks.workunit.client.0.smithi189.stderr:pg 1.0 already creating 2020-11-16T22:16:24.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: wait_for_clean 2020-11-16T22:16:24.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:16:24.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:16:24.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:16:24.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:16:24.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:16:24.207 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:16:24.207 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:16:24.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:16:24.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:16:24.208 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:16:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:16:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:16:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:16:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:16:24.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:16:24.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:16:24.818 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:16:24.818 INFO:tasks.workunit.client.0.smithi189.stderr:2' 2020-11-16T22:16:24.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:16:24.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:16:24.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:16:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215110 2020-11-16T22:16:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215110 2020-11-16T22:16:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-103079215110' 2020-11-16T22:16:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:16:24.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:16:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149701 2020-11-16T22:16:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149701 2020-11-16T22:16:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-103079215110 1-111669149701' 2020-11-16T22:16:25.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:16:25.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:16:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084293 2020-11-16T22:16:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084293 2020-11-16T22:16:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-103079215110 1-111669149701 2-120259084293' 2020-11-16T22:16:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:16:25.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-103079215110 2020-11-16T22:16:25.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:16:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:16:25.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-103079215110 2020-11-16T22:16:25.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:16:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215110 2020-11-16T22:16:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2020-11-16T22:16:25.297 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 103079215110 2020-11-16T22:16:25.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:16:25.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215108 -lt 103079215110 2020-11-16T22:16:25.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:16:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:16:26.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:16:27.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215110 -lt 103079215110 2020-11-16T22:16:27.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:16:27.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-111669149701 2020-11-16T22:16:27.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:16:27.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:16:27.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:16:27.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-111669149701 2020-11-16T22:16:27.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149701 2020-11-16T22:16:27.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 111669149701' 2020-11-16T22:16:27.085 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 111669149701 2020-11-16T22:16:27.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:16:27.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149702 -lt 111669149701 2020-11-16T22:16:27.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:16:27.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-120259084293 2020-11-16T22:16:27.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:16:27.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:16:27.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-120259084293 2020-11-16T22:16:27.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:16:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084293 2020-11-16T22:16:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 120259084293' 2020-11-16T22:16:27.467 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 120259084293 2020-11-16T22:16:27.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:16:27.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084293 -lt 120259084293 2020-11-16T22:16:27.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:16:27.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:16:27.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:16:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 50 == 0 2020-11-16T22:16:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:16:28.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:16:28.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:16:28.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:16:28.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:16:28.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:16:28.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:16:28.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=50 2020-11-16T22:16:28.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:16:28.741 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:16:28.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:16:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 50 = 50 2020-11-16T22:16:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:16:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:16:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T22:16:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-force-create-pg 0 2020-11-16T22:16:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2020-11-16T22:16:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T22:16:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2020-11-16T22:16:29.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:16:29.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:16:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:16:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:16:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:16:29.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:16:29.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:16:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:16:29.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:16:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:16:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:16:29.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:16:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:16:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:16:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:16:29.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:16:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:16:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T22:16:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2020-11-16T22:16:29.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:16:29.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:29.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.657260 2020-11-16T22:16:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.657260 2020-11-16T22:16:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:16:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:16:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T22:16:29.353 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T22:16:29.354 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T22:16:29.409 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2020-11-16T22:16:29.409 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2020-11-16T22:16:29.468 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T22:16:29.484 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T22:16:29.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:16:29.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:16:29.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T22:16:29.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T22:16:29.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T22:16:29.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T22:16:29.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T22:16:29.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T22:16:29.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-markdown 2020-11-16T22:16:29.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2020-11-16T22:16:29.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2020-11-16T22:16:29.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2020-11-16T22:16:29.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2020-11-16T22:16:29.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2020-11-16T22:16:29.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2020-11-16T22:16:29.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none ' 2020-11-16T22:16:29.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2020-11-16T22:16:29.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2020-11-16T22:16:29.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T22:16:29.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2020-11-16T22:16:29.490 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_markdown_boot_exceed_time 2020-11-16T22:16:29.490 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_markdown_exceed_maxdown_count 2020-11-16T22:16:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_osd_stop' 2020-11-16T22:16:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2020-11-16T22:16:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2020-11-16T22:16:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2020-11-16T22:16:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2020-11-16T22:16:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2020-11-16T22:16:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:16:29.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2020-11-16T22:16:29.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:16:29.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:16:29.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:16:29.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:16:29.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:16:29.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:16:29.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:16:29.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:16:29.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:16:29.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:16:29.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:16:29.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:16:29.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:16:29.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:16:29.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:16:29.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:16:29.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:16:29.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:16:29.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2020-11-16T22:16:29.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:16:29.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:29.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:16:29.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.663547 2020-11-16T22:16:29.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:16:29.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:16:29.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2020-11-16T22:16:29.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:16:29.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:29.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:16:29.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.663547 2020-11-16T22:16:29.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:16:29.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T22:16:29.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T22:16:29.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:16:29.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2020-11-16T22:16:29.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2020-11-16T22:16:29.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2020-11-16T22:16:29.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2020-11-16T22:16:29.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2020-11-16T22:16:29.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:16:29.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:16:29.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:16:29.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2020-11-16T22:16:29.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2020-11-16T22:16:29.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:16:29.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:16:29.811 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:16:29.811 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:16:29.812 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:29.812 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:16:29.812 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:16:29.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:16:30.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:16:30.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:16:30.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:16:30.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:16:30.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:16:30.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:16:30.026 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:16:30.026 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:16:30.026 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:16:30.027 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:16:30.027 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:30.027 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:16:30.027 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.663547/ceph-mon.a.asok 2020-11-16T22:16:30.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:16:30.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.663547/ceph-mon.a.asok config get fsid 2020-11-16T22:16:30.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:16:30.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:16:30.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:16:30.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:16:30.133 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:16:30.133 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:16:30.133 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:16:30.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:16:30.134 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:16:30.134 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:30.134 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:16:30.134 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.663547/ceph-mon.a.asok 2020-11-16T22:16:30.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:16:30.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.663547/ceph-mon.a.asok config get mon_host 2020-11-16T22:16:30.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2020-11-16T22:16:30.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-markdown 2020-11-16T22:16:30.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:16:30.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:16:30.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:16:30.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-markdown/x 2020-11-16T22:16:30.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:16:30.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:16:30.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:16:30.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:16:30.769 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:16:30.770 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:30.770 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:16:30.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:16:30.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:16:30.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:16:30.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2020-11-16T22:16:30.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:16:30.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:16:30.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:16:30.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:16:30.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/0 2020-11-16T22:16:30.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:16:30.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:16:30.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:16:30.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:16:30.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2020-11-16T22:16:30.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2020-11-16T22:16:30.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:16:30.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:16:30.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:16:30.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:16:30.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:16:30.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:16:30.804 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:16:30.804 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:30.804 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:16:30.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:16:30.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:16:30.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:16:30.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:16:30.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:16:30.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:16:30.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:16:30.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:16:30.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:16:30.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:30.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:16:30.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:16:30.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/0 2020-11-16T22:16:30.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:16:30.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=bf7471aa-1832-438f-a256-b370d3ce4aa5 2020-11-16T22:16:30.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 bf7471aa-1832-438f-a256-b370d3ce4aa5' 2020-11-16T22:16:30.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:16:30.808 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 bf7471aa-1832-438f-a256-b370d3ce4aa5 2020-11-16T22:16:30.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA++rJfjDKIMRAAoS94ASeQ9JvPd9GNYFdA0Q== 2020-11-16T22:16:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA++rJfjDKIMRAAoS94ASeQ9JvPd9GNYFdA0Q=="}' 2020-11-16T22:16:30.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new bf7471aa-1832-438f-a256-b370d3ce4aa5 -i td/osd-markdown/0/new.json 2020-11-16T22:16:31.164 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:16:31.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/0/new.json 2020-11-16T22:16:31.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA++rJfjDKIMRAAoS94ASeQ9JvPd9GNYFdA0Q== --osd-uuid bf7471aa-1832-438f-a256-b370d3ce4aa5 2020-11-16T22:16:31.199 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:31.206+0000 7f4e01175d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:31.202 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:31.210+0000 7f4e01175d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:31.204 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:31.214+0000 7f4e01175d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:31.204 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:31.214+0000 7f4e01175d80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:16:31.205 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:31.214+0000 7f4e01175d80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:16:31.205 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:31.214+0000 7f4e01175d80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:16:31.247 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:31.258+0000 7f4e01175d80 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2020-11-16T22:16:35.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/0/keyring 2020-11-16T22:16:35.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:16:35.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:16:35.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:16:35.803 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:16:36.302 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:16:36.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:16:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:16:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:16:36.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:16:36.328 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:36.338+0000 7fae2849fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:36.331 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:36.338+0000 7fae2849fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:36.332 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:36.342+0000 7fae2849fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:36.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:16:36.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:16:36.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:16:36.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:16:36.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:16:36.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:36.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:16:36.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:36.681 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:16:36.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:16:36.874 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:36.882+0000 7fae2849fd80 -1 Falling back to public interface 2020-11-16T22:16:37.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:16:37.702 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:37.710+0000 7fae2849fd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:16:38.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:16:38.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:38.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:16:38.053 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:16:38.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:38.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:16:38.470 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/664400,v1:127.0.0.1:6803/664400] [v2:127.0.0.1:6804/664400,v1:127.0.0.1:6805/664400] exists,up bf7471aa-1832-438f-a256-b370d3ce4aa5 2020-11-16T22:16:38.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:16:38.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:16:38.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:16:38.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2020-11-16T22:16:38.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:16:38.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:16:38.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:16:38.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:16:38.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/1 2020-11-16T22:16:38.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:16:38.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:16:38.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:16:38.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:16:38.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2020-11-16T22:16:38.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2020-11-16T22:16:38.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:16:38.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:16:38.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:16:38.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:16:38.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:16:38.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:16:38.475 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:16:38.475 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:38.475 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:16:38.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:16:38.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:16:38.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:16:38.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:16:38.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:16:38.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:16:38.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:16:38.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:16:38.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:16:38.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:38.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:16:38.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:16:38.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/1 2020-11-16T22:16:38.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:16:38.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b0e5059b-961a-4ba8-a50e-ade805fce3d6 2020-11-16T22:16:38.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 b0e5059b-961a-4ba8-a50e-ade805fce3d6' 2020-11-16T22:16:38.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:16:38.478 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 b0e5059b-961a-4ba8-a50e-ade805fce3d6 2020-11-16T22:16:38.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBG+rJfMZfcHRAAFxj2P6qxXB3zb2znR2+JVw== 2020-11-16T22:16:38.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBG+rJfMZfcHRAAFxj2P6qxXB3zb2znR2+JVw=="}' 2020-11-16T22:16:38.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b0e5059b-961a-4ba8-a50e-ade805fce3d6 -i td/osd-markdown/1/new.json 2020-11-16T22:16:39.074 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:16:39.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/1/new.json 2020-11-16T22:16:39.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBG+rJfMZfcHRAAFxj2P6qxXB3zb2znR2+JVw== --osd-uuid b0e5059b-961a-4ba8-a50e-ade805fce3d6 2020-11-16T22:16:39.112 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:39.122+0000 7fd4619fbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:39.115 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:39.122+0000 7fd4619fbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:39.116 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:39.126+0000 7fd4619fbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:39.117 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:39.126+0000 7fd4619fbd80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:16:39.117 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:39.126+0000 7fd4619fbd80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:16:39.117 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:39.126+0000 7fd4619fbd80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:16:39.167 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:39.174+0000 7fd4619fbd80 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2020-11-16T22:16:43.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/1/keyring 2020-11-16T22:16:43.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:16:43.813 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:16:43.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:16:43.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:16:44.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:16:44.313 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:16:44.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:16:44.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:44.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:16:44.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:16:44.337 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:44.346+0000 7f60f85f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:44.339 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:44.350+0000 7f60f85f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:44.340 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:44.350+0000 7f60f85f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:16:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:16:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:16:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:16:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:16:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:16:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:16:44.664 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:16:44.889 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:44.898+0000 7f60f85f3d80 -1 Falling back to public interface 2020-11-16T22:16:45.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:16:45.756 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:45.766+0000 7f60f85f3d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:16:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:16:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:16:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:46.038 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:16:46.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:16:46.447 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/665394,v1:127.0.0.1:6811/665394] [v2:127.0.0.1:6812/665394,v1:127.0.0.1:6813/665394] exists,up b0e5059b-961a-4ba8-a50e-ade805fce3d6 2020-11-16T22:16:46.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:16:46.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:16:46.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:16:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2020-11-16T22:16:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:16:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:16:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T22:16:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:16:46.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/2 2020-11-16T22:16:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:16:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:16:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:16:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:16:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2020-11-16T22:16:46.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2020-11-16T22:16:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:16:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:16:46.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:16:46.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:16:46.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:16:46.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:16:46.454 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:16:46.454 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:16:46.454 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:16:46.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:16:46.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:16:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:16:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:16:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:16:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:16:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:16:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:16:46.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:16:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:16:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:16:46.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/2 2020-11-16T22:16:46.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:16:46.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b5d5ab80-f398-47db-ba9d-15f1d2d65b7a 2020-11-16T22:16:46.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 b5d5ab80-f398-47db-ba9d-15f1d2d65b7a' 2020-11-16T22:16:46.458 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 b5d5ab80-f398-47db-ba9d-15f1d2d65b7a 2020-11-16T22:16:46.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:16:46.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBO+rJfWXUfHRAAJ+5A+PxisfYldBH1sVAL+Q== 2020-11-16T22:16:46.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBO+rJfWXUfHRAAJ+5A+PxisfYldBH1sVAL+Q=="}' 2020-11-16T22:16:46.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b5d5ab80-f398-47db-ba9d-15f1d2d65b7a -i td/osd-markdown/2/new.json 2020-11-16T22:16:46.972 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:16:46.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/2/new.json 2020-11-16T22:16:46.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBO+rJfWXUfHRAAJ+5A+PxisfYldBH1sVAL+Q== --osd-uuid b5d5ab80-f398-47db-ba9d-15f1d2d65b7a 2020-11-16T22:16:47.011 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:47.018+0000 7f11b8399d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:47.014 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:47.022+0000 7f11b8399d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:47.015 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:47.022+0000 7f11b8399d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:47.016 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:47.026+0000 7f11b8399d80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:16:47.016 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:47.026+0000 7f11b8399d80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:16:47.016 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:47.026+0000 7f11b8399d80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:16:47.053 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:47.062+0000 7f11b8399d80 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2020-11-16T22:16:51.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/2/keyring 2020-11-16T22:16:51.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:16:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T22:16:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:16:51.389 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T22:16:51.895 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:16:51.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T22:16:51.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:16:51.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:16:51.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:16:51.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:16:51.917 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:51.926+0000 7f3c40a3ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:51.920 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:51.930+0000 7f3c40a3ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:51.921 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:51.930+0000 7f3c40a3ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:16:52.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T22:16:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:16:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:16:56.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:16:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:16:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:16:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:56.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:16:56.411 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:16:56.413 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:52.482+0000 7f3c40a3ad80 -1 Falling back to public interface 2020-11-16T22:16:56.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:16:56.414 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:16:53.326+0000 7f3c40a3ad80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T22:16:56.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:16:56.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:16:56.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:16:56.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:16:56.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:16:56.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:16:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:16:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:16:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2020-11-16T22:16:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2020-11-16T22:16:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'rbd' does not exist 2020-11-16T22:16:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2020-11-16T22:16:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-11-16T22:16:56.416 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'rbd' already exists 2020-11-16T22:16:56.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:16:56.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2020-11-16T22:16:56.416 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:16:56.416 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/666388,v1:127.0.0.1:6819/666388] [v2:127.0.0.1:6820/666388,v1:127.0.0.1:6821/666388] exists,up b5d5ab80-f398-47db-ba9d-15f1d2d65b7a 2020-11-16T22:16:57.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2020-11-16T22:16:57.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2020-11-16T22:16:57.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2020-11-16T22:16:57.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2020-11-16T22:16:57.446 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2020-11-16T22:16:57.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2020-11-16T22:16:57.603 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2020-11-16T22:16:57.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2020-11-16T22:16:57.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2020-11-16T22:16:57.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2020-11-16T22:16:57.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2020-11-16T22:16:57.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2020-11-16T22:16:57.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2020-11-16T22:16:57.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:16:57.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2020-11-16T22:16:58.144 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2020-11-16T22:16:58.144 INFO:tasks.workunit.client.0.smithi189.stdout:-1 0.29306 root default 2020-11-16T22:16:58.145 INFO:tasks.workunit.client.0.smithi189.stdout:-3 0.29306 host smithi189 2020-11-16T22:16:58.145 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:16:58.145 INFO:tasks.workunit.client.0.smithi189.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2020-11-16T22:16:58.145 INFO:tasks.workunit.client.0.smithi189.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2020-11-16T22:16:58.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2020-11-16T22:16:58.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2020-11-16T22:16:58.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2020-11-16T22:16:58.586 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:16:58.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2020-11-16T22:16:58.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2020-11-16T22:16:59.013 INFO:tasks.workunit.client.0.smithi189.stderr:marked down osd.0. 2020-11-16T22:16:59.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2020-11-16T22:17:09.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2020-11-16T22:17:09.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:17:09.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2020-11-16T22:17:09.565 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2020-11-16T22:17:09.566 INFO:tasks.workunit.client.0.smithi189.stdout:-1 0.29306 root default 2020-11-16T22:17:09.566 INFO:tasks.workunit.client.0.smithi189.stdout:-3 0.29306 host smithi189 2020-11-16T22:17:09.566 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:17:09.566 INFO:tasks.workunit.client.0.smithi189.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2020-11-16T22:17:09.566 INFO:tasks.workunit.client.0.smithi189.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2020-11-16T22:17:09.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2020-11-16T22:17:09.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2020-11-16T22:17:09.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2020-11-16T22:17:09.969 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:17:09.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2020-11-16T22:17:09.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2020-11-16T22:17:10.457 INFO:tasks.workunit.client.0.smithi189.stderr:marked down osd.0. 2020-11-16T22:17:10.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2020-11-16T22:17:20.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2020-11-16T22:17:20.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:17:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2020-11-16T22:17:21.048 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2020-11-16T22:17:21.049 INFO:tasks.workunit.client.0.smithi189.stdout:-1 0.29306 root default 2020-11-16T22:17:21.049 INFO:tasks.workunit.client.0.smithi189.stdout:-3 0.29306 host smithi189 2020-11-16T22:17:21.049 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:17:21.049 INFO:tasks.workunit.client.0.smithi189.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2020-11-16T22:17:21.049 INFO:tasks.workunit.client.0.smithi189.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2020-11-16T22:17:21.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2020-11-16T22:17:21.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2020-11-16T22:17:21.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2020-11-16T22:17:21.448 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:17:21.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2020-11-16T22:17:21.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2020-11-16T22:17:21.865 INFO:tasks.workunit.client.0.smithi189.stderr:marked down osd.0. 2020-11-16T22:17:21.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2020-11-16T22:17:31.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2020-11-16T22:17:46.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:17:47.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2020-11-16T22:17:47.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2020-11-16T22:17:47.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2020-11-16T22:17:47.440 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:17:47.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2020-11-16T22:17:47.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2020-11-16T22:17:47.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:17:47.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2020-11-16T22:17:47.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:17:47.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:17:47.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:17:47.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:17:47.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:17:47.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:17:47.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:17:47.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:17:47.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:17:47.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:17:47.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:17:47.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:17:47.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:17:47.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:17:47.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:17:47.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:17:47.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:17:47.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:17:47.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2020-11-16T22:17:47.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:17:47.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:17:47.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:17:47.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.663547 2020-11-16T22:17:47.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:17:47.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:17:47.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2020-11-16T22:17:47.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2020-11-16T22:17:47.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2020-11-16T22:17:47.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2020-11-16T22:17:47.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2020-11-16T22:17:47.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:17:47.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2020-11-16T22:17:47.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:17:47.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:17:47.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:17:47.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:17:47.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:17:47.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:17:47.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:17:47.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:17:47.588 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:17:47.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:17:47.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:17:47.590 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:17:47.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:17:47.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:17:47.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:17:47.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:17:47.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:17:47.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:17:47.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2020-11-16T22:17:47.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:17:47.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:17:47.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:17:47.598 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.663547 2020-11-16T22:17:47.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:17:47.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:17:47.599 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2020-11-16T22:17:47.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:17:47.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:17:47.601 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:17:47.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.663547 2020-11-16T22:17:47.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:17:47.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T22:17:47.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:17:47.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2020-11-16T22:17:47.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2020-11-16T22:17:47.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2020-11-16T22:17:47.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2020-11-16T22:17:47.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2020-11-16T22:17:47.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:17:47.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:17:47.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:17:47.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2020-11-16T22:17:47.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2020-11-16T22:17:47.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:17:47.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:17:47.911 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:17:47.912 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:17:47.912 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:17:47.912 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:17:47.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:17:47.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:17:48.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:17:48.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:17:48.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:17:48.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:17:48.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:17:48.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:17:48.117 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:17:48.117 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:17:48.117 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:17:48.118 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:17:48.118 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:17:48.118 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:17:48.118 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.663547/ceph-mon.a.asok 2020-11-16T22:17:48.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:17:48.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.663547/ceph-mon.a.asok config get fsid 2020-11-16T22:17:48.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:17:48.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:17:48.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:17:48.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:17:48.224 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:17:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:17:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:17:48.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:17:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:17:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:17:48.225 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:17:48.226 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.663547/ceph-mon.a.asok 2020-11-16T22:17:48.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:17:48.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.663547/ceph-mon.a.asok config get mon_host 2020-11-16T22:17:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2020-11-16T22:17:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-markdown 2020-11-16T22:17:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:17:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:17:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:17:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-markdown/x 2020-11-16T22:17:48.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:17:48.854 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:17:48.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:17:48.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:17:48.855 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:17:48.856 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:17:48.856 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:17:48.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:17:48.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:17:48.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:17:48.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2020-11-16T22:17:48.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:17:48.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:17:48.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:17:48.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:17:48.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/0 2020-11-16T22:17:48.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:17:48.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:17:48.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:17:48.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:17:48.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2020-11-16T22:17:48.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2020-11-16T22:17:48.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:17:48.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:17:48.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:17:48.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:17:48.883 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:17:48.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:17:48.884 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:17:48.884 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:17:48.884 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:17:48.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:17:48.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:17:48.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:17:48.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:17:48.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:17:48.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:17:48.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:17:48.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:17:48.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:17:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:17:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:17:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:17:48.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/0 2020-11-16T22:17:48.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:17:48.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4cf503e0-2378-4bfe-bed0-e506cf152aea 2020-11-16T22:17:48.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 4cf503e0-2378-4bfe-bed0-e506cf152aea' 2020-11-16T22:17:48.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:17:48.887 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 4cf503e0-2378-4bfe-bed0-e506cf152aea 2020-11-16T22:17:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCM+rJf051YNhAAoKbOauQ4j5mI9rkj2zgIbg== 2020-11-16T22:17:48.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCM+rJf051YNhAAoKbOauQ4j5mI9rkj2zgIbg=="}' 2020-11-16T22:17:48.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4cf503e0-2378-4bfe-bed0-e506cf152aea -i td/osd-markdown/0/new.json 2020-11-16T22:17:49.248 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:17:49.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/0/new.json 2020-11-16T22:17:49.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCM+rJf051YNhAAoKbOauQ4j5mI9rkj2zgIbg== --osd-uuid 4cf503e0-2378-4bfe-bed0-e506cf152aea 2020-11-16T22:17:49.288 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:49.297+0000 7fbeb0f93d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:17:49.291 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:49.297+0000 7fbeb0f93d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:17:49.292 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:49.301+0000 7fbeb0f93d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:17:49.293 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:49.301+0000 7fbeb0f93d80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:17:49.294 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:49.301+0000 7fbeb0f93d80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:17:49.294 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:49.301+0000 7fbeb0f93d80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:17:49.344 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:49.353+0000 7fbeb0f93d80 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2020-11-16T22:17:53.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/0/keyring 2020-11-16T22:17:53.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:17:53.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:17:53.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:17:53.716 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:17:54.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:17:54.203 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:17:54.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:17:54.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:17:54.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:17:54.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:17:54.221 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:54.229+0000 7fc87f763d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:17:54.223 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:54.233+0000 7fc87f763d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:17:54.225 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:54.233+0000 7fc87f763d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:17:54.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:17:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:17:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:17:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:17:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:17:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:17:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:17:54.581 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:17:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:17:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:17:54.765 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:54.773+0000 7fc87f763d80 -1 Falling back to public interface 2020-11-16T22:17:54.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:17:55.630 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:55.637+0000 7fc87f763d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:17:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:17:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:17:55.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:17:55.941 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:17:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:17:55.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:17:56.330 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/668118,v1:127.0.0.1:6803/668118] [v2:127.0.0.1:6804/668118,v1:127.0.0.1:6805/668118] exists,up 4cf503e0-2378-4bfe-bed0-e506cf152aea 2020-11-16T22:17:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:17:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:17:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:17:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2020-11-16T22:17:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:17:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:17:56.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:17:56.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:17:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/1 2020-11-16T22:17:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:17:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:17:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:17:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:17:56.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2020-11-16T22:17:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2020-11-16T22:17:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:17:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:17:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:17:56.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:17:56.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:17:56.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:17:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:17:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:17:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:17:56.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:17:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:17:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:17:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:17:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:17:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:17:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:17:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:17:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:17:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:17:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:17:56.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:17:56.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/1 2020-11-16T22:17:56.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:17:56.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a9b40665-718f-43ec-a201-a4320bc5fe8a 2020-11-16T22:17:56.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 a9b40665-718f-43ec-a201-a4320bc5fe8a' 2020-11-16T22:17:56.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:17:56.339 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 a9b40665-718f-43ec-a201-a4320bc5fe8a 2020-11-16T22:17:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCU+rJfhyZTFRAAxk/U+ieDzPheM3WmghsG5w== 2020-11-16T22:17:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCU+rJfhyZTFRAAxk/U+ieDzPheM3WmghsG5w=="}' 2020-11-16T22:17:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a9b40665-718f-43ec-a201-a4320bc5fe8a -i td/osd-markdown/1/new.json 2020-11-16T22:17:56.887 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:17:56.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/1/new.json 2020-11-16T22:17:56.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCU+rJfhyZTFRAAxk/U+ieDzPheM3WmghsG5w== --osd-uuid a9b40665-718f-43ec-a201-a4320bc5fe8a 2020-11-16T22:17:56.933 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:56.941+0000 7fa9460b5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:17:56.943 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:56.949+0000 7fa9460b5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:17:56.944 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:56.953+0000 7fa9460b5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:17:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:56.953+0000 7fa9460b5d80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:17:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:56.953+0000 7fa9460b5d80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:17:56.945 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:56.953+0000 7fa9460b5d80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:17:57.035 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:17:57.041+0000 7fa9460b5d80 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2020-11-16T22:18:01.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/1/keyring 2020-11-16T22:18:01.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:18:01.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:18:01.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:18:01.691 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:18:02.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:18:02.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:18:02.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:18:02.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:18:02.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:18:02.185 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:18:02.211 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:02.217+0000 7fd37747bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:18:02.213 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:02.221+0000 7fd37747bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:18:02.214 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:02.221+0000 7fd37747bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:18:02.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:18:02.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:18:02.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:18:02.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:18:02.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:18:02.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:18:02.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:18:02.566 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:18:02.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:18:02.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:18:02.754 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:02.761+0000 7fd37747bd80 -1 Falling back to public interface 2020-11-16T22:18:02.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:18:03.604 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:03.605+0000 7fd37747bd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:18:03.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:18:03.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:18:03.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:18:03.944 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:18:03.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:18:03.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:18:04.327 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/669112,v1:127.0.0.1:6811/669112] [v2:127.0.0.1:6812/669112,v1:127.0.0.1:6813/669112] exists,up a9b40665-718f-43ec-a201-a4320bc5fe8a 2020-11-16T22:18:04.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:18:04.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:18:04.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:18:04.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2020-11-16T22:18:04.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:18:04.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:18:04.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T22:18:04.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:18:04.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/2 2020-11-16T22:18:04.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:18:04.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:18:04.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:18:04.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:18:04.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2020-11-16T22:18:04.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2020-11-16T22:18:04.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:18:04.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:18:04.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:18:04.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:18:04.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:18:04.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:18:04.331 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:18:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:18:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:18:04.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:18:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:18:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:18:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:18:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:18:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:18:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:18:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:18:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:18:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:18:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:18:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:18:04.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/2 2020-11-16T22:18:04.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:18:04.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=45d40347-ab36-4632-a58e-f8cbc3f72f24 2020-11-16T22:18:04.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 45d40347-ab36-4632-a58e-f8cbc3f72f24' 2020-11-16T22:18:04.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:18:04.335 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 45d40347-ab36-4632-a58e-f8cbc3f72f24 2020-11-16T22:18:04.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCc+rJfSKpjFRAAWC8Z+pYqZpR7wBuOnExQeQ== 2020-11-16T22:18:04.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCc+rJfSKpjFRAAWC8Z+pYqZpR7wBuOnExQeQ=="}' 2020-11-16T22:18:04.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 45d40347-ab36-4632-a58e-f8cbc3f72f24 -i td/osd-markdown/2/new.json 2020-11-16T22:18:04.894 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:18:04.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/2/new.json 2020-11-16T22:18:04.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCc+rJfSKpjFRAAWC8Z+pYqZpR7wBuOnExQeQ== --osd-uuid 45d40347-ab36-4632-a58e-f8cbc3f72f24 2020-11-16T22:18:04.935 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:04.941+0000 7ff5b746ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:18:04.938 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:04.945+0000 7ff5b746ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:18:04.939 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:04.949+0000 7ff5b746ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:18:04.940 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:04.949+0000 7ff5b746ad80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:18:04.940 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:04.949+0000 7ff5b746ad80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:18:04.940 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:04.949+0000 7ff5b746ad80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:18:04.990 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:04.997+0000 7ff5b746ad80 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2020-11-16T22:18:09.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/2/keyring 2020-11-16T22:18:09.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:18:09.368 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T22:18:09.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T22:18:09.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:18:09.871 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:18:09.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T22:18:09.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:18:09.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:18:09.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:18:09.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:18:09.898 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:09.904+0000 7f3699798d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:18:09.900 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:09.908+0000 7f3699798d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:18:09.902 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:09.908+0000 7f3699798d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:18:10.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T22:18:10.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:18:10.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:18:10.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:18:10.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:18:10.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:18:10.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:18:10.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:18:10.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:18:10.249 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:18:10.446 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:10.452+0000 7f3699798d80 -1 Falling back to public interface 2020-11-16T22:18:10.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:18:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:18:11.092+0000 7f3699798d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T22:18:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:18:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:18:11.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:18:11.661 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:18:11.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:18:11.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:18:12.071 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/670107,v1:127.0.0.1:6819/670107] [v2:127.0.0.1:6820/670107,v1:127.0.0.1:6821/670107] exists,up 45d40347-ab36-4632-a58e-f8cbc3f72f24 2020-11-16T22:18:12.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:18:12.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:18:12.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:18:12.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2020-11-16T22:18:12.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2020-11-16T22:18:12.471 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'rbd' does not exist 2020-11-16T22:18:12.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2020-11-16T22:18:12.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-11-16T22:18:13.043 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'rbd' already exists 2020-11-16T22:18:13.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:18:14.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2020-11-16T22:18:15.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2020-11-16T22:18:15.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2020-11-16T22:18:15.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2020-11-16T22:18:15.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2020-11-16T22:18:15.313 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2020-11-16T22:18:15.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2020-11-16T22:18:15.473 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2020-11-16T22:18:15.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2020-11-16T22:18:15.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2020-11-16T22:18:15.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2020-11-16T22:18:15.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2020-11-16T22:18:15.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2020-11-16T22:18:15.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2020-11-16T22:18:15.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:18:15.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2020-11-16T22:18:16.019 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2020-11-16T22:18:16.019 INFO:tasks.workunit.client.0.smithi189.stdout:-1 0.29306 root default 2020-11-16T22:18:16.019 INFO:tasks.workunit.client.0.smithi189.stdout:-3 0.29306 host smithi189 2020-11-16T22:18:16.019 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:18:16.020 INFO:tasks.workunit.client.0.smithi189.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2020-11-16T22:18:16.020 INFO:tasks.workunit.client.0.smithi189.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2020-11-16T22:18:16.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2020-11-16T22:18:16.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2020-11-16T22:18:16.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2020-11-16T22:18:16.420 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:18:16.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2020-11-16T22:18:16.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2020-11-16T22:18:16.840 INFO:tasks.workunit.client.0.smithi189.stderr:marked down osd.0. 2020-11-16T22:18:16.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2020-11-16T22:18:26.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2020-11-16T22:18:26.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:18:27.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2020-11-16T22:18:27.391 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2020-11-16T22:18:27.391 INFO:tasks.workunit.client.0.smithi189.stdout:-1 0.29306 root default 2020-11-16T22:18:27.391 INFO:tasks.workunit.client.0.smithi189.stdout:-3 0.29306 host smithi189 2020-11-16T22:18:27.391 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:18:27.391 INFO:tasks.workunit.client.0.smithi189.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2020-11-16T22:18:27.391 INFO:tasks.workunit.client.0.smithi189.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2020-11-16T22:18:27.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2020-11-16T22:18:27.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2020-11-16T22:18:27.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2020-11-16T22:18:27.799 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:18:27.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2020-11-16T22:18:27.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2020-11-16T22:18:28.200 INFO:tasks.workunit.client.0.smithi189.stderr:marked down osd.0. 2020-11-16T22:18:28.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2020-11-16T22:18:38.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2020-11-16T22:18:38.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:18:38.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2020-11-16T22:18:38.753 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2020-11-16T22:18:38.754 INFO:tasks.workunit.client.0.smithi189.stdout:-1 0.29306 root default 2020-11-16T22:18:38.754 INFO:tasks.workunit.client.0.smithi189.stdout:-3 0.29306 host smithi189 2020-11-16T22:18:38.754 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:18:38.754 INFO:tasks.workunit.client.0.smithi189.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2020-11-16T22:18:38.754 INFO:tasks.workunit.client.0.smithi189.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2020-11-16T22:18:38.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2020-11-16T22:18:38.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2020-11-16T22:18:38.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2020-11-16T22:18:39.158 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:18:39.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2020-11-16T22:18:39.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2020-11-16T22:18:39.551 INFO:tasks.workunit.client.0.smithi189.stderr:marked down osd.0. 2020-11-16T22:18:39.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2020-11-16T22:18:49.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2020-11-16T22:18:49.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:18:49.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2020-11-16T22:18:50.111 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2020-11-16T22:18:50.111 INFO:tasks.workunit.client.0.smithi189.stdout:-1 0.29306 root default 2020-11-16T22:18:50.111 INFO:tasks.workunit.client.0.smithi189.stdout:-3 0.29306 host smithi189 2020-11-16T22:18:50.111 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:18:50.112 INFO:tasks.workunit.client.0.smithi189.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2020-11-16T22:18:50.112 INFO:tasks.workunit.client.0.smithi189.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2020-11-16T22:18:50.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2020-11-16T22:18:50.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2020-11-16T22:18:50.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2020-11-16T22:18:50.526 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:18:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2020-11-16T22:18:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2020-11-16T22:18:50.959 INFO:tasks.workunit.client.0.smithi189.stderr:marked down osd.0. 2020-11-16T22:18:50.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2020-11-16T22:19:00.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2020-11-16T22:19:15.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:19:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2020-11-16T22:19:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2020-11-16T22:19:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2020-11-16T22:19:16.519 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:19:16.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2020-11-16T22:19:16.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2020-11-16T22:19:16.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:19:16.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2020-11-16T22:19:16.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:19:16.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:19:16.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:19:16.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:19:16.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:19:16.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:19:16.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:19:16.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:19:16.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:19:16.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:19:16.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:19:16.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:19:16.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:19:16.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:19:16.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:19:16.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:19:16.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:19:16.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:19:16.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2020-11-16T22:19:16.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:19:16.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:19:16.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:19:16.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.663547 2020-11-16T22:19:16.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:19:16.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:19:16.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2020-11-16T22:19:16.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2020-11-16T22:19:16.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2020-11-16T22:19:16.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2020-11-16T22:19:16.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2020-11-16T22:19:16.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:19:16.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2020-11-16T22:19:16.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:19:16.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:19:16.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:19:16.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:19:16.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:19:16.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:19:16.668 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:19:16.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:19:16.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:19:16.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:19:16.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:19:16.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:19:16.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:19:16.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:19:16.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:19:16.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:19:16.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:19:16.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:19:16.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2020-11-16T22:19:16.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:19:16.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:19:16.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:19:16.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.663547 2020-11-16T22:19:16.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:19:16.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:19:16.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2020-11-16T22:19:16.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:19:16.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:19:16.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:19:16.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.663547 2020-11-16T22:19:16.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:19:16.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T22:19:16.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:19:16.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2020-11-16T22:19:16.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2020-11-16T22:19:16.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2020-11-16T22:19:16.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2020-11-16T22:19:16.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2020-11-16T22:19:16.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:19:16.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:19:16.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:19:16.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2020-11-16T22:19:16.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2020-11-16T22:19:16.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:19:16.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:19:16.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:19:16.988 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:19:16.988 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:19:16.988 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:19:16.988 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:19:16.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:19:17.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:19:17.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:19:17.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:19:17.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:19:17.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:19:17.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:19:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:19:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:19:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:19:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:19:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:19:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:19:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.663547/ceph-mon.a.asok 2020-11-16T22:19:17.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:19:17.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.663547/ceph-mon.a.asok config get fsid 2020-11-16T22:19:17.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:19:17.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:19:17.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:19:17.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:19:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:19:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:19:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:19:17.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:19:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:19:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:19:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:19:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.663547/ceph-mon.a.asok 2020-11-16T22:19:17.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:19:17.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.663547/ceph-mon.a.asok config get mon_host 2020-11-16T22:19:17.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2020-11-16T22:19:17.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-markdown 2020-11-16T22:19:17.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:19:17.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:19:17.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:19:17.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-markdown/x 2020-11-16T22:19:17.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:19:17.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:19:17.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:19:17.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:19:17.940 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:19:17.940 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:19:17.940 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:19:17.940 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:19:17.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:19:17.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:19:17.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2020-11-16T22:19:17.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:19:17.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:19:17.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:19:17.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:19:17.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/0 2020-11-16T22:19:17.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:19:17.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:19:17.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:19:17.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:19:17.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2020-11-16T22:19:17.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2020-11-16T22:19:17.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:19:17.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:19:17.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:19:17.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:19:17.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:19:17.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:19:17.971 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:19:17.971 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:19:17.971 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:19:17.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:19:17.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:19:17.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:19:17.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:19:17.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:19:17.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:19:17.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:19:17.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:19:17.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:19:17.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:19:17.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:19:17.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:19:17.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/0 2020-11-16T22:19:17.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:19:17.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=42d1d0d0-3e12-4ec4-af08-991431a8f29e 2020-11-16T22:19:17.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 42d1d0d0-3e12-4ec4-af08-991431a8f29e' 2020-11-16T22:19:17.974 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 42d1d0d0-3e12-4ec4-af08-991431a8f29e 2020-11-16T22:19:17.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:19:17.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDm+rJfpA0DABAAr04e94ysGBTe80QrFu9wfw== 2020-11-16T22:19:17.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDm+rJfpA0DABAAr04e94ysGBTe80QrFu9wfw=="}' 2020-11-16T22:19:17.988 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 42d1d0d0-3e12-4ec4-af08-991431a8f29e -i td/osd-markdown/0/new.json 2020-11-16T22:19:18.348 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:19:18.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/0/new.json 2020-11-16T22:19:18.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDm+rJfpA0DABAAr04e94ysGBTe80QrFu9wfw== --osd-uuid 42d1d0d0-3e12-4ec4-af08-991431a8f29e 2020-11-16T22:19:18.387 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:18.395+0000 7f1a5c08ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:19:18.389 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:18.399+0000 7f1a5c08ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:19:18.391 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:18.399+0000 7f1a5c08ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:19:18.391 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:18.403+0000 7f1a5c08ed80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:19:18.392 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:18.403+0000 7f1a5c08ed80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:19:18.392 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:18.403+0000 7f1a5c08ed80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:19:18.433 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:18.443+0000 7f1a5c08ed80 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2020-11-16T22:19:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/0/keyring 2020-11-16T22:20:30.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:20:30.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:20:30.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:20:30.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:20:30.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:20:30.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:20:30.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:20:30.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:20:30.520 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:23.263+0000 7efff2475d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.520 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:23.263+0000 7efff2475d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.520 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:23.267+0000 7efff2475d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:20:30.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:20:30.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:20:30.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:20:30.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:20:30.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:30.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:20:30.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:30.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:20:30.522 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:23.799+0000 7efff2475d80 -1 Falling back to public interface 2020-11-16T22:20:30.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:20:30.522 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:24.659+0000 7efff2475d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:20:30.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:20:30.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:30.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:20:30.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:30.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:20:30.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:20:30.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:20:30.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:20:30.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2020-11-16T22:20:30.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:20:30.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:20:30.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:20:30.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:20:30.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/1 2020-11-16T22:20:30.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:20:30.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:20:30.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:20:30.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:20:30.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2020-11-16T22:20:30.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2020-11-16T22:20:30.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:20:30.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:20:30.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:20:30.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:20:30.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:20:30.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:20:30.526 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:20:30.526 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:30.526 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:30.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:30.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:30.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:20:30.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:20:30.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:20:30.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:20:30.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:20:30.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:20:30.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:20:30.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:20:30.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:20:30.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:20:30.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/1 2020-11-16T22:20:30.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:20:30.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3d6feb5d-0a0b-475e-9bd8-dec1dcf63397 2020-11-16T22:20:30.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 3d6feb5d-0a0b-475e-9bd8-dec1dcf63397' 2020-11-16T22:20:30.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:20:30.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDt+rJfDLZrGRAAQYwoUQCBtkgneeKtjrLQ7Q== 2020-11-16T22:20:30.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDt+rJfDLZrGRAAQYwoUQCBtkgneeKtjrLQ7Q=="}' 2020-11-16T22:20:30.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3d6feb5d-0a0b-475e-9bd8-dec1dcf63397 -i td/osd-markdown/1/new.json 2020-11-16T22:20:30.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/1/new.json 2020-11-16T22:20:30.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDt+rJfDLZrGRAAQYwoUQCBtkgneeKtjrLQ7Q== --osd-uuid 3d6feb5d-0a0b-475e-9bd8-dec1dcf63397 2020-11-16T22:20:30.529 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:26.003+0000 7ff89495bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.529 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:26.003+0000 7ff89495bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.529 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:26.007+0000 7ff89495bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.530 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:26.007+0000 7ff89495bd80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:20:30.530 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:26.007+0000 7ff89495bd80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:20:30.530 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:26.007+0000 7ff89495bd80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:20:30.530 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:26.075+0000 7ff89495bd80 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2020-11-16T22:20:30.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/1/keyring 2020-11-16T22:20:30.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:20:30.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:20:30.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:20:30.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:20:30.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:20:30.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:20:30.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:20:30.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:20:30.531 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:31.255+0000 7faaffcb9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.532 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:31.259+0000 7faaffcb9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.532 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:31.259+0000 7faaffcb9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:20:30.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:20:30.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:20:30.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:20:30.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:20:30.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:30.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:20:30.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:30.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:20:30.533 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:31.795+0000 7faaffcb9d80 -1 Falling back to public interface 2020-11-16T22:20:30.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:20:30.533 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:32.651+0000 7faaffcb9d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:20:30.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:20:30.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:30.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:20:30.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:30.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:20:30.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:20:30.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:20:30.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:20:30.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2020-11-16T22:20:30.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:20:30.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:20:30.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T22:20:30.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:20:30.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/2 2020-11-16T22:20:30.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:20:30.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:20:30.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:20:30.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:20:30.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2020-11-16T22:20:30.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2020-11-16T22:20:30.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:20:30.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:20:30.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:20:30.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:20:30.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:20:30.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:20:30.537 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:20:30.537 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:30.537 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:30.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:30.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:30.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:20:30.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:20:30.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:20:30.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:20:30.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:20:30.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:20:30.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:20:30.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:20:30.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:20:30.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:20:30.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/2 2020-11-16T22:20:30.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:20:30.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=cb394e65-65fd-4322-a7e3-0d6d77565e6d 2020-11-16T22:20:30.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 cb394e65-65fd-4322-a7e3-0d6d77565e6d' 2020-11-16T22:20:30.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:20:30.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD1+rJf4srQFxAAd8zGAEaiZZUhPDgH+E2siw== 2020-11-16T22:20:30.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD1+rJf4srQFxAAd8zGAEaiZZUhPDgH+E2siw=="}' 2020-11-16T22:20:30.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new cb394e65-65fd-4322-a7e3-0d6d77565e6d -i td/osd-markdown/2/new.json 2020-11-16T22:20:30.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/2/new.json 2020-11-16T22:20:30.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD1+rJf4srQFxAAd8zGAEaiZZUhPDgH+E2siw== --osd-uuid cb394e65-65fd-4322-a7e3-0d6d77565e6d 2020-11-16T22:20:30.540 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:34.231+0000 7f19bd7ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.540 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:34.235+0000 7f19bd7ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:34.235+0000 7f19bd7ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:34.235+0000 7f19bd7ccd80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:20:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:34.235+0000 7f19bd7ccd80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:20:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:34.235+0000 7f19bd7ccd80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:20:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:34.275+0000 7f19bd7ccd80 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2020-11-16T22:20:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/2/keyring 2020-11-16T22:20:30.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:20:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T22:20:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:20:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T22:20:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:20:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:20:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:20:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:20:30.542 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:39.447+0000 7f2afd3b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.543 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:39.451+0000 7f2afd3b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.543 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:39.451+0000 7f2afd3b1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:30.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T22:20:30.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:20:30.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:20:30.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:20:30.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:20:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:20:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:20:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:39.983+0000 7f2afd3b1d80 -1 Falling back to public interface 2020-11-16T22:20:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:20:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:19:40.803+0000 7f2afd3b1d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T22:20:30.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:20:30.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:30.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:20:30.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:30.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:20:30.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:20:30.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:20:30.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:20:30.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2020-11-16T22:20:30.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2020-11-16T22:20:30.546 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'rbd' does not exist 2020-11-16T22:20:30.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2020-11-16T22:20:30.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-11-16T22:20:30.546 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'rbd' already exists 2020-11-16T22:20:30.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:20:30.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2020-11-16T22:20:30.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2020-11-16T22:20:30.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2020-11-16T22:20:30.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2020-11-16T22:20:30.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2020-11-16T22:20:30.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2020-11-16T22:20:30.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2020-11-16T22:20:30.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2020-11-16T22:20:30.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2020-11-16T22:20:30.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2020-11-16T22:20:30.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2020-11-16T22:20:30.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2020-11-16T22:20:30.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:20:30.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2020-11-16T22:20:30.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2020-11-16T22:20:30.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2020-11-16T22:20:30.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2020-11-16T22:20:30.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2020-11-16T22:20:30.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2020-11-16T22:20:30.549 INFO:tasks.workunit.client.0.smithi189.stderr:marked down osd.0. 2020-11-16T22:20:30.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2020-11-16T22:20:30.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2020-11-16T22:20:30.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:20:30.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2020-11-16T22:20:30.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2020-11-16T22:20:30.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2020-11-16T22:20:30.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2020-11-16T22:20:30.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2020-11-16T22:20:30.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2020-11-16T22:20:30.550 INFO:tasks.workunit.client.0.smithi189.stderr:marked down osd.0. 2020-11-16T22:20:30.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2020-11-16T22:20:30.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2020-11-16T22:20:30.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:20:30.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2020-11-16T22:20:30.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2020-11-16T22:20:30.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2020-11-16T22:20:30.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2020-11-16T22:20:30.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2020-11-16T22:20:30.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2020-11-16T22:20:30.552 INFO:tasks.workunit.client.0.smithi189.stderr:marked down osd.0. 2020-11-16T22:20:30.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2020-11-16T22:20:30.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2020-11-16T22:20:30.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2020-11-16T22:20:30.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2020-11-16T22:20:30.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2020-11-16T22:20:30.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2020-11-16T22:20:30.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2020-11-16T22:20:30.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2020-11-16T22:20:30.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2020-11-16T22:20:30.553 INFO:tasks.workunit.client.0.smithi189.stderr:marked down osd.0. 2020-11-16T22:20:30.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2020-11-16T22:20:30.553 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:20:30.553 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:20:30.554 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:20:30.554 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:20:30.554 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/671936,v1:127.0.0.1:6803/671936] [v2:127.0.0.1:6804/671936,v1:127.0.0.1:6805/671936] exists,up 42d1d0d0-3e12-4ec4-af08-991431a8f29e 2020-11-16T22:20:30.554 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 3d6feb5d-0a0b-475e-9bd8-dec1dcf63397 2020-11-16T22:20:30.554 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:20:30.554 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:20:30.554 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:20:30.554 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:20:30.555 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:20:30.555 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/672930,v1:127.0.0.1:6811/672930] [v2:127.0.0.1:6812/672930,v1:127.0.0.1:6813/672930] exists,up 3d6feb5d-0a0b-475e-9bd8-dec1dcf63397 2020-11-16T22:20:30.555 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 cb394e65-65fd-4322-a7e3-0d6d77565e6d 2020-11-16T22:20:30.555 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:20:30.555 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T22:20:30.555 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:20:30.555 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:20:30.556 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:20:30.556 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/673924,v1:127.0.0.1:6819/673924] [v2:127.0.0.1:6820/673924,v1:127.0.0.1:6821/673924] exists,up cb394e65-65fd-4322-a7e3-0d6d77565e6d 2020-11-16T22:20:30.556 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2020-11-16T22:20:30.556 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2020-11-16T22:20:30.556 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2020-11-16T22:20:30.556 INFO:tasks.workunit.client.0.smithi189.stdout:-1 0.29306 root default 2020-11-16T22:20:30.556 INFO:tasks.workunit.client.0.smithi189.stdout:-3 0.29306 host smithi189 2020-11-16T22:20:30.556 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:20:30.557 INFO:tasks.workunit.client.0.smithi189.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2020-11-16T22:20:30.557 INFO:tasks.workunit.client.0.smithi189.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2020-11-16T22:20:30.557 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:20:30.557 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2020-11-16T22:20:30.557 INFO:tasks.workunit.client.0.smithi189.stdout:-1 0.29306 root default 2020-11-16T22:20:30.557 INFO:tasks.workunit.client.0.smithi189.stdout:-3 0.29306 host smithi189 2020-11-16T22:20:30.557 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:20:30.558 INFO:tasks.workunit.client.0.smithi189.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2020-11-16T22:20:30.558 INFO:tasks.workunit.client.0.smithi189.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2020-11-16T22:20:30.558 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:20:30.558 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2020-11-16T22:20:30.558 INFO:tasks.workunit.client.0.smithi189.stdout:-1 0.29306 root default 2020-11-16T22:20:30.558 INFO:tasks.workunit.client.0.smithi189.stdout:-3 0.29306 host smithi189 2020-11-16T22:20:30.558 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:20:30.558 INFO:tasks.workunit.client.0.smithi189.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2020-11-16T22:20:30.559 INFO:tasks.workunit.client.0.smithi189.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2020-11-16T22:20:30.559 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:20:30.559 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2020-11-16T22:20:30.559 INFO:tasks.workunit.client.0.smithi189.stdout:-1 0.29306 root default 2020-11-16T22:20:30.559 INFO:tasks.workunit.client.0.smithi189.stdout:-3 0.29306 host smithi189 2020-11-16T22:20:30.559 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:20:30.559 INFO:tasks.workunit.client.0.smithi189.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2020-11-16T22:20:30.559 INFO:tasks.workunit.client.0.smithi189.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2020-11-16T22:20:30.560 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:20:30.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2020-11-16T22:20:30.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2020-11-16T22:20:30.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2020-11-16T22:20:31.362 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2020-11-16T22:20:31.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2020-11-16T22:20:31.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2020-11-16T22:20:31.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:20:31.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2020-11-16T22:20:31.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:20:31.363 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:20:31.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:20:31.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:20:31.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:20:31.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:20:31.473 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:20:31.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:20:31.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:20:31.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:20:31.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:20:31.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:20:31.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:20:31.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:20:31.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:20:31.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:20:31.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:20:31.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:20:31.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2020-11-16T22:20:31.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:20:31.496 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:31.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:31.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.663547 2020-11-16T22:20:31.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:20:31.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:20:31.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2020-11-16T22:20:31.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2020-11-16T22:20:31.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2020-11-16T22:20:31.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2020-11-16T22:20:31.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2020-11-16T22:20:31.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:20:31.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2020-11-16T22:20:31.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:20:31.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:20:31.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:20:31.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:20:31.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:20:31.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:20:31.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:20:31.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:20:31.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:20:31.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:20:31.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:20:31.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:20:31.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:20:31.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:20:31.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:20:31.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:20:31.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:20:31.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:20:31.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2020-11-16T22:20:31.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:20:31.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:31.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:31.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.663547 2020-11-16T22:20:31.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:20:31.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:20:31.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2020-11-16T22:20:31.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:20:31.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:31.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:31.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.663547 2020-11-16T22:20:31.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:20:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T22:20:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:20:31.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2020-11-16T22:20:31.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2020-11-16T22:20:31.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2020-11-16T22:20:31.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2020-11-16T22:20:31.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2020-11-16T22:20:31.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:20:31.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:20:31.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:20:31.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2020-11-16T22:20:31.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2020-11-16T22:20:31.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:20:31.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:20:31.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:20:31.830 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:20:31.830 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:31.830 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:31.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:31.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:20:32.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:20:32.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:20:32.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:20:32.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:20:32.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:20:32.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:20:32.027 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:20:32.027 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:20:32.027 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:20:32.027 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:20:32.028 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:32.028 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:32.028 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.663547/ceph-mon.a.asok 2020-11-16T22:20:32.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:20:32.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.663547/ceph-mon.a.asok config get fsid 2020-11-16T22:20:32.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:20:32.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:20:32.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:20:32.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:20:32.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:20:32.138 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:20:32.139 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:20:32.139 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:20:32.139 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:20:32.139 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:32.139 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:32.140 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.663547/ceph-mon.a.asok 2020-11-16T22:20:32.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:20:32.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.663547/ceph-mon.a.asok config get mon_host 2020-11-16T22:20:32.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2020-11-16T22:20:32.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-markdown 2020-11-16T22:20:32.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:20:32.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:20:32.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:20:32.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-markdown/x 2020-11-16T22:20:32.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:20:32.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:20:32.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:20:32.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:20:32.784 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:20:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:32.785 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:32.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:32.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:20:32.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:20:32.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2020-11-16T22:20:32.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:20:32.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:20:32.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:20:32.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:20:32.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/0 2020-11-16T22:20:32.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:20:32.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:20:32.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:20:32.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:20:32.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2020-11-16T22:20:32.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2020-11-16T22:20:32.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:20:32.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:20:32.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:20:32.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:20:32.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:20:32.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:20:32.814 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:20:32.814 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:32.814 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:32.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:32.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:32.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:20:32.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:20:32.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:20:32.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:20:32.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:20:32.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:20:32.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:20:32.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:20:32.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:20:32.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:20:32.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/0 2020-11-16T22:20:32.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:20:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1413e36e-eecc-4d52-921c-3e00c872df67 2020-11-16T22:20:32.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 1413e36e-eecc-4d52-921c-3e00c872df67' 2020-11-16T22:20:32.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:20:32.817 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 1413e36e-eecc-4d52-921c-3e00c872df67 2020-11-16T22:20:32.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAw+7JfxF42MhAAFVBT/WRgmrwVt8dHPLpd5Q== 2020-11-16T22:20:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAw+7JfxF42MhAAFVBT/WRgmrwVt8dHPLpd5Q=="}' 2020-11-16T22:20:32.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1413e36e-eecc-4d52-921c-3e00c872df67 -i td/osd-markdown/0/new.json 2020-11-16T22:20:33.168 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:20:33.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/0/new.json 2020-11-16T22:20:33.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAw+7JfxF42MhAAFVBT/WRgmrwVt8dHPLpd5Q== --osd-uuid 1413e36e-eecc-4d52-921c-3e00c872df67 2020-11-16T22:20:33.208 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:33.218+0000 7f09c085bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:33.210 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:33.218+0000 7f09c085bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:33.212 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:33.222+0000 7f09c085bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:33.212 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:33.222+0000 7f09c085bd80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:20:33.213 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:33.222+0000 7f09c085bd80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:20:33.213 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:33.222+0000 7f09c085bd80 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2020-11-16T22:20:33.268 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:33.278+0000 7f09c085bd80 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2020-11-16T22:20:37.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/0/keyring 2020-11-16T22:20:37.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:20:37.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:20:37.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:20:37.634 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:20:38.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:20:38.127 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:20:38.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:20:38.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:20:38.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:20:38.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:20:38.155 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:38.162+0000 7fbc76260d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:38.158 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:38.166+0000 7fbc76260d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:38.159 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:38.166+0000 7fbc76260d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:38.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:20:38.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:20:38.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:20:38.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:20:38.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:20:38.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:38.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:20:38.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:38.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:20:38.519 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:20:38.701 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:38.710+0000 7fbc76260d80 -1 Falling back to public interface 2020-11-16T22:20:38.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:20:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:39.298+0000 7fbc76260d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:20:39.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:20:39.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:39.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:20:39.891 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:20:39.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:39.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:20:40.275 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/675724,v1:127.0.0.1:6803/675724] [v2:127.0.0.1:6804/675724,v1:127.0.0.1:6805/675724] exists,up 1413e36e-eecc-4d52-921c-3e00c872df67 2020-11-16T22:20:40.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:20:40.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:20:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:20:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2020-11-16T22:20:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:20:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:20:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:20:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:20:40.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/1 2020-11-16T22:20:40.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:20:40.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:20:40.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:20:40.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:20:40.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2020-11-16T22:20:40.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2020-11-16T22:20:40.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:20:40.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:20:40.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:20:40.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:20:40.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:20:40.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:20:40.280 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:20:40.280 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:40.280 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:40.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:40.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:40.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:20:40.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:20:40.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:20:40.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:20:40.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:20:40.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:20:40.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:20:40.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:20:40.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:20:40.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:20:40.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/1 2020-11-16T22:20:40.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:20:40.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fe0c1097-6422-48b4-8f72-d23b94813d04 2020-11-16T22:20:40.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 fe0c1097-6422-48b4-8f72-d23b94813d04' 2020-11-16T22:20:40.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:20:40.283 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 fe0c1097-6422-48b4-8f72-d23b94813d04 2020-11-16T22:20:40.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA4+7Jfl3GGEhAABQEeIFYdOY5jwb/HLTo8Iw== 2020-11-16T22:20:40.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA4+7Jfl3GGEhAABQEeIFYdOY5jwb/HLTo8Iw=="}' 2020-11-16T22:20:40.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fe0c1097-6422-48b4-8f72-d23b94813d04 -i td/osd-markdown/1/new.json 2020-11-16T22:20:40.816 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:20:40.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/1/new.json 2020-11-16T22:20:40.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA4+7Jfl3GGEhAABQEeIFYdOY5jwb/HLTo8Iw== --osd-uuid fe0c1097-6422-48b4-8f72-d23b94813d04 2020-11-16T22:20:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:40.866+0000 7f6de2af6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:40.870+0000 7f6de2af6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:40.861 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:40.870+0000 7f6de2af6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:40.861 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:40.870+0000 7f6de2af6d80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:20:40.861 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:40.870+0000 7f6de2af6d80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:20:40.862 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:40.870+0000 7f6de2af6d80 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2020-11-16T22:20:40.937 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:40.946+0000 7f6de2af6d80 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2020-11-16T22:20:45.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/1/keyring 2020-11-16T22:20:45.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:20:45.335 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:20:45.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:20:45.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:20:45.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:20:45.823 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:20:45.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:20:45.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:20:45.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:20:45.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:20:45.847 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:45.853+0000 7f9a3c0ddd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:45.849 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:45.857+0000 7f9a3c0ddd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:45.850 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:45.857+0000 7f9a3c0ddd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:46.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:20:46.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:20:46.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:20:46.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:20:46.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:20:46.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:46.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:20:46.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:46.205 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:20:46.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:20:46.406 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:46.413+0000 7f9a3c0ddd80 -1 Falling back to public interface 2020-11-16T22:20:46.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:20:47.353 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:47.353+0000 7f9a3c0ddd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:20:47.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:20:47.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:47.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:20:47.573 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:20:47.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:47.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:20:48.069 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/676718,v1:127.0.0.1:6811/676718] [v2:127.0.0.1:6812/676718,v1:127.0.0.1:6813/676718] exists,up fe0c1097-6422-48b4-8f72-d23b94813d04 2020-11-16T22:20:48.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:20:48.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:20:48.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:20:48.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2020-11-16T22:20:48.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2020-11-16T22:20:48.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:20:48.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T22:20:48.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:20:48.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/2 2020-11-16T22:20:48.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2020-11-16T22:20:48.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:20:48.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:20:48.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:20:48.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2020-11-16T22:20:48.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2020-11-16T22:20:48.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:20:48.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:20:48.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2020-11-16T22:20:48.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:20:48.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:20:48.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:20:48.074 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:20:48.074 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:20:48.074 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:20:48.074 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:48.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' 2020-11-16T22:20:48.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:20:48.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:20:48.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:20:48.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2020-11-16T22:20:48.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2020-11-16T22:20:48.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:20:48.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:20:48.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:20:48.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:20:48.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:20:48.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/2 2020-11-16T22:20:48.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:20:48.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a3c2949a-6826-49c7-a383-cef36b4dbb76 2020-11-16T22:20:48.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 a3c2949a-6826-49c7-a383-cef36b4dbb76' 2020-11-16T22:20:48.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:20:48.077 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 a3c2949a-6826-49c7-a383-cef36b4dbb76 2020-11-16T22:20:48.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBA+7Jfw/z+BRAAKNie54CFXIzzYinp1SB9ZQ== 2020-11-16T22:20:48.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBA+7Jfw/z+BRAAKNie54CFXIzzYinp1SB9ZQ=="}' 2020-11-16T22:20:48.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a3c2949a-6826-49c7-a383-cef36b4dbb76 -i td/osd-markdown/2/new.json 2020-11-16T22:20:48.695 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:20:48.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/2/new.json 2020-11-16T22:20:48.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBA+7Jfw/z+BRAAKNie54CFXIzzYinp1SB9ZQ== --osd-uuid a3c2949a-6826-49c7-a383-cef36b4dbb76 2020-11-16T22:20:48.741 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:48.749+0000 7f2bbd7f6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:48.744 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:48.753+0000 7f2bbd7f6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:48.745 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:48.753+0000 7f2bbd7f6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:48.746 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:48.753+0000 7f2bbd7f6d80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:20:48.747 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:48.753+0000 7f2bbd7f6d80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:20:48.747 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:48.753+0000 7f2bbd7f6d80 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2020-11-16T22:20:48.810 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:48.817+0000 7f2bbd7f6d80 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2020-11-16T22:20:53.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/2/keyring 2020-11-16T22:20:53.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:20:53.359 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T22:20:53.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T22:20:53.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:20:53.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T22:20:53.856 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:20:53.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:20:53.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=fb456687-fde5-4d24-833a-d6ea233b93e4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.663547/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:20:53.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:20:53.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:20:53.883 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:53.889+0000 7f8b3daf1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:53.885 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:53.893+0000 7f8b3daf1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:53.886 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:53.893+0000 7f8b3daf1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:20:54.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T22:20:54.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:20:54.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:20:54.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:20:54.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:20:54.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:54.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:20:54.219 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:20:54.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:54.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:20:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:54.433+0000 7f8b3daf1d80 -1 Falling back to public interface 2020-11-16T22:20:54.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:20:55.288 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:20:55.289+0000 7f8b3daf1d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T22:20:55.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:20:55.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:20:55.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:20:55.605 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:20:55.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:20:55.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:20:56.020 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/677712,v1:127.0.0.1:6819/677712] [v2:127.0.0.1:6820/677712,v1:127.0.0.1:6821/677712] exists,up a3c2949a-6826-49c7-a383-cef36b4dbb76 2020-11-16T22:20:56.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:20:56.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:20:56.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:20:56.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2020-11-16T22:20:56.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=675724 2020-11-16T22:20:56.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 675724 2020-11-16T22:20:56.032 INFO:tasks.workunit.client.0.smithi189.stdout: PID TTY TIME CMD 2020-11-16T22:20:56.033 INFO:tasks.workunit.client.0.smithi189.stdout: 675724 ? 00:00:00 ceph-osd 2020-11-16T22:20:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2020-11-16T22:20:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2020-11-16T22:20:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2020-11-16T22:20:56.420 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2020-11-16T22:20:56.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2020-11-16T22:20:56.898 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already stopped. 2020-11-16T22:20:56.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2020-11-16T22:21:11.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2020-11-16T22:21:11.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2020-11-16T22:21:11.910 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2020-11-16T22:21:12.287 INFO:tasks.workunit.client.0.smithi189.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2020-11-16T22:21:12.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 675724 2020-11-16T22:21:12.297 INFO:tasks.workunit.client.0.smithi189.stdout: PID TTY TIME CMD 2020-11-16T22:21:12.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2020-11-16T22:21:12.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2020-11-16T22:21:12.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:21:12.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2020-11-16T22:21:12.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:21:12.299 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:21:12.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:21:12.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:21:12.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:21:12.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:21:12.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:21:12.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:21:12.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:21:12.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:21:12.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:21:12.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:21:12.414 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:12.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:21:12.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:21:12.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:12.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:21:12.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:21:12.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2020-11-16T22:21:12.425 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:21:12.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:12.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:21:12.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.663547 2020-11-16T22:21:12.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:21:12.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:21:12.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T22:21:12.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-markdown 0 2020-11-16T22:21:12.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2020-11-16T22:21:12.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T22:21:12.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2020-11-16T22:21:12.429 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:21:12.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:21:12.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:21:12.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:21:12.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:21:12.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:21:12.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:21:12.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:21:12.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:21:12.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:21:12.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:21:12.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:21:12.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:12.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:21:12.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:21:12.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:12.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:21:12.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T22:21:12.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2020-11-16T22:21:12.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:21:12.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:12.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.663547 2020-11-16T22:21:12.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.663547 2020-11-16T22:21:12.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:21:12.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:21:12.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T22:21:12.446 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T22:21:12.447 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T22:21:12.458 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2020-11-16T22:21:12.458 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2020-11-16T22:21:12.516 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T22:21:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T22:21:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:21:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:21:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T22:21:12.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T22:21:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T22:21:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T22:21:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T22:21:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T22:21:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-reactivate 2020-11-16T22:21:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2020-11-16T22:21:12.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2020-11-16T22:21:12.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2020-11-16T22:21:12.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2020-11-16T22:21:12.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2020-11-16T22:21:12.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2020-11-16T22:21:12.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=54224f28-64a4-4c7e-b810-2542d2264e6f --auth-supported=none ' 2020-11-16T22:21:12.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2020-11-16T22:21:12.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2020-11-16T22:21:12.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T22:21:12.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2020-11-16T22:21:12.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2020-11-16T22:21:12.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2020-11-16T22:21:12.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2020-11-16T22:21:12.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2020-11-16T22:21:12.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2020-11-16T22:21:12.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:21:12.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2020-11-16T22:21:12.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:21:12.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:21:12.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:21:12.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:21:12.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:21:12.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:21:12.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:21:12.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:21:12.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:21:12.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:21:12.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:21:12.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:21:12.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:12.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:21:12.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:21:12.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:12.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:21:12.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:21:12.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2020-11-16T22:21:12.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:21:12.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:12.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.678230 2020-11-16T22:21:12.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.678230 2020-11-16T22:21:12.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:21:12.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:21:12.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2020-11-16T22:21:12.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:21:12.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:12.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.678230 2020-11-16T22:21:12.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.678230 2020-11-16T22:21:12.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:21:12.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T22:21:12.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T22:21:12.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:21:12.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2020-11-16T22:21:12.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2020-11-16T22:21:12.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2020-11-16T22:21:12.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2020-11-16T22:21:12.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2020-11-16T22:21:12.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:21:12.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:21:12.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:21:12.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2020-11-16T22:21:12.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2020-11-16T22:21:12.863 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:21:12.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:21:12.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:21:12.865 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:21:12.865 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:12.865 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.678230 2020-11-16T22:21:12.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.678230/$cluster-$name.asok' 2020-11-16T22:21:12.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.678230/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:21:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:21:13.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:21:13.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:21:13.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:21:13.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:21:13.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:21:13.064 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:21:13.064 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:21:13.064 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:21:13.064 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:21:13.064 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:13.064 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.678230 2020-11-16T22:21:13.065 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.678230/ceph-mon.a.asok 2020-11-16T22:21:13.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:21:13.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.678230/ceph-mon.a.asok config get fsid 2020-11-16T22:21:13.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:21:13.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:21:13.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:21:13.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:21:13.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:21:13.170 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:21:13.170 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:21:13.170 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:21:13.171 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:21:13.171 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:13.171 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.678230 2020-11-16T22:21:13.171 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.678230/ceph-mon.a.asok 2020-11-16T22:21:13.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:21:13.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.678230/ceph-mon.a.asok config get mon_host 2020-11-16T22:21:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2020-11-16T22:21:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-reactivate 2020-11-16T22:21:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:21:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:21:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:21:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-reactivate/x 2020-11-16T22:21:13.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:21:13.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:21:13.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:21:13.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:21:13.826 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:21:13.826 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:13.826 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.678230 2020-11-16T22:21:13.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.678230/$cluster-$name.asok' 2020-11-16T22:21:13.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:21:13.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.678230/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:21:13.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2020-11-16T22:21:13.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-reactivate 2020-11-16T22:21:13.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:21:13.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:21:13.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:21:13.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-reactivate/0 2020-11-16T22:21:13.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=54224f28-64a4-4c7e-b810-2542d2264e6f --auth-supported=none --mon-host=127.0.0.1:7122 ' 2020-11-16T22:21:13.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:21:13.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:21:13.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:21:13.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2020-11-16T22:21:13.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2020-11-16T22:21:13.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:21:13.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:21:13.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2020-11-16T22:21:13.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:21:13.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:21:13.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:21:13.858 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:21:13.858 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:13.858 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.678230 2020-11-16T22:21:13.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.678230/$cluster-$name.asok' 2020-11-16T22:21:13.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.678230/$cluster-$name.asok' 2020-11-16T22:21:13.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:21:13.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:21:13.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:21:13.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2020-11-16T22:21:13.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2020-11-16T22:21:13.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:21:13.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:21:13.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:21:13.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:21:13.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:21:13.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-reactivate/0 2020-11-16T22:21:13.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:21:13.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=348bf41a-150a-4171-b58e-06b19394b72c 2020-11-16T22:21:13.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 348bf41a-150a-4171-b58e-06b19394b72c' 2020-11-16T22:21:13.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:21:13.861 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 348bf41a-150a-4171-b58e-06b19394b72c 2020-11-16T22:21:13.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBZ+7Jf0MreNBAAGuAQhbkIVNhwxGXO5cb/dg== 2020-11-16T22:21:13.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBZ+7Jf0MreNBAAGuAQhbkIVNhwxGXO5cb/dg=="}' 2020-11-16T22:21:13.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 348bf41a-150a-4171-b58e-06b19394b72c -i td/osd-reactivate/0/new.json 2020-11-16T22:21:14.208 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:21:14.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-reactivate/0/new.json 2020-11-16T22:21:14.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=54224f28-64a4-4c7e-b810-2542d2264e6f --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.678230/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBZ+7Jf0MreNBAAGuAQhbkIVNhwxGXO5cb/dg== --osd-uuid 348bf41a-150a-4171-b58e-06b19394b72c 2020-11-16T22:21:14.253 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:14.261+0000 7f1d2fcc9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:14.255 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:14.265+0000 7f1d2fcc9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:14.257 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:14.265+0000 7f1d2fcc9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:14.258 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:14.265+0000 7f1d2fcc9d80 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2020-11-16T22:21:14.258 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:14.265+0000 7f1d2fcc9d80 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2020-11-16T22:21:14.258 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:14.265+0000 7f1d2fcc9d80 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2020-11-16T22:21:14.312 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:14.321+0000 7f1d2fcc9d80 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2020-11-16T22:21:18.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-reactivate/0/keyring 2020-11-16T22:21:18.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:21:18.711 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:21:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:21:18.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:21:19.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:21:19.207 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:21:19.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=54224f28-64a4-4c7e-b810-2542d2264e6f --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.678230/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:21:19.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:21:19.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:21:19.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:21:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:19.241+0000 7fa9e9a56d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:19.245+0000 7fa9e9a56d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:19.245+0000 7fa9e9a56d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:19.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:21:19.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:21:19.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:21:19.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:21:19.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:21:19.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:21:19.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:21:19.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:21:19.591 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:21:19.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:21:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:19.777+0000 7fa9e9a56d80 -1 Falling back to public interface 2020-11-16T22:21:19.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:21:20.370 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:20.373+0000 7fa9e9a56d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:21:20.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:21:20.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:21:20.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:21:20.954 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:21:20.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:21:20.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:21:21.352 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/679083,v1:127.0.0.1:6803/679083] [v2:127.0.0.1:6804/679083,v1:127.0.0.1:6805/679083] exists,up 348bf41a-150a-4171-b58e-06b19394b72c 2020-11-16T22:21:21.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:21:21.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:21:21.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:21:21.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2020-11-16T22:21:21.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:21:21.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:21:21.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:21:21.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:21:21.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:21:21.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:21:21.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2020-11-16T22:21:21.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2020-11-16T22:21:21.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2020-11-16T22:21:21.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2020-11-16T22:21:21.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-reactivate 2020-11-16T22:21:21.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-11-16T22:21:21.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2020-11-16T22:21:21.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-11-16T22:21:21.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-reactivate/0 2020-11-16T22:21:21.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=54224f28-64a4-4c7e-b810-2542d2264e6f --auth-supported=none --mon-host=127.0.0.1:7122 ' 2020-11-16T22:21:21.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:21:21.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:21:21.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:21:21.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2020-11-16T22:21:21.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2020-11-16T22:21:21.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-11-16T22:21:21.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-11-16T22:21:21.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2020-11-16T22:21:21.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-11-16T22:21:21.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:21:21.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:21:21.469 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:21:21.469 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:21.469 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.678230 2020-11-16T22:21:21.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.678230/$cluster-$name.asok' 2020-11-16T22:21:21.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.678230/$cluster-$name.asok' 2020-11-16T22:21:21.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:21:21.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2020-11-16T22:21:21.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2020-11-16T22:21:21.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:21:21.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:21:21.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:21:21.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-11-16T22:21:21.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-11-16T22:21:21.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-reactivate/0 2020-11-16T22:21:21.471 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:21:21.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2020-11-16T22:21:21.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=54224f28-64a4-4c7e-b810-2542d2264e6f --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.678230/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:21:21.472 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-reactivate/0/whoami 2020-11-16T22:21:21.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2020-11-16T22:21:21.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-11-16T22:21:21.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-11-16T22:21:21.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-11-16T22:21:21.493 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:21.501+0000 7fd0ff6d7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:21.496 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:21.505+0000 7fd0ff6d7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:21.498 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:21.505+0000 7fd0ff6d7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:21.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2020-11-16T22:21:21.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:21:21.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:21:21.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:21:21.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:21:21.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:21:21.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:21:21.844 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:21:21.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:21:21.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:21:22.037 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:22.045+0000 7fd0ff6d7d80 -1 Falling back to public interface 2020-11-16T22:21:22.234 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/679083,v1:127.0.0.1:6803/679083] [v2:127.0.0.1:6804/679083,v1:127.0.0.1:6805/679083] exists,up 348bf41a-150a-4171-b58e-06b19394b72c 2020-11-16T22:21:22.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:21:22.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:21:22.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:21:22.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2020-11-16T22:21:22.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2020-11-16T22:21:22.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:21:22.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2020-11-16T22:21:22.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:21:22.236 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:21:22.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:21:22.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:21:22.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:21:22.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:21:22.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:21:22.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:21:22.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:21:22.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:21:22.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:21:22.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:21:22.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:22.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:21:22.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:21:22.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:22.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:21:22.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:21:22.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2020-11-16T22:21:22.361 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:21:22.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:22.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.678230 2020-11-16T22:21:22.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.678230 2020-11-16T22:21:22.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:21:22.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:21:22.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T22:21:22.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-reactivate 0 2020-11-16T22:21:22.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2020-11-16T22:21:22.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T22:21:22.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2020-11-16T22:21:22.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:21:22.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:21:22.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:21:22.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:21:22.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:21:22.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:21:22.369 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:21:22.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:21:22.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:21:22.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:21:22.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:21:22.373 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:21:22.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:22.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:21:22.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:21:22.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:22.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:21:22.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T22:21:22.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2020-11-16T22:21:22.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:21:22.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:22.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.678230 2020-11-16T22:21:22.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.678230 2020-11-16T22:21:22.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:21:22.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:21:22.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T22:21:22.384 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T22:21:22.384 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T22:21:22.440 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2020-11-16T22:21:22.441 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2020-11-16T22:21:22.501 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T22:21:22.515 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T22:21:22.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:21:22.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:21:22.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T22:21:22.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T22:21:22.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T22:21:22.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T22:21:22.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T22:21:22.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T22:21:22.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-recovery-prio 2020-11-16T22:21:22.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2020-11-16T22:21:22.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2020-11-16T22:21:22.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-11-16T22:21:22.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-11-16T22:21:22.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2020-11-16T22:21:22.518 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2020-11-16T22:21:22.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none ' 2020-11-16T22:21:22.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2020-11-16T22:21:22.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: export objects=200 2020-11-16T22:21:22.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: objects=200 2020-11-16T22:21:22.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: export poolprefix=test 2020-11-16T22:21:22.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: poolprefix=test 2020-11-16T22:21:22.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: export FORCE_PRIO=255 2020-11-16T22:21:22.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: FORCE_PRIO=255 2020-11-16T22:21:22.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export NORMAL_PRIO=190 2020-11-16T22:21:22.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: NORMAL_PRIO=190 2020-11-16T22:21:22.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: set 2020-11-16T22:21:22.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T22:21:22.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: local 'funcs=TEST_recovery_pool_priority 2020-11-16T22:21:22.522 INFO:tasks.workunit.client.0.smithi189.stderr:TEST_recovery_priority' 2020-11-16T22:21:22.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: for func in $funcs 2020-11-16T22:21:22.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:36: run: setup td/osd-recovery-prio 2020-11-16T22:21:22.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2020-11-16T22:21:22.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2020-11-16T22:21:22.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2020-11-16T22:21:22.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:21:22.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2020-11-16T22:21:22.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:21:22.524 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:21:22.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:21:22.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:21:22.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:21:22.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:21:22.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:21:22.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:21:22.528 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:21:22.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:21:22.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:21:22.531 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:21:22.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:22.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:21:22.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:21:22.532 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:21:22.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:21:22.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:21:22.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2020-11-16T22:21:22.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:21:22.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:22.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:21:22.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.679603 2020-11-16T22:21:22.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:21:22.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:21:22.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2020-11-16T22:21:22.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:21:22.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:22.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:21:22.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.679603 2020-11-16T22:21:22.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:21:22.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T22:21:22.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T22:21:22.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:21:22.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2020-11-16T22:21:22.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: TEST_recovery_pool_priority td/osd-recovery-prio 2020-11-16T22:21:22.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:323: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2020-11-16T22:21:22.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:324: TEST_recovery_pool_priority: local pools=3 2020-11-16T22:21:22.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:325: TEST_recovery_pool_priority: local OSDS=2 2020-11-16T22:21:22.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2020-11-16T22:21:22.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2020-11-16T22:21:22.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:21:22.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:21:22.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:21:22.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2020-11-16T22:21:22.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2020-11-16T22:21:22.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:21:22.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:21:22.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:21:22.850 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:21:22.850 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:22.850 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:21:22.851 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:21:22.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:21:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:21:23.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:21:23.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:21:23.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:21:23.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:21:23.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:21:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:21:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:21:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:21:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:21:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:21:23.058 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-mon.a.asok 2020-11-16T22:21:23.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:21:23.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.679603/ceph-mon.a.asok config get fsid 2020-11-16T22:21:23.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:21:23.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:21:23.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:21:23.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:21:23.165 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:21:23.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:21:23.165 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:21:23.165 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:21:23.165 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:21:23.166 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:23.166 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:21:23.166 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-mon.a.asok 2020-11-16T22:21:23.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:21:23.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.679603/ceph-mon.a.asok config get mon_host 2020-11-16T22:21:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2020-11-16T22:21:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-prio 2020-11-16T22:21:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:21:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:21:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:21:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-prio/x 2020-11-16T22:21:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:21:23.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:21:23.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:21:23.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:21:23.794 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:21:23.795 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:23.795 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:21:23.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:21:23.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:21:23.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:21:23.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:329: TEST_recovery_pool_priority: export CEPH_ARGS 2020-11-16T22:21:23.825 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: expr 2 - 1 2020-11-16T22:21:23.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: seq 0 1 2020-11-16T22:21:23.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T22:21:23.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2020-11-16T22:21:23.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2020-11-16T22:21:23.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:21:23.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:21:23.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:21:23.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/0 2020-11-16T22:21:23.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2020-11-16T22:21:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:21:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:21:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:21:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2020-11-16T22:21:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2020-11-16T22:21:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:21:23.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:21:23.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2020-11-16T22:21:23.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:21:23.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:21:23.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:21:23.832 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:21:23.832 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:23.833 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:21:23.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:21:23.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:21:23.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:21:23.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:21:23.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:21:23.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2020-11-16T22:21:23.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2020-11-16T22:21:23.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:21:23.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:21:23.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:21:23.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:21:23.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:21:23.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/0 2020-11-16T22:21:23.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:21:23.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e37a9433-f558-404b-ae93-5111bea5360e 2020-11-16T22:21:23.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 e37a9433-f558-404b-ae93-5111bea5360e' 2020-11-16T22:21:23.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:21:23.835 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 e37a9433-f558-404b-ae93-5111bea5360e 2020-11-16T22:21:23.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBj+7JfbHN3MxAAmiQH2+b/eWiMxAqyDmA8Bw== 2020-11-16T22:21:23.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBj+7JfbHN3MxAAmiQH2+b/eWiMxAqyDmA8Bw=="}' 2020-11-16T22:21:23.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e37a9433-f558-404b-ae93-5111bea5360e -i td/osd-recovery-prio/0/new.json 2020-11-16T22:21:24.207 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:21:24.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/0/new.json 2020-11-16T22:21:24.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBj+7JfbHN3MxAAmiQH2+b/eWiMxAqyDmA8Bw== --osd-uuid e37a9433-f558-404b-ae93-5111bea5360e 2020-11-16T22:21:24.246 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:24.253+0000 7f51f9de5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:24.248 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:24.257+0000 7f51f9de5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:24.249 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:24.257+0000 7f51f9de5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:24.250 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:24.257+0000 7f51f9de5d80 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2020-11-16T22:21:24.250 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:24.257+0000 7f51f9de5d80 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2020-11-16T22:21:24.250 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:24.257+0000 7f51f9de5d80 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2020-11-16T22:21:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:24.297+0000 7f51f9de5d80 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2020-11-16T22:21:28.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2020-11-16T22:21:28.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:21:28.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:21:28.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:21:28.627 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:21:29.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:21:29.130 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:21:29.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:21:29.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:21:29.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:21:29.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:21:29.158 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:29.165+0000 7f4dcf135d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:29.161 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:29.169+0000 7f4dcf135d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:29.162 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:29.169+0000 7f4dcf135d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:29.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:21:29.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:21:29.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:21:29.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:21:29.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:21:29.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:21:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:21:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:21:29.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:21:29.491 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:21:29.701 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:29.709+0000 7f4dcf135d80 -1 Falling back to public interface 2020-11-16T22:21:29.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:21:30.576 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:30.577+0000 7f4dcf135d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:21:30.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:21:30.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:21:30.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:21:30.896 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:21:30.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:21:30.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:21:31.298 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/680459,v1:127.0.0.1:6803/680459] [v2:127.0.0.1:6804/680459,v1:127.0.0.1:6805/680459] exists,up e37a9433-f558-404b-ae93-5111bea5360e 2020-11-16T22:21:31.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:21:31.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:21:31.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:21:31.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T22:21:31.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2020-11-16T22:21:31.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2020-11-16T22:21:31.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:21:31.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:21:31.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:21:31.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/1 2020-11-16T22:21:31.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2020-11-16T22:21:31.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:21:31.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:21:31.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:21:31.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2020-11-16T22:21:31.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2020-11-16T22:21:31.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:21:31.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:21:31.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2020-11-16T22:21:31.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:21:31.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:21:31.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:21:31.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:21:31.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:21:31.303 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:21:31.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:21:31.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:21:31.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:21:31.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:21:31.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:21:31.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2020-11-16T22:21:31.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2020-11-16T22:21:31.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:21:31.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:21:31.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:21:31.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:21:31.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:21:31.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/1 2020-11-16T22:21:31.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:21:31.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ce1f03fe-1277-42bb-9acf-e2a8a59e8f95 2020-11-16T22:21:31.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 ce1f03fe-1277-42bb-9acf-e2a8a59e8f95' 2020-11-16T22:21:31.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:21:31.306 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 ce1f03fe-1277-42bb-9acf-e2a8a59e8f95 2020-11-16T22:21:31.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBr+7JfWcPhExAAy4i5s96YnS2rVxZIYfweMw== 2020-11-16T22:21:31.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBr+7JfWcPhExAAy4i5s96YnS2rVxZIYfweMw=="}' 2020-11-16T22:21:31.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ce1f03fe-1277-42bb-9acf-e2a8a59e8f95 -i td/osd-recovery-prio/1/new.json 2020-11-16T22:21:31.969 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:21:31.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/1/new.json 2020-11-16T22:21:31.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBr+7JfWcPhExAAy4i5s96YnS2rVxZIYfweMw== --osd-uuid ce1f03fe-1277-42bb-9acf-e2a8a59e8f95 2020-11-16T22:21:32.017 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:32.025+0000 7f7dd2203d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:32.025 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:32.033+0000 7f7dd2203d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:32.027 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:32.037+0000 7f7dd2203d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:32.028 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:32.037+0000 7f7dd2203d80 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2020-11-16T22:21:32.028 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:32.037+0000 7f7dd2203d80 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2020-11-16T22:21:32.028 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:32.037+0000 7f7dd2203d80 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2020-11-16T22:21:32.095 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:32.105+0000 7f7dd2203d80 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2020-11-16T22:21:36.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2020-11-16T22:21:36.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:21:36.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:21:36.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:21:36.430 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:21:36.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:21:36.925 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:21:36.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:21:36.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:21:36.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:21:36.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:21:36.954 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:36.960+0000 7ff5ae8acd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:36.964+0000 7ff5ae8acd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:36.957 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:36.964+0000 7ff5ae8acd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:21:37.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:21:37.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:21:37.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:21:37.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:21:37.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:21:37.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:21:37.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:21:37.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:21:37.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:21:37.302 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:21:37.501 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:37.508+0000 7ff5ae8acd80 -1 Falling back to public interface 2020-11-16T22:21:37.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:21:38.403 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:21:38.404+0000 7ff5ae8acd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:21:38.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:21:38.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:21:38.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:21:38.687 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:21:38.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:21:38.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:21:39.107 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/681453,v1:127.0.0.1:6811/681453] [v2:127.0.0.1:6812/681453,v1:127.0.0.1:6813/681453] exists,up ce1f03fe-1277-42bb-9acf-e2a8a59e8f95 2020-11-16T22:21:39.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:21:39.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:21:39.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:21:39.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: seq 1 3 2020-11-16T22:21:39.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2020-11-16T22:21:39.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test1 1 1 2020-11-16T22:21:39.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2020-11-16T22:21:39.662 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' already exists 2020-11-16T22:21:39.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:21:40.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2020-11-16T22:21:41.400 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T22:21:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2020-11-16T22:21:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test2 1 1 2020-11-16T22:21:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2020-11-16T22:21:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' already exists 2020-11-16T22:21:42.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:21:43.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2020-11-16T22:21:43.807 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 2 2020-11-16T22:21:43.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2020-11-16T22:21:43.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test3 1 1 2020-11-16T22:21:43.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2020-11-16T22:21:44.390 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' already exists 2020-11-16T22:21:44.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:21:45.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2020-11-16T22:21:46.108 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 2 2020-11-16T22:21:46.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: sleep 5 2020-11-16T22:21:51.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:343: TEST_recovery_pool_priority: wait_for_clean 2020-11-16T22:21:51.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:21:51.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:21:51.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:21:51.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:21:51.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:21:51.124 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:21:51.124 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:21:51.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:21:51.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:21:51.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:21:51.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:21:51.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:21:51.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:21:51.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:21:51.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:21:51.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:21:51.747 INFO:tasks.workunit.client.0.smithi189.stderr:1' 2020-11-16T22:21:51.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:21:51.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:21:51.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:21:51.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836486 2020-11-16T22:21:51.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836486 2020-11-16T22:21:51.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486' 2020-11-16T22:21:51.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:21:51.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:21:52.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2020-11-16T22:21:52.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2020-11-16T22:21:52.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-11-16T22:21:52.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:21:52.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836486 2020-11-16T22:21:52.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:21:52.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:21:52.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836486 2020-11-16T22:21:52.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:21:52.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836486 2020-11-16T22:21:52.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-11-16T22:21:52.087 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836486 2020-11-16T22:21:52.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:21:52.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836484 -lt 21474836486 2020-11-16T22:21:52.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:21:53.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:21:53.480 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:21:53.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836486 2020-11-16T22:21:53.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:21:53.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2020-11-16T22:21:53.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:21:53.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:21:53.894 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2020-11-16T22:21:53.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:21:53.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2020-11-16T22:21:53.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-11-16T22:21:53.896 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672965 2020-11-16T22:21:53.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:21:54.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672965 -lt 42949672965 2020-11-16T22:21:54.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:21:54.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:21:54.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:21:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2020-11-16T22:21:54.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:21:54.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:21:54.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:21:54.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:21:54.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:21:54.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:21:54.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:21:55.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2020-11-16T22:21:55.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:21:55.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:21:55.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:21:55.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2020-11-16T22:21:55.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:21:55.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:21:55.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:345: TEST_recovery_pool_priority: ceph pg dump pgs 2020-11-16T22:21:55.988 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T22:21:55.988 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:21:46.475821+0000 0'0 26:18 [1,0] 1 [1,0] 1 0'0 2020-11-16T22:21:44.225744+0000 0'0 2020-11-16T22:21:44.225744+0000 0 2020-11-16T22:21:55.988 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:21:41.623700+0000 0'0 25:27 [1,0] 1 [1,0] 1 0'0 2020-11-16T22:21:39.487880+0000 0'0 2020-11-16T22:21:39.487880+0000 0 2020-11-16T22:21:55.988 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:21:44.492305+0000 0'0 24:22 [0,1] 0 [0,1] 0 0'0 2020-11-16T22:21:41.874396+0000 0'0 2020-11-16T22:21:41.874396+0000 0 2020-11-16T22:21:55.989 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T22:21:55.989 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T22:21:55.990 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T22:21:56.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:349: TEST_recovery_pool_priority: local PG1 2020-11-16T22:21:56.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:350: TEST_recovery_pool_priority: local POOLNUM1 2020-11-16T22:21:56.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:351: TEST_recovery_pool_priority: local pool1 2020-11-16T22:21:56.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local chk_osd1_1 2020-11-16T22:21:56.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local chk_osd1_2 2020-11-16T22:21:56.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local PG2 2020-11-16T22:21:56.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local POOLNUM2 2020-11-16T22:21:56.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:357: TEST_recovery_pool_priority: local pool2 2020-11-16T22:21:56.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local chk_osd2_1 2020-11-16T22:21:56.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local chk_osd2_2 2020-11-16T22:21:56.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: seq 1 3 2020-11-16T22:21:56.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2020-11-16T22:21:56.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2020-11-16T22:21:56.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2020-11-16T22:21:56.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2020-11-16T22:21:56.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=1 2020-11-16T22:21:56.379 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2020-11-16T22:21:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=0 2020-11-16T22:21:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z '' ']' 2020-11-16T22:21:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: PG1=1.0 2020-11-16T22:21:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: POOLNUM1=1 2020-11-16T22:21:56.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:370: TEST_recovery_pool_priority: pool1=test1 2020-11-16T22:21:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: chk_osd1_1=1 2020-11-16T22:21:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: chk_osd1_2=0 2020-11-16T22:21:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2020-11-16T22:21:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2020-11-16T22:21:56.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2020-11-16T22:21:56.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2020-11-16T22:21:56.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=0 2020-11-16T22:21:56.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2020-11-16T22:21:56.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=1 2020-11-16T22:21:56.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2020-11-16T22:21:56.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2020-11-16T22:21:56.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: PG2=2.0 2020-11-16T22:21:56.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: POOLNUM2=2 2020-11-16T22:21:56.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:377: TEST_recovery_pool_priority: pool2=test2 2020-11-16T22:21:56.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: chk_osd2_1=0 2020-11-16T22:21:56.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: chk_osd2_2=1 2020-11-16T22:21:56.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: break 2020-11-16T22:21:56.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2020-11-16T22:21:56.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:385: TEST_recovery_pool_priority: '[' test2 = '' ']' 2020-11-16T22:21:56.746 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: seq 1 3 2020-11-16T22:21:56.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2020-11-16T22:21:56.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2020-11-16T22:21:56.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2020-11-16T22:21:56.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2020-11-16T22:21:56.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2020-11-16T22:21:56.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2020-11-16T22:21:56.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:395: TEST_recovery_pool_priority: delete_pool test3 2020-11-16T22:21:56.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2020-11-16T22:21:56.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-11-16T22:21:57.329 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' does not exist 2020-11-16T22:21:57.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:399: TEST_recovery_pool_priority: pool1_extra_prio=1 2020-11-16T22:21:57.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:400: TEST_recovery_pool_priority: pool2_extra_prio=2 2020-11-16T22:21:57.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: expr 190 + 1 2020-11-16T22:21:57.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: pool1_prio=191 2020-11-16T22:21:57.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: expr 190 + 2 2020-11-16T22:21:57.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool2_prio=192 2020-11-16T22:21:57.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2020-11-16T22:21:58.323 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 1 2020-11-16T22:21:58.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2020-11-16T22:21:59.040 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 recovery_priority to 1 2020-11-16T22:21:59.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:406: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2020-11-16T22:21:59.826 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 1 2020-11-16T22:21:59.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2020-11-16T22:22:00.521 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 recovery_priority to 2 2020-11-16T22:22:00.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: wait_for_clean 2020-11-16T22:22:00.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:22:00.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:22:00.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:22:00.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:22:00.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:22:00.537 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:22:00.537 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:22:00.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:22:00.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:22:00.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:22:00.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:22:00.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:22:00.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:22:00.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:22:00.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:22:01.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:22:01.124 INFO:tasks.workunit.client.0.smithi189.stderr:1' 2020-11-16T22:22:01.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:22:01.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:22:01.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:22:01.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2020-11-16T22:22:01.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2020-11-16T22:22:01.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2020-11-16T22:22:01.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:22:01.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:22:01.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672969 2020-11-16T22:22:01.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672969 2020-11-16T22:22:01.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2020-11-16T22:22:01.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:22:01.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2020-11-16T22:22:01.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:22:01.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:22:01.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2020-11-16T22:22:01.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:22:01.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2020-11-16T22:22:01.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-11-16T22:22:01.461 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836490 2020-11-16T22:22:01.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:22:01.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2020-11-16T22:22:01.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:22:01.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672969 2020-11-16T22:22:01.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:22:01.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:22:01.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2020-11-16T22:22:01.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:22:01.862 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672969 2020-11-16T22:22:01.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672969 2020-11-16T22:22:01.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-11-16T22:22:01.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:22:02.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672969 -lt 42949672969 2020-11-16T22:22:02.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:22:02.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:22:02.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:22:02.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2020-11-16T22:22:02.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:22:02.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:22:02.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:22:02.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:22:02.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:22:02.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:22:02.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:22:03.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:22:03.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:22:03.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:22:03.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:22:03.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2020-11-16T22:22:03.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:22:03.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:22:03.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2020-11-16T22:22:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:10+0 records in 2020-11-16T22:22:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:10+0 records out 2020-11-16T22:22:03.556 INFO:tasks.workunit.client.0.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0524146 s, 200 MB/s 2020-11-16T22:22:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: p=1 2020-11-16T22:22:03.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2020-11-16T22:22:03.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2020-11-16T22:22:03.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:03.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2020-11-16T22:22:03.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:03.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2020-11-16T22:22:04.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:04.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2020-11-16T22:22:04.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:04.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2020-11-16T22:22:04.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:04.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2020-11-16T22:22:05.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:05.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2020-11-16T22:22:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2020-11-16T22:22:06.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:06.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2020-11-16T22:22:06.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:06.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2020-11-16T22:22:06.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:06.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2020-11-16T22:22:07.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:07.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2020-11-16T22:22:07.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:07.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2020-11-16T22:22:07.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:07.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2020-11-16T22:22:08.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:08.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2020-11-16T22:22:09.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:09.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2020-11-16T22:22:09.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:09.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2020-11-16T22:22:09.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:09.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2020-11-16T22:22:10.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:10.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2020-11-16T22:22:10.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:10.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2020-11-16T22:22:10.805 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:10.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2020-11-16T22:22:11.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:11.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2020-11-16T22:22:11.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:11.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2020-11-16T22:22:12.421 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:12.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2020-11-16T22:22:12.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:12.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2020-11-16T22:22:13.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:13.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2020-11-16T22:22:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2020-11-16T22:22:14.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:14.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2020-11-16T22:22:14.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:14.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2020-11-16T22:22:14.632 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:14.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2020-11-16T22:22:14.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:14.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2020-11-16T22:22:15.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:15.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2020-11-16T22:22:16.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:16.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2020-11-16T22:22:16.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:16.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2020-11-16T22:22:16.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:16.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2020-11-16T22:22:17.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:17.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2020-11-16T22:22:17.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:17.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2020-11-16T22:22:17.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:17.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2020-11-16T22:22:18.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:18.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2020-11-16T22:22:18.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:18.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2020-11-16T22:22:19.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:19.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2020-11-16T22:22:19.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:19.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2020-11-16T22:22:20.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:20.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2020-11-16T22:22:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2020-11-16T22:22:20.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:20.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2020-11-16T22:22:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2020-11-16T22:22:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2020-11-16T22:22:22.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:22.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2020-11-16T22:22:22.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:22.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2020-11-16T22:22:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:23.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2020-11-16T22:22:24.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:24.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2020-11-16T22:22:24.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:24.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2020-11-16T22:22:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2020-11-16T22:22:25.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:25.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2020-11-16T22:22:25.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:25.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2020-11-16T22:22:26.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:26.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2020-11-16T22:22:26.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:26.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2020-11-16T22:22:26.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:26.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2020-11-16T22:22:27.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:27.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2020-11-16T22:22:27.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:27.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2020-11-16T22:22:28.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:28.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2020-11-16T22:22:28.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:28.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2020-11-16T22:22:29.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:29.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2020-11-16T22:22:29.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:29.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2020-11-16T22:22:29.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:29.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2020-11-16T22:22:30.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:30.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2020-11-16T22:22:30.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:30.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2020-11-16T22:22:31.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:31.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2020-11-16T22:22:31.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:31.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2020-11-16T22:22:32.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2020-11-16T22:22:32.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:32.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2020-11-16T22:22:32.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:32.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2020-11-16T22:22:34.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:34.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2020-11-16T22:22:34.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:34.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2020-11-16T22:22:34.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:34.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2020-11-16T22:22:35.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:35.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2020-11-16T22:22:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2020-11-16T22:22:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:35.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2020-11-16T22:22:36.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:36.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2020-11-16T22:22:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2020-11-16T22:22:37.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2020-11-16T22:22:37.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:37.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2020-11-16T22:22:37.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:37.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2020-11-16T22:22:38.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:38.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2020-11-16T22:22:38.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:38.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2020-11-16T22:22:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:38.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2020-11-16T22:22:39.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:39.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2020-11-16T22:22:39.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:39.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2020-11-16T22:22:40.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:40.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2020-11-16T22:22:40.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:40.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2020-11-16T22:22:41.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:41.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2020-11-16T22:22:41.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:41.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2020-11-16T22:22:41.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:41.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2020-11-16T22:22:42.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:42.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2020-11-16T22:22:42.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:42.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2020-11-16T22:22:43.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:43.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2020-11-16T22:22:44.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:44.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2020-11-16T22:22:46.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:46.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2020-11-16T22:22:47.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:47.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2020-11-16T22:22:47.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:47.804 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2020-11-16T22:22:48.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:48.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2020-11-16T22:22:48.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:48.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2020-11-16T22:22:49.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:49.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2020-11-16T22:22:49.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:49.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2020-11-16T22:22:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:49.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2020-11-16T22:22:50.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:50.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2020-11-16T22:22:50.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:50.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2020-11-16T22:22:50.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:50.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2020-11-16T22:22:51.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:51.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2020-11-16T22:22:51.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:51.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2020-11-16T22:22:52.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:52.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2020-11-16T22:22:52.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:52.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2020-11-16T22:22:53.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:53.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2020-11-16T22:22:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2020-11-16T22:22:53.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:53.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2020-11-16T22:22:54.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:54.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2020-11-16T22:22:54.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:54.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2020-11-16T22:22:55.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:55.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2020-11-16T22:22:55.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:55.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2020-11-16T22:22:56.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:56.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2020-11-16T22:22:56.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:56.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2020-11-16T22:22:57.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:57.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2020-11-16T22:22:57.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:57.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2020-11-16T22:22:57.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:57.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2020-11-16T22:22:58.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:58.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2020-11-16T22:22:58.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:58.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2020-11-16T22:22:59.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:59.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2020-11-16T22:22:59.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:59.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2020-11-16T22:22:59.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:22:59.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2020-11-16T22:23:00.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:00.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2020-11-16T22:23:00.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:00.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2020-11-16T22:23:01.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:01.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2020-11-16T22:23:01.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:01.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2020-11-16T22:23:02.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:02.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2020-11-16T22:23:02.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:02.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2020-11-16T22:23:02.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:02.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2020-11-16T22:23:03.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:03.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2020-11-16T22:23:03.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:03.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2020-11-16T22:23:04.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:04.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2020-11-16T22:23:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:04.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2020-11-16T22:23:05.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:05.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2020-11-16T22:23:05.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:05.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2020-11-16T22:23:06.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:06.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2020-11-16T22:23:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2020-11-16T22:23:07.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:07.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2020-11-16T22:23:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:07.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2020-11-16T22:23:07.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:07.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2020-11-16T22:23:08.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:08.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2020-11-16T22:23:08.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:08.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2020-11-16T22:23:08.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:08.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2020-11-16T22:23:09.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:09.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2020-11-16T22:23:09.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:09.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2020-11-16T22:23:10.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:10.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2020-11-16T22:23:10.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:10.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2020-11-16T22:23:11.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:11.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2020-11-16T22:23:11.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:11.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2020-11-16T22:23:11.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:11.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2020-11-16T22:23:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2020-11-16T22:23:12.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:12.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2020-11-16T22:23:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2020-11-16T22:23:13.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:13.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2020-11-16T22:23:13.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:13.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2020-11-16T22:23:14.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:14.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2020-11-16T22:23:14.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:14.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2020-11-16T22:23:14.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:14.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2020-11-16T22:23:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2020-11-16T22:23:16.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:16.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2020-11-16T22:23:17.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:17.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2020-11-16T22:23:17.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:17.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2020-11-16T22:23:17.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:17.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2020-11-16T22:23:18.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:18.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2020-11-16T22:23:18.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:18.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2020-11-16T22:23:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2020-11-16T22:23:19.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:19.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2020-11-16T22:23:19.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:19.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2020-11-16T22:23:20.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:20.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2020-11-16T22:23:20.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:20.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2020-11-16T22:23:20.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:20.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2020-11-16T22:23:21.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:21.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2020-11-16T22:23:21.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:21.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2020-11-16T22:23:21.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:21.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2020-11-16T22:23:22.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:22.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2020-11-16T22:23:23.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:23.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2020-11-16T22:23:23.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:23.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2020-11-16T22:23:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2020-11-16T22:23:24.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:24.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2020-11-16T22:23:24.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:24.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2020-11-16T22:23:25.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:25.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2020-11-16T22:23:25.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:25.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2020-11-16T22:23:26.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:26.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2020-11-16T22:23:27.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:27.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2020-11-16T22:23:29.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:29.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2020-11-16T22:23:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2020-11-16T22:23:29.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:29.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2020-11-16T22:23:30.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:30.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2020-11-16T22:23:30.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:30.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2020-11-16T22:23:30.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:30.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2020-11-16T22:23:31.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:31.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2020-11-16T22:23:31.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:31.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2020-11-16T22:23:32.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:32.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2020-11-16T22:23:32.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:32.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2020-11-16T22:23:32.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 1 + 1 2020-11-16T22:23:32.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=2 2020-11-16T22:23:32.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2020-11-16T22:23:32.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2020-11-16T22:23:32.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:32.850 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2020-11-16T22:23:33.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:33.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2020-11-16T22:23:33.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:33.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2020-11-16T22:23:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:33.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2020-11-16T22:23:34.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:34.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2020-11-16T22:23:35.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:35.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2020-11-16T22:23:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2020-11-16T22:23:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2020-11-16T22:23:36.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:36.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2020-11-16T22:23:36.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:36.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2020-11-16T22:23:36.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:36.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2020-11-16T22:23:37.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:37.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2020-11-16T22:23:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:37.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2020-11-16T22:23:38.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:38.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2020-11-16T22:23:38.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:38.631 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2020-11-16T22:23:38.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:38.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2020-11-16T22:23:39.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:39.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2020-11-16T22:23:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2020-11-16T22:23:40.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:40.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2020-11-16T22:23:40.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:40.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2020-11-16T22:23:41.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:41.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2020-11-16T22:23:41.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:41.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2020-11-16T22:23:41.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:41.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2020-11-16T22:23:42.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:42.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2020-11-16T22:23:43.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:43.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2020-11-16T22:23:43.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:43.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2020-11-16T22:23:44.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:44.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2020-11-16T22:23:44.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:44.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2020-11-16T22:23:44.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:44.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2020-11-16T22:23:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2020-11-16T22:23:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2020-11-16T22:23:45.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:45.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2020-11-16T22:23:46.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:46.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2020-11-16T22:23:47.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:47.034 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2020-11-16T22:23:47.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:47.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2020-11-16T22:23:47.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:47.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2020-11-16T22:23:48.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:48.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2020-11-16T22:23:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2020-11-16T22:23:48.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:48.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2020-11-16T22:23:49.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:49.548 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2020-11-16T22:23:49.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:49.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2020-11-16T22:23:50.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:50.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2020-11-16T22:23:50.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:50.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2020-11-16T22:23:51.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:51.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2020-11-16T22:23:51.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:51.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2020-11-16T22:23:51.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:51.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2020-11-16T22:23:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2020-11-16T22:23:53.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:53.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2020-11-16T22:23:53.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:53.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2020-11-16T22:23:53.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:53.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2020-11-16T22:23:54.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:54.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2020-11-16T22:23:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2020-11-16T22:23:54.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:54.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2020-11-16T22:23:55.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:55.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2020-11-16T22:23:55.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:55.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2020-11-16T22:23:56.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:56.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2020-11-16T22:23:56.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:56.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2020-11-16T22:23:56.880 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:56.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2020-11-16T22:23:57.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:57.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2020-11-16T22:23:57.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:57.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2020-11-16T22:23:57.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:57.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2020-11-16T22:23:58.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:58.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2020-11-16T22:23:59.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:59.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2020-11-16T22:23:59.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:59.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2020-11-16T22:23:59.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:23:59.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2020-11-16T22:24:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:00.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2020-11-16T22:24:00.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:00.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2020-11-16T22:24:00.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:00.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2020-11-16T22:24:00.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:00.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2020-11-16T22:24:01.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:01.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2020-11-16T22:24:02.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:02.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2020-11-16T22:24:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:02.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2020-11-16T22:24:03.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:03.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2020-11-16T22:24:03.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:03.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2020-11-16T22:24:03.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:03.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2020-11-16T22:24:04.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:04.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2020-11-16T22:24:04.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:04.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2020-11-16T22:24:05.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:05.252 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2020-11-16T22:24:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2020-11-16T22:24:05.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:05.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2020-11-16T22:24:06.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:06.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2020-11-16T22:24:06.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:06.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2020-11-16T22:24:06.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:06.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2020-11-16T22:24:07.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:07.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2020-11-16T22:24:08.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:08.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2020-11-16T22:24:08.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:08.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2020-11-16T22:24:08.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:08.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2020-11-16T22:24:09.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:09.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2020-11-16T22:24:09.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:09.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2020-11-16T22:24:09.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:09.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2020-11-16T22:24:10.639 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:10.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2020-11-16T22:24:11.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:11.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2020-11-16T22:24:11.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:11.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2020-11-16T22:24:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:11.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2020-11-16T22:24:12.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:12.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2020-11-16T22:24:13.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:13.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2020-11-16T22:24:14.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:14.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2020-11-16T22:24:15.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:15.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2020-11-16T22:24:15.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:15.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2020-11-16T22:24:15.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:15.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2020-11-16T22:24:16.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:16.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2020-11-16T22:24:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2020-11-16T22:24:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2020-11-16T22:24:17.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:17.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2020-11-16T22:24:18.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:18.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2020-11-16T22:24:18.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:18.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2020-11-16T22:24:18.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:18.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2020-11-16T22:24:19.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:19.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2020-11-16T22:24:19.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:19.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2020-11-16T22:24:20.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:20.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2020-11-16T22:24:20.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:20.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2020-11-16T22:24:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:20.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2020-11-16T22:24:21.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:21.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2020-11-16T22:24:21.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:21.590 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2020-11-16T22:24:21.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:21.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2020-11-16T22:24:22.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:22.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2020-11-16T22:24:23.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:23.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2020-11-16T22:24:23.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:23.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2020-11-16T22:24:24.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:24.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2020-11-16T22:24:24.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:24.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2020-11-16T22:24:24.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:24.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2020-11-16T22:24:24.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:24.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2020-11-16T22:24:25.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:25.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2020-11-16T22:24:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2020-11-16T22:24:26.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:26.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2020-11-16T22:24:26.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:26.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2020-11-16T22:24:27.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:27.087 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2020-11-16T22:24:27.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:27.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2020-11-16T22:24:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:27.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2020-11-16T22:24:28.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:28.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2020-11-16T22:24:28.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:28.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2020-11-16T22:24:29.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:29.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2020-11-16T22:24:29.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:29.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2020-11-16T22:24:29.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:29.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2020-11-16T22:24:30.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:30.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2020-11-16T22:24:30.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:30.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2020-11-16T22:24:30.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:30.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2020-11-16T22:24:31.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:31.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2020-11-16T22:24:32.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:32.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2020-11-16T22:24:32.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:32.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2020-11-16T22:24:32.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:32.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2020-11-16T22:24:33.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:33.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2020-11-16T22:24:33.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:33.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2020-11-16T22:24:34.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:34.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2020-11-16T22:24:35.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:35.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2020-11-16T22:24:35.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:35.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2020-11-16T22:24:35.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:35.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2020-11-16T22:24:35.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:35.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2020-11-16T22:24:36.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:36.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2020-11-16T22:24:36.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:36.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2020-11-16T22:24:36.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:36.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2020-11-16T22:24:37.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:37.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2020-11-16T22:24:38.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:38.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2020-11-16T22:24:38.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:38.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2020-11-16T22:24:38.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:38.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2020-11-16T22:24:39.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:39.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2020-11-16T22:24:39.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:39.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2020-11-16T22:24:39.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:39.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2020-11-16T22:24:40.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:40.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2020-11-16T22:24:41.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:41.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2020-11-16T22:24:41.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:41.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2020-11-16T22:24:41.913 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:41.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2020-11-16T22:24:42.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:42.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2020-11-16T22:24:42.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:42.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2020-11-16T22:24:42.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:42.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2020-11-16T22:24:44.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2020-11-16T22:24:44.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:44.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2020-11-16T22:24:44.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:44.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2020-11-16T22:24:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:45.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2020-11-16T22:24:45.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:45.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2020-11-16T22:24:45.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:45.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2020-11-16T22:24:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:46.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2020-11-16T22:24:47.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:47.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2020-11-16T22:24:47.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:47.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2020-11-16T22:24:47.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:47.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2020-11-16T22:24:48.144 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:48.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2020-11-16T22:24:48.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:48.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2020-11-16T22:24:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2020-11-16T22:24:49.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:49.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2020-11-16T22:24:50.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:50.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2020-11-16T22:24:50.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:50.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2020-11-16T22:24:50.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:50.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2020-11-16T22:24:50.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:50.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2020-11-16T22:24:51.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:51.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2020-11-16T22:24:51.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:51.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2020-11-16T22:24:51.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:51.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2020-11-16T22:24:52.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:52.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2020-11-16T22:24:53.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:53.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2020-11-16T22:24:53.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:53.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2020-11-16T22:24:55.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2020-11-16T22:24:55.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:55.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2020-11-16T22:24:56.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:56.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2020-11-16T22:24:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2020-11-16T22:24:56.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:56.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2020-11-16T22:24:57.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:57.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2020-11-16T22:24:57.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:57.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2020-11-16T22:24:57.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:57.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2020-11-16T22:24:58.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:58.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2020-11-16T22:24:59.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:59.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2020-11-16T22:24:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2020-11-16T22:24:59.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2020-11-16T22:24:59.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 2 + 1 2020-11-16T22:24:59.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=3 2020-11-16T22:24:59.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2020-11-16T22:24:59.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test1 2020-11-16T22:24:59.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1-p1 2020-11-16T22:24:59.776 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test1 obj1-p1 2020-11-16T22:24:59.776 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test1 2020-11-16T22:24:59.777 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1-p1 2020-11-16T22:24:59.777 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test1 obj1-p1 2020-11-16T22:24:59.777 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T22:25:00.183 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T22:25:00.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-11-16T22:25:00.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T22:25:00.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: local otherosd=null 2020-11-16T22:25:00.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:423: TEST_recovery_pool_priority: ceph pg dump pgs 2020-11-16T22:25:00.945 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T22:25:00.945 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2020-11-16T22:21:58.252067+0000 37'600 37:643 [1] 1 [1] 1 0'0 2020-11-16T22:21:39.487880+0000 0'0 2020-11-16T22:21:39.487880+0000 0 2020-11-16T22:25:00.946 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 193 0 0 0 0 2023751680 0 0 579 579 active+clean 2020-11-16T22:22:00.192763+0000 37'579 37:618 [0] 0 [0] 0 0'0 2020-11-16T22:21:41.874396+0000 0'0 2020-11-16T22:21:41.874396+0000 0 2020-11-16T22:25:00.946 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T22:25:00.946 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T22:25:00.947 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T22:25:00.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: ERRORS=0 2020-11-16T22:25:00.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2020-11-16T22:25:01.924 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T22:25:01.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2020-11-16T22:25:02.963 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 2 2020-11-16T22:25:02.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:428: TEST_recovery_pool_priority: sleep 10 2020-11-16T22:25:12.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: get_asok_path osd.1 2020-11-16T22:25:12.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T22:25:12.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T22:25:12.984 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:25:12.985 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:25:12.985 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:25:12.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.1.asok 2020-11-16T22:25:12.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: CEPH_ARGS= 2020-11-16T22:25:12.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:25:13.102 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 2020-11-16T22:25:13.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: echo osd.1 2020-11-16T22:25:13.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:431: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2020-11-16T22:25:13.103 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:25:13.103 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:25:13.104 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:25:13.104 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:25:13.104 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:25:13.104 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:25:13.104 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:25:13.104 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:25:13.104 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 191, 2020-11-16T22:25:13.105 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:25:13.105 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:25:13.105 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:25:13.105 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:25:13.105 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:25:13.105 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:25:13.105 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:25:13.106 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:25:13.106 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:25:13.106 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:25:13.106 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "2.0", 2020-11-16T22:25:13.106 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 192, 2020-11-16T22:25:13.106 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:25:13.106 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:25:13.107 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:25:13.107 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:25:13.107 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:25:13.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:432: TEST_recovery_pool_priority: get_asok_path osd.0 2020-11-16T22:25:13.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-11-16T22:25:13.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-11-16T22:25:13.108 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:25:13.108 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:25:13.108 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:25:13.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.0.asok 2020-11-16T22:25:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:432: TEST_recovery_pool_priority: CEPH_ARGS= 2020-11-16T22:25:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:432: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.0.asok dump_recovery_reservations 2020-11-16T22:25:13.215 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 2020-11-16T22:25:13.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: echo osd.0 2020-11-16T22:25:13.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:434: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2020-11-16T22:25:13.216 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:25:13.216 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:25:13.216 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:25:13.217 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:25:13.217 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:25:13.217 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:25:13.217 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:25:13.217 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "2.0", 2020-11-16T22:25:13.217 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 192, 2020-11-16T22:25:13.217 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:25:13.218 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:25:13.218 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:25:13.218 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:25:13.218 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:25:13.218 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:25:13.218 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:25:13.218 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:25:13.219 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:25:13.219 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:25:13.219 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:25:13.219 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 191, 2020-11-16T22:25:13.219 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:25:13.219 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:25:13.219 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:25:13.219 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:25:13.220 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:25:13.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2020-11-16T22:25:13.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-11-16T22:25:13.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2020-11-16T22:25:13.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: ITEM=1.0 2020-11-16T22:25:13.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-11-16T22:25:13.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2020-11-16T22:25:13.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-11-16T22:25:13.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: PRIO=191 2020-11-16T22:25:13.235 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:444: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2020-11-16T22:25:13.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:452: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2020-11-16T22:25:13.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:452: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-11-16T22:25:13.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:452: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2020-11-16T22:25:13.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:452: TEST_recovery_pool_priority: ITEM=1.0 2020-11-16T22:25:13.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-11-16T22:25:13.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2020-11-16T22:25:13.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-11-16T22:25:13.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: PRIO=191 2020-11-16T22:25:13.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2020-11-16T22:25:13.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2020-11-16T22:25:13.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-11-16T22:25:13.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2020-11-16T22:25:13.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: ITEM=2.0 2020-11-16T22:25:13.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:468: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-11-16T22:25:13.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:473: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2020-11-16T22:25:13.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:473: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-11-16T22:25:13.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:473: TEST_recovery_pool_priority: PRIO=192 2020-11-16T22:25:13.263 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:474: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2020-11-16T22:25:13.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2020-11-16T22:25:13.265 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-11-16T22:25:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2020-11-16T22:25:13.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: ITEM=2.0 2020-11-16T22:25:13.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:483: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-11-16T22:25:13.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:488: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2020-11-16T22:25:13.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:488: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-11-16T22:25:13.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:488: TEST_recovery_pool_priority: PRIO=192 2020-11-16T22:25:13.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:489: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2020-11-16T22:25:13.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:496: TEST_recovery_pool_priority: wait_for_clean 2020-11-16T22:25:13.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:25:13.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:25:13.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:25:13.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:25:13.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:25:13.279 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:25:13.280 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:25:13.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:25:13.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:25:13.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:25:13.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:25:13.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:25:13.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:25:13.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:25:13.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:25:14.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:25:14.099 INFO:tasks.workunit.client.0.smithi189.stderr:1' 2020-11-16T22:25:14.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:25:14.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:25:14.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:25:14.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836531 2020-11-16T22:25:14.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836531 2020-11-16T22:25:14.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836531' 2020-11-16T22:25:14.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:25:14.266 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:25:14.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673009 2020-11-16T22:25:14.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673009 2020-11-16T22:25:14.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836531 1-42949673009' 2020-11-16T22:25:14.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:25:14.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836531 2020-11-16T22:25:14.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:25:14.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:25:14.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836531 2020-11-16T22:25:14.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:25:14.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836531 2020-11-16T22:25:14.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2020-11-16T22:25:14.540 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836531 2020-11-16T22:25:14.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:25:14.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836529 -lt 21474836531 2020-11-16T22:25:14.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:25:15.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:25:15.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:25:16.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836531 -lt 21474836531 2020-11-16T22:25:16.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:25:16.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673009 2020-11-16T22:25:16.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:25:16.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:25:16.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673009 2020-11-16T22:25:16.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:25:16.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673009 2020-11-16T22:25:16.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2020-11-16T22:25:16.871 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673009 2020-11-16T22:25:16.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:25:17.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673010 -lt 42949673009 2020-11-16T22:25:17.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:25:17.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:17.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:17.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2020-11-16T22:25:17.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:17.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:17.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:17.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:17.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:17.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:17.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:18.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:18.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:18.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:18.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:18.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:18.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T22:25:18.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T22:25:18.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T22:25:18.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:18.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:18.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:18.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:18.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:18.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:18.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:18.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:18.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:18.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:19.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:19.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:19.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:19.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:19.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:19.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:19.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:19.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:19.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:19.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:19.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:19.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:19.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:20.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20965109 2020-11-16T22:25:20.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20965109 '!=' null 2020-11-16T22:25:20.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:20.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:20.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:20.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:20.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:20.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:20.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:20.544 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:20.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:20.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:20.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:20.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:20.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:20.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:20.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:21.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:21.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:21.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:21.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:21.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:21.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:21.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:21.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:21.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:21.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26206581 2020-11-16T22:25:21.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26206581 '!=' null 2020-11-16T22:25:21.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:21.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:21.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:22.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:22.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:22.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:22.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:22.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:22.057 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:22.057 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:22.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:22.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:22.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:22.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:23.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:23.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:23.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:23.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:23.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:23.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:23.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:23.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:23.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26206581 2020-11-16T22:25:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26206581 '!=' null 2020-11-16T22:25:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:23.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:23.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:23.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:23.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:23.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:23.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:24.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:24.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:24.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:24.068 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:24.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:24.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:24.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:24.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:24.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:24.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:24.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:24.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:24.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:25.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333545 2020-11-16T22:25:25.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333545 '!=' null 2020-11-16T22:25:25.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:25.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:25.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:25.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:25.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:25.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:25.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:25.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:25.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:25.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:25.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:25.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:25.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:25.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:25.667 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:26.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:26.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:26.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:26.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:26.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:26.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:26.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:26.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26207348 2020-11-16T22:25:26.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26207348 '!=' null 2020-11-16T22:25:26.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:26.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:26.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:26.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:26.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:26.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:26.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:26.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:26.876 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:26.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:26.877 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:27.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:27.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:27.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:27.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:27.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:27.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:27.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:27.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:27.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:27.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:27.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:27.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:27.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:28.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333969 2020-11-16T22:25:28.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333969 '!=' null 2020-11-16T22:25:28.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:28.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:28.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:28.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:28.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:28.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:28.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:28.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:28.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:28.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:28.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:28.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:28.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:29.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:29.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:29.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:29.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:29.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:29.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:29.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:29.361 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:29.362 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:29.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22713265 2020-11-16T22:25:29.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22713265 '!=' null 2020-11-16T22:25:29.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:29.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:29.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:30.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:30.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:30.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:30.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:30.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:30.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:30.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:30.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:30.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:30.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:30.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:30.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:30.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:30.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:30.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:30.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:30.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:30.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:30.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:30.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:30.861 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:31.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22713265 2020-11-16T22:25:31.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22713265 '!=' null 2020-11-16T22:25:31.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:31.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:31.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:31.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:31.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:31.485 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:31.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:31.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:31.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:31.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:31.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:31.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:31.868 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:31.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:31.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:32.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:32.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:32.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:32.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:32.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:32.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:32.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:32.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:32.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:33.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27955004 2020-11-16T22:25:33.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27955004 '!=' null 2020-11-16T22:25:33.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:33.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:33.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:33.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:33.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:33.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:33.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:33.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:33.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:33.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:33.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:33.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:33.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:33.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:33.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:34.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:34.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:34.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28828613 2020-11-16T22:25:34.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28828613 '!=' null 2020-11-16T22:25:34.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:34.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:34.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:35.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:35.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:35.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:35.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:35.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:35.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:35.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:35.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:35.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:35.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:35.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:35.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:35.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:35.938 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:36.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27081347 2020-11-16T22:25:36.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27081347 '!=' null 2020-11-16T22:25:36.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:36.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:36.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:36.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:36.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:36.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:36.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:36.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:36.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:36.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:36.678 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:37.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:37.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:37.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:37.158 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:37.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:37.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:37.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:37.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:37.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:37.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:37.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:37.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:37.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:38.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27081347 2020-11-16T22:25:38.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27081347 '!=' null 2020-11-16T22:25:38.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:38.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:38.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:38.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:38.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:38.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:38.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:38.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:38.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:38.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:38.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:38.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:38.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:38.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:39.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:39.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:39.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20966423 2020-11-16T22:25:39.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20966423 '!=' null 2020-11-16T22:25:39.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:39.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:39.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:39.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:39.944 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:39.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:39.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:39.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:39.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:39.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:39.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:40.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:40.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:40.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:41.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:41.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:41.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:41.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:41.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:41.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:41.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:41.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:41.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:41.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20966423 2020-11-16T22:25:41.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20966423 '!=' null 2020-11-16T22:25:41.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:41.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:41.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:41.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:41.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:41.720 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:41.720 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:41.720 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:41.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:41.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:41.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:42.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:42.074 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:42.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:42.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:42.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:42.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:42.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:42.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:42.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:42.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:42.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:42.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:42.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:43.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27081851 2020-11-16T22:25:43.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27081851 '!=' null 2020-11-16T22:25:43.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:43.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:43.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:43.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:43.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:43.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:43.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:43.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:43.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:43.671 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:43.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:44.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:44.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:44.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:44.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:44.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:44.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:44.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:44.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:44.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:44.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:44.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:44.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:44.724 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:45.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27081584 2020-11-16T22:25:45.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27081584 '!=' null 2020-11-16T22:25:45.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:45.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:45.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:45.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:45.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:45.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:45.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:45.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:45.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:45.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:45.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:45.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:45.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:45.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:45.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:46.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:46.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:46.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:46.154 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:47.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26207751 2020-11-16T22:25:47.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26207751 '!=' null 2020-11-16T22:25:47.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:47.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:47.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:47.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:47.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:47.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:47.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:47.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:47.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:47.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:47.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:47.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:47.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:47.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:47.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:48.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:48.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:48.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:48.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:48.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:48.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:48.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:48.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:48.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:48.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27081105 2020-11-16T22:25:48.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27081105 '!=' null 2020-11-16T22:25:48.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:48.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:48.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:48.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:48.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:48.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:48.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:48.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:48.680 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:48.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:48.681 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:49.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:49.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:49.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:49.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:49.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:49.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:49.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:49.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:49.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:49.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:49.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:49.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:49.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:50.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27081105 2020-11-16T22:25:50.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27081105 '!=' null 2020-11-16T22:25:50.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:50.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:50.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:50.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:50.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:50.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:50.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:50.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:50.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:50.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:50.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:50.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:50.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:50.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:50.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:51.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:51.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:51.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:51.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:51.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:51.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:51.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:51.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:51.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:52.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27080049 2020-11-16T22:25:52.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27080049 '!=' null 2020-11-16T22:25:52.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:52.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:52.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:52.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:52.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:52.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:52.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:52.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:52.181 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:52.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:52.182 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:52.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:52.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:52.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:52.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:53.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:53.207 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:53.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26206149 2020-11-16T22:25:53.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26206149 '!=' null 2020-11-16T22:25:53.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:53.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:53.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:54.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:54.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:54.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:54.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:54.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:54.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:54.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:54.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:54.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:54.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:54.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:54.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:55.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:55.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:55.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:55.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:55.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:55.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:55.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:55.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:55.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:55.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26206149 2020-11-16T22:25:55.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26206149 '!=' null 2020-11-16T22:25:55.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:55.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:55.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:55.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:55.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:55.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:55.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:55.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:55.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:55.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:55.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:56.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:56.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:56.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:56.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:56.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:56.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:56.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:56.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:56.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:56.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:56.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:56.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:56.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:57.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26206507 2020-11-16T22:25:57.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26206507 '!=' null 2020-11-16T22:25:57.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:57.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:57.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:57.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:57.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:57.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:57.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:57.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:57.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:58.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:58.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:58.004 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:58.004 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:58.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:58.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:58.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:58.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:58.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:58.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:58.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:58.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:58.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:25:59.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25332899 2020-11-16T22:25:59.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25332899 '!=' null 2020-11-16T22:25:59.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:25:59.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:25:59.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:25:59.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:25:59.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:25:59.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:25:59.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:25:59.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:25:59.132 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:25:59.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:25:59.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:25:59.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:25:59.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:25:59.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:25:59.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:25:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:25:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:25:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:25:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:25:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:25:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:25:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:25:59.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:25:59.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:00.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20091032 2020-11-16T22:26:00.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20091032 '!=' null 2020-11-16T22:26:00.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:00.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:00.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:00.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:00.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:00.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:00.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:00.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:00.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:00.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:00.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:01.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:01.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:01.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:01.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:01.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:01.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:01.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:01.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:02.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20091032 2020-11-16T22:26:02.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20091032 '!=' null 2020-11-16T22:26:02.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:02.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:02.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:02.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:02.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:02.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:02.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:02.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:02.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:02.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:02.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:02.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:02.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:02.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:02.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:03.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:03.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:03.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:03.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:03.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:03.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:03.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:03.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:03.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333082 2020-11-16T22:26:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333082 '!=' null 2020-11-16T22:26:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:03.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:03.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:03.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:03.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:03.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:03.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:03.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:03.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:03.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:03.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:04.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:04.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:04.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:04.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:04.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:04.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:04.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:04.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:04.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:04.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:04.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:04.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:04.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:05.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333068 2020-11-16T22:26:05.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333068 '!=' null 2020-11-16T22:26:05.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:05.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:05.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:05.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:05.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:05.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:05.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:05.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:05.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:05.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:05.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:05.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:05.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:05.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:05.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:06.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:06.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:06.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:06.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:06.213 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:06.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:06.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:06.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:06.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:06.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25332949 2020-11-16T22:26:06.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25332949 '!=' null 2020-11-16T22:26:06.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:06.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:06.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:06.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:06.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:06.833 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:06.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:06.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:06.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:06.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:06.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:07.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:07.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:07.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:07.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:07.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:07.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:07.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:07.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:07.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:07.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:07.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:07.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:07.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:08.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25332949 2020-11-16T22:26:08.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25332949 '!=' null 2020-11-16T22:26:08.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:08.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:08.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:08.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:08.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:08.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:08.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:08.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:08.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:08.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:08.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:09.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:09.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:09.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:09.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:09.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:09.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:09.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:09.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:09.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:10.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21838897 2020-11-16T22:26:10.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21838897 '!=' null 2020-11-16T22:26:10.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:10.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:10.066 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:10.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:10.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:10.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:10.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:10.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:10.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:10.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:10.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:10.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:10.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:10.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:10.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:11.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:11.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:11.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:11.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:11.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27081006 2020-11-16T22:26:11.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27081006 '!=' null 2020-11-16T22:26:11.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:11.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:11.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:12.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:12.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:12.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:12.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:12.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:12.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:12.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:12.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:12.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:12.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:12.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:12.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:13.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:13.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:13.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:13.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:13.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27081006 2020-11-16T22:26:13.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27081006 '!=' null 2020-11-16T22:26:13.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:13.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:13.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:13.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:13.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:13.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:13.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:13.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:13.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:13.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:14.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:14.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:14.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:14.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:15.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:15.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:15.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:15.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:15.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:15.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:15.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:15.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:15.262 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:16.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333539 2020-11-16T22:26:16.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333539 '!=' null 2020-11-16T22:26:16.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:16.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:16.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:16.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:16.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:16.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:16.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:16.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:16.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:16.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:16.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:16.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:16.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:16.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:16.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:17.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:17.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:17.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:18.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333539 2020-11-16T22:26:18.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333539 '!=' null 2020-11-16T22:26:18.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:18.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:18.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:18.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:18.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:18.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:18.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:18.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:18.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:18.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:18.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:18.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:18.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:18.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:18.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:19.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:19.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:19.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:19.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:19.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:19.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:19.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:19.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:19.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:19.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724214 2020-11-16T22:26:19.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724214 '!=' null 2020-11-16T22:26:19.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:19.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:19.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:20.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:20.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:20.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:20.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:20.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:20.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:20.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:20.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:20.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:20.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:20.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:20.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:21.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:21.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:21.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724214 2020-11-16T22:26:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724214 '!=' null 2020-11-16T22:26:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:21.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:21.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:21.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:21.765 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:21.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:21.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:21.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:21.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:21.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:22.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:22.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:22.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:22.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:22.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:22.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:22.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:22.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:22.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:22.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:22.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:22.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:22.800 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:23.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20092316 2020-11-16T22:26:23.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20092316 '!=' null 2020-11-16T22:26:23.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:23.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:23.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:24.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:24.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:24.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:24.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:24.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:24.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:24.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:24.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:24.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:24.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:24.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:24.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:24.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:24.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:24.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:24.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:24.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:24.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:24.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:24.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:24.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19218541 2020-11-16T22:26:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19218541 '!=' null 2020-11-16T22:26:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:25.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:25.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:25.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:25.588 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:25.588 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:25.588 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:25.588 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:25.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:25.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:25.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:25.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:25.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:25.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:26.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:26.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:26.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:26.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:26.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:26.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:26.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:26.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:26.906 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:27.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18344706 2020-11-16T22:26:27.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18344706 '!=' null 2020-11-16T22:26:27.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:27.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:27.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:27.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:27.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:27.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:27.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:27.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:27.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:27.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:27.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:27.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:27.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:27.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:27.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:28.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:28.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:28.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:28.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:28.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:28.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:28.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:28.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:28.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:28.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20091772 2020-11-16T22:26:28.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20091772 '!=' null 2020-11-16T22:26:28.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:28.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:28.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:29.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:29.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:29.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:29.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:29.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:29.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:29.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:29.035 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:29.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:29.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:29.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:29.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:30.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:30.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:30.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:30.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:30.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:30.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:30.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:30.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:30.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:30.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18344820 2020-11-16T22:26:30.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18344820 '!=' null 2020-11-16T22:26:30.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:30.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:30.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:30.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:30.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:30.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:30.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:30.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:30.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:30.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:30.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:31.139 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:31.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:31.141 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:31.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:31.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:31.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:31.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:31.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:31.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:31.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:31.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:31.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:31.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:32.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23585948 2020-11-16T22:26:32.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23585948 '!=' null 2020-11-16T22:26:32.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:32.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:32.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:32.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:32.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:32.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:32.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:32.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:32.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:32.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:32.598 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:32.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:32.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:32.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:32.988 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:33.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:33.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:33.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:33.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:33.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:33.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:33.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:33.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:33.525 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:34.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25332499 2020-11-16T22:26:34.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25332499 '!=' null 2020-11-16T22:26:34.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:34.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:34.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:34.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:34.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:34.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:34.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:34.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:34.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:34.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:34.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:34.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:34.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:35.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:35.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:35.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:35.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:35.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:35.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:35.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:35.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:35.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:35.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25332499 2020-11-16T22:26:35.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25332499 '!=' null 2020-11-16T22:26:35.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:35.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:35.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:35.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:35.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:35.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:35.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:35.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:35.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:35.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:36.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:36.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:36.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:36.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:36.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:36.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:36.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:36.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:36.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:36.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:36.755 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27079744 2020-11-16T22:26:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27079744 '!=' null 2020-11-16T22:26:37.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:37.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:37.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:37.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:37.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:37.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:37.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:37.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:37.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:37.740 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:37.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:37.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:37.742 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:38.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:38.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:38.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:38.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:38.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:38.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:38.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:38.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:38.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:39.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27079744 2020-11-16T22:26:39.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27079744 '!=' null 2020-11-16T22:26:39.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:39.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:39.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:39.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:39.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:39.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:39.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:39.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:39.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:39.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:39.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:39.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:39.602 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:39.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:39.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:40.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:40.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:40.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:40.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:40.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:40.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:40.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:40.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:40.097 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:40.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21838562 2020-11-16T22:26:40.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21838562 '!=' null 2020-11-16T22:26:40.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:40.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:40.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:40.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:40.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:40.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:40.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:40.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:40.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:40.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:40.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:41.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:41.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:41.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:41.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:41.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:41.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:41.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:41.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:41.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:41.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:41.932 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:42.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26206124 2020-11-16T22:26:42.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26206124 '!=' null 2020-11-16T22:26:42.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:42.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:42.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:42.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:42.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:42.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:42.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:42.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:42.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:42.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:42.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:43.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:43.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:43.075 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:43.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:43.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:43.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:43.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:43.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:43.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:43.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:43.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:43.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:43.557 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:44.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26206733 2020-11-16T22:26:44.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26206733 '!=' null 2020-11-16T22:26:44.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:44.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:44.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:44.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:44.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:44.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:44.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:44.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:44.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:44.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:44.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:45.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:45.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:45.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:45.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:45.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:45.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:45.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:45.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:45.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:45.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:45.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:45.744 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:45.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:46.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333570 2020-11-16T22:26:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333570 '!=' null 2020-11-16T22:26:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:46.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:46.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:46.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:46.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:46.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:46.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:46.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:46.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:46.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:46.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:46.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:46.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:46.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:47.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:47.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:47.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:47.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:47.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:47.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:47.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:47.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:47.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:47.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333570 2020-11-16T22:26:47.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333570 '!=' null 2020-11-16T22:26:47.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:47.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:47.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:47.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:47.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:47.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:47.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:47.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:47.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:47.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:47.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:48.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:48.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:48.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:48.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:48.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:48.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:48.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:48.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:48.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:48.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:48.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:48.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:49.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24460180 2020-11-16T22:26:49.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24460180 '!=' null 2020-11-16T22:26:49.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:49.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:49.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:49.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:49.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:49.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:49.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:49.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:49.565 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:49.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:49.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:49.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:49.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:49.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:49.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:50.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:50.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:50.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:50.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:50.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:50.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:50.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:50.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:50.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20092323 2020-11-16T22:26:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20092323 '!=' null 2020-11-16T22:26:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:51.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:51.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:51.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:51.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:51.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:51.324 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:51.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:51.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:51.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:51.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:51.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:51.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:52.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:52.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:52.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:52.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:52.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:52.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:52.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:52.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:52.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:52.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333295 2020-11-16T22:26:52.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333295 '!=' null 2020-11-16T22:26:52.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:52.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:52.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:52.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:52.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:52.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:52.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:52.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:52.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:52.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:52.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:53.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:53.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:53.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:53.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:54.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:54.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:54.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:54.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:54.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:54.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:54.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:54.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:54.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:54.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26207201 2020-11-16T22:26:54.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26207201 '!=' null 2020-11-16T22:26:54.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:54.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:54.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:54.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:54.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:54.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:54.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:54.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:54.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:54.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:54.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:55.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:55.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:55.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:55.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:55.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:55.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:55.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:55.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:55.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:55.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:55.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:55.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:55.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26207120 2020-11-16T22:26:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26207120 '!=' null 2020-11-16T22:26:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:56.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:56.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:56.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:56.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:56.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:56.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:56.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:56.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:56.530 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:56.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:56.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:56.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:56.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:57.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:57.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:57.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:57.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:57.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:57.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:57.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:57.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:57.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:26:58.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333763 2020-11-16T22:26:58.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333763 '!=' null 2020-11-16T22:26:58.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:26:58.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:26:58.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:26:58.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:26:58.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:26:58.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:26:58.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:26:58.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:26:58.156 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:26:58.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:26:58.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:26:58.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:26:58.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:26:58.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:26:58.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:26:59.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:26:59.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:26:59.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:26:59.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:26:59.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:26:59.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:26:59.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:26:59.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:26:59.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:00.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20092325 2020-11-16T22:27:00.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20092325 '!=' null 2020-11-16T22:27:00.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:00.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:00.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:00.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:00.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:00.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:00.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:00.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:00.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:00.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:00.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:00.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:00.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:00.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:00.726 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:01.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:01.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:01.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:01.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:01.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:01.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:01.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:01.230 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:01.230 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25334099 2020-11-16T22:27:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25334099 '!=' null 2020-11-16T22:27:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:01.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:01.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:01.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:01.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:01.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:01.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:01.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:01.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:02.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:02.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:02.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:02.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:02.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:02.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:02.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:02.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:02.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:02.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:02.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:02.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:02.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:03.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25334099 2020-11-16T22:27:03.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25334099 '!=' null 2020-11-16T22:27:03.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:03.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:03.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:03.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:03.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:03.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:03.722 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:03.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:03.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:03.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:03.723 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:04.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:04.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:04.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:04.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:04.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:04.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:04.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:04.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:04.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:04.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:04.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:04.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:04.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25334799 2020-11-16T22:27:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25334799 '!=' null 2020-11-16T22:27:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:05.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:05.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:05.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:05.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:05.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:05.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:05.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:05.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:05.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:05.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:05.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:05.603 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:06.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:06.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:06.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:06.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:06.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26208641 2020-11-16T22:27:06.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26208641 '!=' null 2020-11-16T22:27:06.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:06.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:06.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:07.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:07.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:07.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:07.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:07.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:07.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:07.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:07.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:07.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:07.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:07.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:07.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:07.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:07.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:07.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:07.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:07.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:07.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:07.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:07.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:07.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:09.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26208632 2020-11-16T22:27:09.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26208632 '!=' null 2020-11-16T22:27:09.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:09.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:09.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:09.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:09.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:09.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:09.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:09.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:09.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:09.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:09.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:09.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:09.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:09.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:09.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:10.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:10.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:10.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:10.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:10.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:10.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:10.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:10.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:10.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:10.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21840575 2020-11-16T22:27:10.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21840575 '!=' null 2020-11-16T22:27:10.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:10.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:10.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:10.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:10.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:10.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:10.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:10.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:10.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:10.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:11.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:11.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:11.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:11.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:11.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:11.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:11.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:11.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:12.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27082358 2020-11-16T22:27:12.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27082358 '!=' null 2020-11-16T22:27:12.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:12.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:12.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:12.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:12.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:12.414 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:12.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:12.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:12.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:12.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:12.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:12.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:12.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:12.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:12.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:13.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:13.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:13.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:13.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:13.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:13.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:13.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:13.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:13.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:13.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27082358 2020-11-16T22:27:13.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27082358 '!=' null 2020-11-16T22:27:13.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:13.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:13.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:13.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:13.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:13.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:13.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:13.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:13.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:13.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:13.995 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:14.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:14.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:14.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:14.457 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:15.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:15.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:15.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27082387 2020-11-16T22:27:15.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27082387 '!=' null 2020-11-16T22:27:15.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:15.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:15.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:15.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:15.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:15.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:15.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:15.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:15.776 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:15.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:15.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:16.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:16.194 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:16.194 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:16.194 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:16.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:16.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:16.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:16.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:16.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:16.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:16.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:16.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:16.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26208588 2020-11-16T22:27:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26208588 '!=' null 2020-11-16T22:27:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:17.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:17.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:17.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:17.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:17.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:17.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:17.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:17.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:18.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:18.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:18.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:18.067 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:18.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:18.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:18.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:18.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:18.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:18.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:18.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:18.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:18.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:19.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25334247 2020-11-16T22:27:19.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25334247 '!=' null 2020-11-16T22:27:19.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:19.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:19.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:19.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:19.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:19.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:19.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:19.193 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:19.193 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:19.193 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:19.193 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:19.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:19.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:19.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:19.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:20.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:20.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:20.069 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:20.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:20.070 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20092176 2020-11-16T22:27:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20092176 '!=' null 2020-11-16T22:27:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:20.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:20.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:20.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:20.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:20.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:20.801 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:20.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:20.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:20.802 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:21.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:21.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:21.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:21.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:21.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:21.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:21.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:21.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:21.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:22.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333198 2020-11-16T22:27:22.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333198 '!=' null 2020-11-16T22:27:22.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:22.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:22.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:22.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:22.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:22.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:22.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:22.226 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:22.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:22.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:22.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:22.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:22.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:22.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:22.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:23.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:23.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:23.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:23.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:23.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:23.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:23.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:23.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:23.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:24.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25332782 2020-11-16T22:27:24.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25332782 '!=' null 2020-11-16T22:27:24.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:24.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:24.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:24.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:24.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:24.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:24.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:24.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:24.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:24.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:24.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:24.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:24.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:24.939 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:24.940 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:25.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:25.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:25.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:25.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:25.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25332398 2020-11-16T22:27:25.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25332398 '!=' null 2020-11-16T22:27:25.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:25.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:25.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:26.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:26.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:26.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:26.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:26.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:26.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:26.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:26.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:26.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:26.563 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:26.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:26.564 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:27.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:27.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:27.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:27.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:27.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:27.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:27.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:27.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:27.196 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:27.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25332398 2020-11-16T22:27:27.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25332398 '!=' null 2020-11-16T22:27:27.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:27.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:27.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:27.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:27.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:27.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:27.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:27.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:27.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:27.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:27.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:28.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:28.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:28.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:28.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:28.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:28.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:28.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:28.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:28.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:28.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:28.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:28.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:28.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:29.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26205908 2020-11-16T22:27:29.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26205908 '!=' null 2020-11-16T22:27:29.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:29.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:29.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:29.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:29.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:29.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:29.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:29.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:29.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:29.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:29.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:29.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:29.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:29.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:29.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:30.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:30.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:30.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:30.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:30.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:30.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:30.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:30.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:30.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:31.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20964906 2020-11-16T22:27:31.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20964906 '!=' null 2020-11-16T22:27:31.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:31.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:31.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:31.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:31.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:31.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:31.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:31.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:31.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:31.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:31.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:31.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:31.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:31.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:31.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:32.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:32.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:32.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:32.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:32.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:32.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:32.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:32.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:32.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:32.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26206441 2020-11-16T22:27:32.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26206441 '!=' null 2020-11-16T22:27:32.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:32.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:32.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:32.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:32.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:32.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:32.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:32.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:32.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:32.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:32.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:33.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:33.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:33.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:33.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:34.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:34.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:34.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:34.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:34.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:34.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:34.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:34.171 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:34.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:34.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26206515 2020-11-16T22:27:34.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26206515 '!=' null 2020-11-16T22:27:34.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:34.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:34.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:34.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:34.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:34.758 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:34.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:34.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:34.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:34.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:34.760 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:35.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:35.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:35.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:35.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:35.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:35.845 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:35.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:35.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:35.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:36.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26206515 2020-11-16T22:27:36.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26206515 '!=' null 2020-11-16T22:27:36.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:36.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:36.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:36.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:36.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:36.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:36.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:36.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:36.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:36.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:36.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:37.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:37.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:37.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:38.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:38.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:38.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:38.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:38.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:38.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:38.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:38.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:38.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:38.907 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26206654 2020-11-16T22:27:38.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26206654 '!=' null 2020-11-16T22:27:38.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:38.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:38.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:39.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:39.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:39.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:39.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:39.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:39.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:39.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:39.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:39.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:39.398 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:39.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:39.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:40.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:40.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:40.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:40.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:40.604 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:40.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:40.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:40.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:40.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:41.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20965282 2020-11-16T22:27:41.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20965282 '!=' null 2020-11-16T22:27:41.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:41.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:41.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:41.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:41.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:41.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:41.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:41.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:41.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:41.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:41.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:41.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:41.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:41.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:41.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:42.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:42.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:42.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:42.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:42.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:42.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:42.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:42.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:42.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:42.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23585542 2020-11-16T22:27:42.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23585542 '!=' null 2020-11-16T22:27:42.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:42.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:42.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:43.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:43.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:43.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:43.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:43.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:43.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:43.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:43.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:43.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:43.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:43.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:43.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:43.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:43.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:43.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:43.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:43.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:43.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:43.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:43.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:43.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:44.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20964674 2020-11-16T22:27:44.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20964674 '!=' null 2020-11-16T22:27:44.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:44.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:44.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:44.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:44.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:44.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:44.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:44.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:44.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:44.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:44.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:45.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:45.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:45.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:45.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:45.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:45.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:45.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:45.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:45.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:45.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:45.779 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:45.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:45.781 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20091134 2020-11-16T22:27:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20091134 '!=' null 2020-11-16T22:27:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:46.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:46.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:46.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:46.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:46.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:46.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:46.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:46.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:46.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:46.797 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:46.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:46.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:47.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:47.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:47.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:47.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:47.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:47.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:47.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:47.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:47.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:48.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20091426 2020-11-16T22:27:48.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20091426 '!=' null 2020-11-16T22:27:48.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:48.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:48.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:48.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:48.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:48.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:48.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:48.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:48.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:48.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:48.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:48.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:48.515 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:49.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:49.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:49.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:49.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:49.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:49.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:49.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:49.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:49.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:49.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20091426 2020-11-16T22:27:49.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20091426 '!=' null 2020-11-16T22:27:49.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:49.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:49.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:49.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:49.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:49.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:49.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:49.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:49.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:49.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:49.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:50.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:50.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:50.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:50.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:50.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:50.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:50.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:50.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:50.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:50.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:50.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:50.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:50.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:51.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850229 2020-11-16T22:27:51.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850229 '!=' null 2020-11-16T22:27:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:51.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:51.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:51.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:51.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:51.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:51.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:51.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:51.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:52.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:52.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:52.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:52.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:52.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:52.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:52.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:52.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:52.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:52.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:52.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:52.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:52.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20092008 2020-11-16T22:27:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20092008 '!=' null 2020-11-16T22:27:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:53.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:53.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:53.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:53.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:53.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:53.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:53.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:53.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:53.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:53.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:53.691 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:54.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:54.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:54.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:54.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:54.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:54.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22713194 2020-11-16T22:27:54.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22713194 '!=' null 2020-11-16T22:27:54.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:54.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:54.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:54.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:54.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:54.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:54.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:54.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:54.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:54.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:54.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:55.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:55.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:55.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:55.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:55.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:55.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:55.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:55.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:55.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:55.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:55.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:55.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:55.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:56.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23587073 2020-11-16T22:27:56.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23587073 '!=' null 2020-11-16T22:27:56.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:56.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:56.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:56.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:56.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:56.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:56.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:56.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:56.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:56.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:56.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:57.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:27:57.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:57.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:57.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:57.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2020-11-16T22:27:57.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:27:57.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:57.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:57.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:57.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:57.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:57.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:57.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:27:57.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=31449222 2020-11-16T22:27:57.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 31449222 '!=' null 2020-11-16T22:27:57.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:27:57.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:57.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:58.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:58.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:58.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:58.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:58.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:58.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:58.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:58.077 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:58.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:27:58.461 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:58.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:58.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:58.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 2 2020-11-16T22:27:58.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 0 2020-11-16T22:27:58.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T22:27:58.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=1 2020-11-16T22:27:58.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:27:58.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:27:59.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:27:59.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:27:59.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:27:59.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:27:59.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:27:59.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:27:59.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:27:59.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:27:59.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:27:59.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:27:59.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:27:59.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:27:59.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 2 2020-11-16T22:27:59.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:27:59.877 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:27:59.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:27:59.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:27:59.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:27:59.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:27:59.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:27:59.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:28:00.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27082706 2020-11-16T22:28:00.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27082706 '!=' null 2020-11-16T22:28:00.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:28:00.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:28:00.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:28:00.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:28:00.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:28:00.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:28:00.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:28:00.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:28:00.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:28:00.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:28:00.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:28:00.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:28:00.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:28:00.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:28:00.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:28:01.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 2 2020-11-16T22:28:01.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:28:01.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:28:01.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:28:01.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:28:01.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:28:01.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:28:01.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:28:01.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:28:01.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=10155789 2020-11-16T22:28:01.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 10155789 '!=' null 2020-11-16T22:28:01.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:28:01.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-11-16T22:28:01.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:28:01.855 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:28:01.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:28:01.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:28:01.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:28:01.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:28:01.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:28:01.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:28:01.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:28:02.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:28:02.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:28:02.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:28:02.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:28:02.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2020-11-16T22:28:02.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:28:02.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:28:02.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:498: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2020-11-16T22:28:02.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:502: TEST_recovery_pool_priority: echo TEST PASSED 2020-11-16T22:28:02.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:505: TEST_recovery_pool_priority: delete_pool test1 2020-11-16T22:28:02.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2020-11-16T22:28:02.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-11-16T22:28:02.730 INFO:tasks.workunit.client.0.smithi189.stdout:TEST PASSED 2020-11-16T22:28:03.341 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' does not exist 2020-11-16T22:28:03.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: delete_pool test2 2020-11-16T22:28:03.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2020-11-16T22:28:03.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-11-16T22:28:04.003 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' does not exist 2020-11-16T22:28:04.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:507: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2020-11-16T22:28:04.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:28:04.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:28:04.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:28:04.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:28:04.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:28:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:28:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:508: TEST_recovery_pool_priority: return 0 2020-11-16T22:28:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: teardown td/osd-recovery-prio 2020-11-16T22:28:04.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2020-11-16T22:28:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:28:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2020-11-16T22:28:04.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:28:04.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:28:04.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:28:04.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:28:04.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:28:04.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:28:04.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:28:04.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:28:04.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:28:04.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:28:04.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:28:04.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:28:04.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:28:04.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:28:04.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:28:04.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:28:04.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:28:04.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:28:04.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2020-11-16T22:28:04.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:28:04.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:04.388 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.679603 2020-11-16T22:28:04.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:28:04.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:28:04.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: for func in $funcs 2020-11-16T22:28:04.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:36: run: setup td/osd-recovery-prio 2020-11-16T22:28:04.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2020-11-16T22:28:04.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2020-11-16T22:28:04.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2020-11-16T22:28:04.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:28:04.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2020-11-16T22:28:04.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:28:04.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:28:04.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:28:04.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:28:04.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:28:04.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:28:04.396 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:28:04.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:28:04.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:28:04.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:28:04.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:28:04.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:28:04.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:28:04.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:28:04.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:28:04.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:28:04.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:28:04.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:28:04.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2020-11-16T22:28:04.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:28:04.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:04.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:04.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.679603 2020-11-16T22:28:04.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:28:04.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:28:04.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2020-11-16T22:28:04.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:28:04.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:04.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:04.412 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.679603 2020-11-16T22:28:04.415 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:28:04.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-11-16T22:28:04.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:28:04.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2020-11-16T22:28:04.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: TEST_recovery_priority td/osd-recovery-prio 2020-11-16T22:28:04.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:44: TEST_recovery_priority: local dir=td/osd-recovery-prio 2020-11-16T22:28:04.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:45: TEST_recovery_priority: local pools=10 2020-11-16T22:28:04.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:46: TEST_recovery_priority: local OSDS=5 2020-11-16T22:28:04.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local max_tries=10 2020-11-16T22:28:04.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2020-11-16T22:28:04.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2020-11-16T22:28:04.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:28:04.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:28:04.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:28:04.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2020-11-16T22:28:04.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2020-11-16T22:28:04.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:28:04.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:28:04.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:28:04.730 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:28:04.730 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:04.730 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:04.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:04.731 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:28:04.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:28:04.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:28:04.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:28:04.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:28:04.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:28:04.943 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:28:04.944 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:28:04.944 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:28:04.944 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:28:04.945 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:28:04.945 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:04.945 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:04.945 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-mon.a.asok 2020-11-16T22:28:04.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:28:04.946 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.679603/ceph-mon.a.asok config get fsid 2020-11-16T22:28:05.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:28:05.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:28:05.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:28:05.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:28:05.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:28:05.056 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:28:05.057 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:28:05.057 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:28:05.057 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:28:05.057 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:05.057 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:05.058 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-mon.a.asok 2020-11-16T22:28:05.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:28:05.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.679603/ceph-mon.a.asok config get mon_host 2020-11-16T22:28:05.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2020-11-16T22:28:05.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-prio 2020-11-16T22:28:05.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:28:05.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:28:05.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:28:05.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-prio/x 2020-11-16T22:28:05.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:28:05.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:28:05.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:28:05.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:28:05.689 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:28:05.690 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:05.690 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:05.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:05.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:28:05.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:28:05.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:51: TEST_recovery_priority: export CEPH_ARGS 2020-11-16T22:28:05.723 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: expr 5 - 1 2020-11-16T22:28:05.725 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: seq 0 4 2020-11-16T22:28:05.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T22:28:05.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2020-11-16T22:28:05.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2020-11-16T22:28:05.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:28:05.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:28:05.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:28:05.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/0 2020-11-16T22:28:05.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2020-11-16T22:28:05.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:28:05.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:28:05.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:28:05.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2020-11-16T22:28:05.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2020-11-16T22:28:05.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:28:05.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:28:05.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2020-11-16T22:28:05.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:28:05.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:28:05.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:28:05.732 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:28:05.732 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:05.732 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:05.732 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:05.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:05.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:28:05.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:28:05.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:28:05.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2020-11-16T22:28:05.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2020-11-16T22:28:05.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:28:05.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:28:05.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:28:05.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:28:05.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:28:05.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/0 2020-11-16T22:28:05.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:28:05.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=40bd4a54-dacb-406e-bb0e-39a87013df11 2020-11-16T22:28:05.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 40bd4a54-dacb-406e-bb0e-39a87013df11' 2020-11-16T22:28:05.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:28:05.735 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 40bd4a54-dacb-406e-bb0e-39a87013df11 2020-11-16T22:28:05.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD1/LJfIUp6LRAAEMRbvN3bRfljs5BpHESTcA== 2020-11-16T22:28:05.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD1/LJfIUp6LRAAEMRbvN3bRfljs5BpHESTcA=="}' 2020-11-16T22:28:05.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 40bd4a54-dacb-406e-bb0e-39a87013df11 -i td/osd-recovery-prio/0/new.json 2020-11-16T22:28:06.118 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:28:06.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/0/new.json 2020-11-16T22:28:06.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD1/LJfIUp6LRAAEMRbvN3bRfljs5BpHESTcA== --osd-uuid 40bd4a54-dacb-406e-bb0e-39a87013df11 2020-11-16T22:28:06.162 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:06.169+0000 7f813522fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:06.165 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:06.173+0000 7f813522fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:06.167 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:06.177+0000 7f813522fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:06.168 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:06.177+0000 7f813522fd80 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2020-11-16T22:28:06.168 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:06.177+0000 7f813522fd80 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2020-11-16T22:28:06.168 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:06.177+0000 7f813522fd80 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2020-11-16T22:28:06.216 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:06.225+0000 7f813522fd80 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2020-11-16T22:28:10.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2020-11-16T22:28:10.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:28:10.703 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:28:10.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:28:10.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:28:11.200 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:28:11.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:28:11.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:28:11.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:28:11.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:28:11.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:28:11.228 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:11.237+0000 7f1988df0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:11.230 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:11.241+0000 7f1988df0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:11.232 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:11.241+0000 7f1988df0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:11.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:28:11.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:28:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:28:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:28:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:28:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:28:11.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:28:11.584 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:28:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:28:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:28:11.764 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:11.773+0000 7f1988df0d80 -1 Falling back to public interface 2020-11-16T22:28:11.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:28:12.615 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:12.617+0000 7f1988df0d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:28:12.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:28:12.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:28:12.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:28:12.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:28:12.962 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:28:12.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:28:13.372 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/699862,v1:127.0.0.1:6803/699862] [v2:127.0.0.1:6804/699862,v1:127.0.0.1:6805/699862] exists,up 40bd4a54-dacb-406e-bb0e-39a87013df11 2020-11-16T22:28:13.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:28:13.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:28:13.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:28:13.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T22:28:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2020-11-16T22:28:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2020-11-16T22:28:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:28:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:28:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:28:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/1 2020-11-16T22:28:13.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2020-11-16T22:28:13.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:28:13.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:28:13.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:28:13.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2020-11-16T22:28:13.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2020-11-16T22:28:13.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:28:13.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:28:13.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2020-11-16T22:28:13.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:28:13.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:28:13.377 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:28:13.377 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:28:13.377 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:13.378 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:28:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:28:13.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:28:13.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2020-11-16T22:28:13.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2020-11-16T22:28:13.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:28:13.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:28:13.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:28:13.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:28:13.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:28:13.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/1 2020-11-16T22:28:13.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:28:13.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=69bf2707-5da5-4724-b9b1-e15e97be57e3 2020-11-16T22:28:13.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 69bf2707-5da5-4724-b9b1-e15e97be57e3' 2020-11-16T22:28:13.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:28:13.381 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 69bf2707-5da5-4724-b9b1-e15e97be57e3 2020-11-16T22:28:13.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD9/LJf2TBBGBAAou5I5Hw7NsjmsTc22fvz/w== 2020-11-16T22:28:13.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD9/LJf2TBBGBAAou5I5Hw7NsjmsTc22fvz/w=="}' 2020-11-16T22:28:13.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 69bf2707-5da5-4724-b9b1-e15e97be57e3 -i td/osd-recovery-prio/1/new.json 2020-11-16T22:28:13.997 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:28:14.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/1/new.json 2020-11-16T22:28:14.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD9/LJf2TBBGBAAou5I5Hw7NsjmsTc22fvz/w== --osd-uuid 69bf2707-5da5-4724-b9b1-e15e97be57e3 2020-11-16T22:28:14.040 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:14.049+0000 7f4492191d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:14.042 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:14.049+0000 7f4492191d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:14.044 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:14.053+0000 7f4492191d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:14.045 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:14.053+0000 7f4492191d80 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2020-11-16T22:28:14.045 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:14.053+0000 7f4492191d80 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2020-11-16T22:28:14.045 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:14.053+0000 7f4492191d80 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2020-11-16T22:28:14.097 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:14.105+0000 7f4492191d80 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2020-11-16T22:28:18.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2020-11-16T22:28:18.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:28:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:28:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:28:18.664 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:28:19.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:28:19.164 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:28:19.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:28:19.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:28:19.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:28:19.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:28:19.193 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:19.201+0000 7f27c5474d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:19.196 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:19.205+0000 7f27c5474d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:19.197 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:19.205+0000 7f27c5474d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:19.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:28:19.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:28:19.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:28:19.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:28:19.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:28:19.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:28:19.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:28:19.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:28:19.551 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:28:19.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:28:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:19.741+0000 7f27c5474d80 -1 Falling back to public interface 2020-11-16T22:28:19.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:28:20.590 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:20.593+0000 7f27c5474d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:28:20.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:28:20.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:28:20.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:28:20.933 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:28:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:28:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:28:21.338 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/700856,v1:127.0.0.1:6811/700856] [v2:127.0.0.1:6812/700856,v1:127.0.0.1:6813/700856] exists,up 69bf2707-5da5-4724-b9b1-e15e97be57e3 2020-11-16T22:28:21.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:28:21.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:28:21.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:28:21.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T22:28:21.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2020-11-16T22:28:21.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2020-11-16T22:28:21.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:28:21.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-11-16T22:28:21.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:28:21.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/2 2020-11-16T22:28:21.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2020-11-16T22:28:21.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:28:21.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:28:21.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:28:21.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2020-11-16T22:28:21.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2020-11-16T22:28:21.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:28:21.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:28:21.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2020-11-16T22:28:21.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:28:21.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:28:21.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:28:21.343 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:28:21.343 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:21.343 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:21.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:21.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:21.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:28:21.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:28:21.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:28:21.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2020-11-16T22:28:21.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2020-11-16T22:28:21.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:28:21.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:28:21.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:28:21.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:28:21.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:28:21.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/2 2020-11-16T22:28:21.345 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:28:21.346 INFO:tasks.workunit.client.0.smithi189.stdout:add osd2 a534df40-1cee-49b7-a541-e10c3e4a0176 2020-11-16T22:28:21.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a534df40-1cee-49b7-a541-e10c3e4a0176 2020-11-16T22:28:21.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 a534df40-1cee-49b7-a541-e10c3e4a0176' 2020-11-16T22:28:21.346 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:28:21.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAF/bJfDXglFhAA9ZAB061MkcOcrCRXrwMYdA== 2020-11-16T22:28:21.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAF/bJfDXglFhAA9ZAB061MkcOcrCRXrwMYdA=="}' 2020-11-16T22:28:21.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a534df40-1cee-49b7-a541-e10c3e4a0176 -i td/osd-recovery-prio/2/new.json 2020-11-16T22:28:21.945 INFO:tasks.workunit.client.0.smithi189.stdout:2 2020-11-16T22:28:21.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/2/new.json 2020-11-16T22:28:21.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAF/bJfDXglFhAA9ZAB061MkcOcrCRXrwMYdA== --osd-uuid a534df40-1cee-49b7-a541-e10c3e4a0176 2020-11-16T22:28:21.983 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:21.993+0000 7f930040ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:21.986 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:21.997+0000 7f930040ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:21.988 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:21.997+0000 7f930040ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:21.988 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:21.997+0000 7f930040ad80 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2020-11-16T22:28:21.988 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:21.997+0000 7f930040ad80 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2020-11-16T22:28:21.988 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:21.997+0000 7f930040ad80 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2020-11-16T22:28:22.042 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:22.049+0000 7f930040ad80 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2020-11-16T22:28:26.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2020-11-16T22:28:26.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:28:26.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-11-16T22:28:26.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:28:26.429 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd2 key to auth repository 2020-11-16T22:28:26.937 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.2 2020-11-16T22:28:26.937 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-11-16T22:28:26.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:28:26.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:28:26.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:28:26.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:28:26.963 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:26.973+0000 7fb70eeaad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:26.965 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:26.973+0000 7fb70eeaad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:26.967 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:26.977+0000 7fb70eeaad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:27.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-11-16T22:28:27.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:28:27.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-11-16T22:28:27.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:28:27.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:28:27.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:28:27.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:28:27.297 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:28:27.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:28:27.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:28:27.516 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:27.525+0000 7fb70eeaad80 -1 Falling back to public interface 2020-11-16T22:28:27.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:28:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:28.353+0000 7fb70eeaad80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-16T22:28:28.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:28:28.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:28:28.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:28:28.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:28:28.699 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:28:28.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-11-16T22:28:29.112 INFO:tasks.workunit.client.0.smithi189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/701850,v1:127.0.0.1:6819/701850] [v2:127.0.0.1:6820/701850,v1:127.0.0.1:6821/701850] exists,up a534df40-1cee-49b7-a541-e10c3e4a0176 2020-11-16T22:28:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:28:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:28:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:28:29.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T22:28:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2020-11-16T22:28:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2020-11-16T22:28:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:28:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-11-16T22:28:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:28:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/3 2020-11-16T22:28:29.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2020-11-16T22:28:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:28:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:28:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:28:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2020-11-16T22:28:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2020-11-16T22:28:29.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:28:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:28:29.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2020-11-16T22:28:29.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:28:29.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:28:29.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:28:29.118 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:28:29.118 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:29.118 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:29.118 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:29.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:29.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:28:29.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:28:29.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:28:29.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2020-11-16T22:28:29.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2020-11-16T22:28:29.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:28:29.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:28:29.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:28:29.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:28:29.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:28:29.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/3 2020-11-16T22:28:29.120 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:28:29.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b259497a-d14f-43cc-adb5-519d1036c306 2020-11-16T22:28:29.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 b259497a-d14f-43cc-adb5-519d1036c306' 2020-11-16T22:28:29.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:28:29.121 INFO:tasks.workunit.client.0.smithi189.stdout:add osd3 b259497a-d14f-43cc-adb5-519d1036c306 2020-11-16T22:28:29.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAN/bJfPjMwCRAAV3vz5fGJxmHRZKJ+Vk9l6A== 2020-11-16T22:28:29.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAN/bJfPjMwCRAAV3vz5fGJxmHRZKJ+Vk9l6A=="}' 2020-11-16T22:28:29.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b259497a-d14f-43cc-adb5-519d1036c306 -i td/osd-recovery-prio/3/new.json 2020-11-16T22:28:29.679 INFO:tasks.workunit.client.0.smithi189.stdout:3 2020-11-16T22:28:29.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/3/new.json 2020-11-16T22:28:29.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAN/bJfPjMwCRAAV3vz5fGJxmHRZKJ+Vk9l6A== --osd-uuid b259497a-d14f-43cc-adb5-519d1036c306 2020-11-16T22:28:29.726 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:29.733+0000 7f695e612d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:29.729 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:29.737+0000 7f695e612d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:29.732 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:29.741+0000 7f695e612d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:29.732 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:29.741+0000 7f695e612d80 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2020-11-16T22:28:29.732 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:29.741+0000 7f695e612d80 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2020-11-16T22:28:29.732 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:29.741+0000 7f695e612d80 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2020-11-16T22:28:29.795 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:29.805+0000 7f695e612d80 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2020-11-16T22:28:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2020-11-16T22:28:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:28:34.418 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd3 key to auth repository 2020-11-16T22:28:34.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-11-16T22:28:34.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:28:34.922 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.3 2020-11-16T22:28:34.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-11-16T22:28:34.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:28:34.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:28:34.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:28:34.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:28:34.950 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:34.956+0000 7fe7c3de7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:34.952 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:34.960+0000 7fe7c3de7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:34.954 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:34.964+0000 7fe7c3de7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:35.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-11-16T22:28:35.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:28:35.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-11-16T22:28:35.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:28:35.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:28:35.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:28:35.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:28:35.300 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:28:35.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:28:35.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:28:35.500 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:35.508+0000 7fe7c3de7d80 -1 Falling back to public interface 2020-11-16T22:28:35.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:28:36.372 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:36.376+0000 7fe7c3de7d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-16T22:28:36.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:28:36.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:28:36.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:28:36.699 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:28:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:28:36.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-11-16T22:28:37.087 INFO:tasks.workunit.client.0.smithi189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/702844,v1:127.0.0.1:6827/702844] [v2:127.0.0.1:6828/702844,v1:127.0.0.1:6829/702844] exists,up b259497a-d14f-43cc-adb5-519d1036c306 2020-11-16T22:28:37.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:28:37.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:28:37.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:28:37.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T22:28:37.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2020-11-16T22:28:37.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2020-11-16T22:28:37.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:28:37.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-11-16T22:28:37.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:28:37.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/4 2020-11-16T22:28:37.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2020-11-16T22:28:37.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:28:37.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:28:37.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:28:37.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2020-11-16T22:28:37.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2020-11-16T22:28:37.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:28:37.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:28:37.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2020-11-16T22:28:37.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:28:37.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:28:37.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:28:37.092 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:28:37.093 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:28:37.093 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:28:37.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:37.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' 2020-11-16T22:28:37.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:28:37.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:28:37.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:28:37.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2020-11-16T22:28:37.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2020-11-16T22:28:37.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:28:37.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:28:37.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:28:37.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:28:37.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:28:37.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/4 2020-11-16T22:28:37.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:28:37.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=833e6751-dc26-4d42-84bf-30c0d7b3b017 2020-11-16T22:28:37.095 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 833e6751-dc26-4d42-84bf-30c0d7b3b017' 2020-11-16T22:28:37.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:28:37.096 INFO:tasks.workunit.client.0.smithi189.stdout:add osd4 833e6751-dc26-4d42-84bf-30c0d7b3b017 2020-11-16T22:28:37.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAV/bJfcEOQBxAAAGJCnHnC9YSexdSIEJ2nhg== 2020-11-16T22:28:37.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAV/bJfcEOQBxAAAGJCnHnC9YSexdSIEJ2nhg=="}' 2020-11-16T22:28:37.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 833e6751-dc26-4d42-84bf-30c0d7b3b017 -i td/osd-recovery-prio/4/new.json 2020-11-16T22:28:37.677 INFO:tasks.workunit.client.0.smithi189.stdout:4 2020-11-16T22:28:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/4/new.json 2020-11-16T22:28:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAV/bJfcEOQBxAAAGJCnHnC9YSexdSIEJ2nhg== --osd-uuid 833e6751-dc26-4d42-84bf-30c0d7b3b017 2020-11-16T22:28:37.721 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:37.728+0000 7f78b1e62d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:37.724 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:37.732+0000 7f78b1e62d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:37.726 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:37.732+0000 7f78b1e62d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:37.726 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:37.736+0000 7f78b1e62d80 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2020-11-16T22:28:37.726 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:37.736+0000 7f78b1e62d80 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2020-11-16T22:28:37.727 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:37.736+0000 7f78b1e62d80 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2020-11-16T22:28:37.790 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:37.800+0000 7f78b1e62d80 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2020-11-16T22:28:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2020-11-16T22:28:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:28:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-11-16T22:28:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:28:42.456 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd4 key to auth repository 2020-11-16T22:28:42.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-11-16T22:28:42.942 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.4 2020-11-16T22:28:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:28:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:28:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=3fb07799-71a9-4175-8217-4a9e4f8628ec --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.679603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:28:42.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:28:42.973 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:42.980+0000 7ff99cd18d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:42.975 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:42.984+0000 7ff99cd18d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:42.977 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:42.984+0000 7ff99cd18d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:28:43.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-11-16T22:28:43.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:28:43.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-11-16T22:28:43.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:28:43.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:28:43.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:28:43.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:28:43.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:28:43.326 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:28:43.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T22:28:43.508 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:43.516+0000 7ff99cd18d80 -1 Falling back to public interface 2020-11-16T22:28:43.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:28:44.387 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:28:44.396+0000 7ff99cd18d80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-16T22:28:44.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:28:44.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:28:44.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:28:44.704 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:28:44.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:28:44.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-11-16T22:28:45.081 INFO:tasks.workunit.client.0.smithi189.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/703838,v1:127.0.0.1:6835/703838] [v2:127.0.0.1:6836/703838,v1:127.0.0.1:6837/703838] exists,up 833e6751-dc26-4d42-84bf-30c0d7b3b017 2020-11-16T22:28:45.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:28:45.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:28:45.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:28:45.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: seq 1 10 2020-11-16T22:28:45.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:28:45.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test1 1 1 2020-11-16T22:28:45.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2020-11-16T22:28:45.657 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' already exists 2020-11-16T22:28:45.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:28:46.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test1 size 2 2020-11-16T22:28:47.421 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 2 2020-11-16T22:28:47.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:28:47.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test2 1 1 2020-11-16T22:28:47.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2020-11-16T22:28:48.046 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' already exists 2020-11-16T22:28:48.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:28:49.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test2 size 2 2020-11-16T22:28:49.851 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 2 size to 2 2020-11-16T22:28:49.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:28:49.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test3 1 1 2020-11-16T22:28:49.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2020-11-16T22:28:50.466 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' already exists 2020-11-16T22:28:50.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:28:51.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test3 size 2 2020-11-16T22:28:52.362 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 2 2020-11-16T22:28:52.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:28:52.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test4 1 1 2020-11-16T22:28:52.377 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test4 1 1 2020-11-16T22:28:53.133 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test4' already exists 2020-11-16T22:28:53.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:28:54.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test4 size 2 2020-11-16T22:28:54.932 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 2 2020-11-16T22:28:54.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:28:54.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test5 1 1 2020-11-16T22:28:54.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test5 1 1 2020-11-16T22:28:55.711 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test5' already exists 2020-11-16T22:28:55.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:28:56.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test5 size 2 2020-11-16T22:28:57.508 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 5 size to 2 2020-11-16T22:28:57.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:28:57.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test6 1 1 2020-11-16T22:28:57.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test6 1 1 2020-11-16T22:28:58.189 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test6' already exists 2020-11-16T22:28:58.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:28:59.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test6 size 2 2020-11-16T22:29:00.221 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 6 size to 2 2020-11-16T22:29:00.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:00.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test7 1 1 2020-11-16T22:29:00.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test7 1 1 2020-11-16T22:29:00.961 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test7' already exists 2020-11-16T22:29:00.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:29:01.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test7 size 2 2020-11-16T22:29:02.782 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 7 size to 2 2020-11-16T22:29:02.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:02.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test8 1 1 2020-11-16T22:29:02.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test8 1 1 2020-11-16T22:29:03.369 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test8' already exists 2020-11-16T22:29:03.379 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:29:04.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test8 size 2 2020-11-16T22:29:05.190 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 8 size to 2 2020-11-16T22:29:05.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:05.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test9 1 1 2020-11-16T22:29:05.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test9 1 1 2020-11-16T22:29:05.917 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test9' already exists 2020-11-16T22:29:05.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:29:06.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test9 size 2 2020-11-16T22:29:07.712 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 9 size to 2 2020-11-16T22:29:07.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:07.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test10 1 1 2020-11-16T22:29:07.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test10 1 1 2020-11-16T22:29:08.454 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test10' already exists 2020-11-16T22:29:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:29:09.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test10 size 2 2020-11-16T22:29:10.642 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 10 size to 2 2020-11-16T22:29:10.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: sleep 5 2020-11-16T22:29:15.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:65: TEST_recovery_priority: wait_for_clean 2020-11-16T22:29:15.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:29:15.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:29:15.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:29:15.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:29:15.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:29:15.660 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:29:15.661 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:29:15.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:29:15.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:29:15.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:29:15.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:29:15.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:29:15.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:29:15.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:29:15.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:29:16.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:29:16.270 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:29:16.270 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:29:16.270 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:29:16.270 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T22:29:16.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:29:16.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:29:16.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:29:16.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836496 2020-11-16T22:29:16.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836496 2020-11-16T22:29:16.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496' 2020-11-16T22:29:16.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:29:16.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:29:16.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672973 2020-11-16T22:29:16.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672973 2020-11-16T22:29:16.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672973' 2020-11-16T22:29:16.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:29:16.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:29:16.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2020-11-16T22:29:16.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2020-11-16T22:29:16.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509452' 2020-11-16T22:29:16.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:29:16.786 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:29:16.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378634 2020-11-16T22:29:16.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378634 2020-11-16T22:29:16.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509452 3-81604378634' 2020-11-16T22:29:16.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:29:16.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:29:17.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247817 2020-11-16T22:29:17.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247817 2020-11-16T22:29:17.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509452 3-81604378634 4-98784247817' 2020-11-16T22:29:17.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:29:17.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836496 2020-11-16T22:29:17.121 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:29:17.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:29:17.123 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2020-11-16T22:29:17.124 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:29:17.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836496 2020-11-16T22:29:17.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2020-11-16T22:29:17.126 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836496 2020-11-16T22:29:17.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:29:17.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836496 2020-11-16T22:29:17.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:29:17.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672973 2020-11-16T22:29:17.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:29:17.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:29:17.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672973 2020-11-16T22:29:17.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:29:17.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672973 2020-11-16T22:29:17.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2020-11-16T22:29:17.522 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672973 2020-11-16T22:29:17.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:29:17.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672973 2020-11-16T22:29:17.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:29:17.909 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2020-11-16T22:29:17.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:29:17.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:29:17.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2020-11-16T22:29:17.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:29:17.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2020-11-16T22:29:17.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2020-11-16T22:29:17.915 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509452 2020-11-16T22:29:17.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:29:18.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509452 -lt 64424509452 2020-11-16T22:29:18.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:29:18.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378634 2020-11-16T22:29:18.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:29:18.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:29:18.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378634 2020-11-16T22:29:18.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:29:18.280 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378634 2020-11-16T22:29:18.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378634 2020-11-16T22:29:18.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378634' 2020-11-16T22:29:18.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:29:18.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378634 -lt 81604378634 2020-11-16T22:29:18.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:29:18.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247817 2020-11-16T22:29:18.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:29:18.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:29:18.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247817 2020-11-16T22:29:18.675 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:29:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247817 2020-11-16T22:29:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247817' 2020-11-16T22:29:18.676 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247817 2020-11-16T22:29:18.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:29:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247817 -lt 98784247817 2020-11-16T22:29:19.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:29:19.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:29:19.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:29:19.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 10 == 0 2020-11-16T22:29:19.629 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:29:19.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:29:19.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:29:19.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:29:19.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:29:19.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:29:19.630 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:29:20.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=10 2020-11-16T22:29:20.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:29:20.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:29:20.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:29:20.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 10 = 10 2020-11-16T22:29:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:29:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:29:20.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:67: TEST_recovery_priority: ceph pg dump pgs 2020-11-16T22:29:20.846 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T22:29:20.846 INFO:tasks.workunit.client.0.smithi189.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:29:11.291176+0000 0'0 79:20 [2,1] 2 [2,1] 2 0'0 2020-11-16T22:29:08.205094+0000 0'0 2020-11-16T22:29:08.205094+0000 0 2020-11-16T22:29:20.846 INFO:tasks.workunit.client.0.smithi189.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:29:08.384305+0000 0'0 77:23 [1,2] 1 [1,2] 1 0'0 2020-11-16T22:29:05.659310+0000 0'0 2020-11-16T22:29:05.659310+0000 0 2020-11-16T22:29:20.847 INFO:tasks.workunit.client.0.smithi189.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:29:06.389020+0000 0'0 76:26 [3,4] 3 [3,4] 3 0'0 2020-11-16T22:29:03.197149+0000 0'0 2020-11-16T22:29:03.197149+0000 0 2020-11-16T22:29:20.847 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:29:03.674662+0000 0'0 77:29 [1,4] 1 [1,4] 1 0'0 2020-11-16T22:29:00.696185+0000 0'0 2020-11-16T22:29:00.696185+0000 0 2020-11-16T22:29:20.847 INFO:tasks.workunit.client.0.smithi189.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:28:50.712513+0000 0'0 76:42 [3,1] 3 [3,1] 3 0'0 2020-11-16T22:28:47.865239+0000 0'0 2020-11-16T22:28:47.865239+0000 0 2020-11-16T22:29:20.847 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:28:48.277918+0000 0'0 77:58 [1,0] 1 [1,0] 1 0'0 2020-11-16T22:28:45.457031+0000 0'0 2020-11-16T22:28:45.457031+0000 0 2020-11-16T22:29:20.847 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:28:52.737656+0000 0'0 77:51 [1,2] 1 [1,2] 1 0'0 2020-11-16T22:28:50.266785+0000 0'0 2020-11-16T22:28:50.266785+0000 0 2020-11-16T22:29:20.847 INFO:tasks.workunit.client.0.smithi189.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:28:55.852867+0000 0'0 78:45 [4,0] 4 [4,0] 4 0'0 2020-11-16T22:28:52.821340+0000 0'0 2020-11-16T22:28:52.821340+0000 0 2020-11-16T22:29:20.847 INFO:tasks.workunit.client.0.smithi189.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:28:58.405595+0000 0'0 76:37 [3,4] 3 [3,4] 3 0'0 2020-11-16T22:28:55.468356+0000 0'0 2020-11-16T22:28:55.468356+0000 0 2020-11-16T22:29:20.848 INFO:tasks.workunit.client.0.smithi189.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-11-16T22:29:01.316387+0000 0'0 78:37 [4,2] 4 [4,2] 4 0'0 2020-11-16T22:28:57.933438+0000 0'0 2020-11-16T22:28:57.933438+0000 0 2020-11-16T22:29:20.848 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T22:29:20.848 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T22:29:20.848 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T22:29:20.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:71: TEST_recovery_priority: local PG1 2020-11-16T22:29:20.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:72: TEST_recovery_priority: local POOLNUM1 2020-11-16T22:29:20.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:73: TEST_recovery_priority: local pool1 2020-11-16T22:29:20.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local chk_osd1_1 2020-11-16T22:29:20.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local chk_osd1_2 2020-11-16T22:29:20.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local PG2 2020-11-16T22:29:20.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local POOLNUM2 2020-11-16T22:29:20.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:79: TEST_recovery_priority: local pool2 2020-11-16T22:29:20.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local chk_osd2 2020-11-16T22:29:20.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local PG3 2020-11-16T22:29:20.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local POOLNUM3 2020-11-16T22:29:20.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:84: TEST_recovery_priority: local pool3 2020-11-16T22:29:20.860 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: seq 1 10 2020-11-16T22:29:20.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:20.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 1.0 --format=json 2020-11-16T22:29:20.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2020-11-16T22:29:21.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2020-11-16T22:29:21.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2020-11-16T22:29:21.224 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2020-11-16T22:29:21.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=0 2020-11-16T22:29:21.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z '' ']' 2020-11-16T22:29:21.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: PG1=1.0 2020-11-16T22:29:21.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: POOLNUM1=1 2020-11-16T22:29:21.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:95: TEST_recovery_priority: pool1=test1 2020-11-16T22:29:21.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: chk_osd1_1=1 2020-11-16T22:29:21.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: chk_osd1_2=0 2020-11-16T22:29:21.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:21.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 2.0 --format=json 2020-11-16T22:29:21.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2020-11-16T22:29:21.602 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2020-11-16T22:29:21.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=3 2020-11-16T22:29:21.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2020-11-16T22:29:21.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=1 2020-11-16T22:29:21.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2020-11-16T22:29:21.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2020-11-16T22:29:21.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2020-11-16T22:29:21.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:21.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 3.0 --format=json 2020-11-16T22:29:21.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2020-11-16T22:29:21.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2020-11-16T22:29:21.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2020-11-16T22:29:21.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2020-11-16T22:29:21.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=2 2020-11-16T22:29:21.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2020-11-16T22:29:21.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2020-11-16T22:29:21.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: PG2=3.0 2020-11-16T22:29:21.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: POOLNUM2=3 2020-11-16T22:29:21.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:102: TEST_recovery_priority: pool2=test3 2020-11-16T22:29:21.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: chk_osd2=2 2020-11-16T22:29:21.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:21.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 4.0 --format=json 2020-11-16T22:29:21.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2020-11-16T22:29:22.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2020-11-16T22:29:22.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=4 2020-11-16T22:29:22.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2020-11-16T22:29:22.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=0 2020-11-16T22:29:22.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2020-11-16T22:29:22.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2020-11-16T22:29:22.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2020-11-16T22:29:22.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:22.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 5.0 --format=json 2020-11-16T22:29:22.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2020-11-16T22:29:22.766 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2020-11-16T22:29:22.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=3 2020-11-16T22:29:22.768 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2020-11-16T22:29:22.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=4 2020-11-16T22:29:22.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2020-11-16T22:29:22.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2020-11-16T22:29:22.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-11-16T22:29:22.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:22.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 6.0 --format=json 2020-11-16T22:29:22.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2020-11-16T22:29:23.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2020-11-16T22:29:23.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=4 2020-11-16T22:29:23.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2020-11-16T22:29:23.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=2 2020-11-16T22:29:23.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2020-11-16T22:29:23.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2020-11-16T22:29:23.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2020-11-16T22:29:23.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:23.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 7.0 --format=json 2020-11-16T22:29:23.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2020-11-16T22:29:23.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2020-11-16T22:29:23.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2020-11-16T22:29:23.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2020-11-16T22:29:23.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=4 2020-11-16T22:29:23.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2020-11-16T22:29:23.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2020-11-16T22:29:23.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-11-16T22:29:23.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: PG3=7.0 2020-11-16T22:29:23.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: POOLNUM3=7 2020-11-16T22:29:23.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:108: TEST_recovery_priority: pool3=test7 2020-11-16T22:29:23.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: break 2020-11-16T22:29:23.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2020-11-16T22:29:23.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:114: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2020-11-16T22:29:23.526 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: seq 1 10 2020-11-16T22:29:23.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:23.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2020-11-16T22:29:23.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:23.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2020-11-16T22:29:23.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test2 2020-11-16T22:29:23.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2020-11-16T22:29:23.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-11-16T22:29:24.048 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test2' does not exist 2020-11-16T22:29:24.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:24.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2020-11-16T22:29:24.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:24.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2020-11-16T22:29:24.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test4 2020-11-16T22:29:24.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test4 2020-11-16T22:29:24.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-11-16T22:29:24.621 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test4' does not exist 2020-11-16T22:29:24.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:24.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2020-11-16T22:29:24.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test5 2020-11-16T22:29:24.634 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test5 2020-11-16T22:29:24.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-11-16T22:29:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test5' does not exist 2020-11-16T22:29:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2020-11-16T22:29:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test6 2020-11-16T22:29:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test6 2020-11-16T22:29:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-11-16T22:29:25.863 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test6' does not exist 2020-11-16T22:29:25.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:25.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2020-11-16T22:29:25.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:25.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2020-11-16T22:29:25.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test8 2020-11-16T22:29:25.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test8 2020-11-16T22:29:25.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-11-16T22:29:26.445 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test8' does not exist 2020-11-16T22:29:26.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:26.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2020-11-16T22:29:26.455 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test9 2020-11-16T22:29:26.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test9 2020-11-16T22:29:26.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-11-16T22:29:27.004 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test9' does not exist 2020-11-16T22:29:27.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2020-11-16T22:29:27.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2020-11-16T22:29:27.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test10 2020-11-16T22:29:27.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test10 2020-11-16T22:29:27.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-11-16T22:29:27.576 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test10' does not exist 2020-11-16T22:29:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:128: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2020-11-16T22:29:28.435 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 1 2020-11-16T22:29:28.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:129: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2020-11-16T22:29:29.180 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 7 size to 1 2020-11-16T22:29:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:130: TEST_recovery_priority: wait_for_clean 2020-11-16T22:29:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:29:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:29:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:29:29.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:29:29.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:29:29.197 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:29:29.198 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:29:29.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:29:29.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:29:29.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:29:29.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:29:29.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:29:29.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:29:29.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:29:29.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:29:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:29:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:29:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:29:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:29:29.782 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T22:29:29.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:29:29.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:29:29.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:29:29.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836501 2020-11-16T22:29:29.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836501 2020-11-16T22:29:29.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501' 2020-11-16T22:29:29.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:29:29.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:29:30.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672978 2020-11-16T22:29:30.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672978 2020-11-16T22:29:30.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672978' 2020-11-16T22:29:30.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:29:30.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:29:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509457 2020-11-16T22:29:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509457 2020-11-16T22:29:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509457' 2020-11-16T22:29:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:29:30.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:29:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378639 2020-11-16T22:29:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378639 2020-11-16T22:29:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509457 3-81604378639' 2020-11-16T22:29:30.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:29:30.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:29:30.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247821 2020-11-16T22:29:30.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247821 2020-11-16T22:29:30.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509457 3-81604378639 4-98784247821' 2020-11-16T22:29:30.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:29:30.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836501 2020-11-16T22:29:30.619 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:29:30.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:29:30.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836501 2020-11-16T22:29:30.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:29:30.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836501 2020-11-16T22:29:30.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2020-11-16T22:29:30.623 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836501 2020-11-16T22:29:30.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:29:31.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836499 -lt 21474836501 2020-11-16T22:29:31.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:29:32.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:29:32.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:29:32.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836501 -lt 21474836501 2020-11-16T22:29:32.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:29:32.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:29:32.405 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672978 2020-11-16T22:29:32.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:29:32.407 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672978 2020-11-16T22:29:32.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:29:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672978 2020-11-16T22:29:32.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2020-11-16T22:29:32.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:29:32.410 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949672978 2020-11-16T22:29:32.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672978 -lt 42949672978 2020-11-16T22:29:32.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:29:32.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509457 2020-11-16T22:29:32.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:29:32.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:29:32.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509457 2020-11-16T22:29:32.778 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:29:32.779 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509457 2020-11-16T22:29:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509457 2020-11-16T22:29:32.780 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2020-11-16T22:29:32.780 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:29:33.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509457 -lt 64424509457 2020-11-16T22:29:33.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:29:33.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378639 2020-11-16T22:29:33.163 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:29:33.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:29:33.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378639 2020-11-16T22:29:33.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:29:33.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378639 2020-11-16T22:29:33.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378639' 2020-11-16T22:29:33.169 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378639 2020-11-16T22:29:33.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:29:33.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378640 -lt 81604378639 2020-11-16T22:29:33.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:29:33.553 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247821 2020-11-16T22:29:33.554 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:29:33.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:29:33.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247821 2020-11-16T22:29:33.556 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:29:33.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247821 2020-11-16T22:29:33.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247821' 2020-11-16T22:29:33.557 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247821 2020-11-16T22:29:33.558 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:29:33.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247822 -lt 98784247821 2020-11-16T22:29:33.947 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:29:33.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:29:33.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:29:34.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2020-11-16T22:29:34.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:29:34.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:29:34.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:29:34.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:29:34.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:29:34.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:29:34.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:29:34.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2020-11-16T22:29:34.798 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:29:34.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:29:34.799 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:29:35.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2020-11-16T22:29:35.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:29:35.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:29:35.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2020-11-16T22:29:35.336 INFO:tasks.workunit.client.0.smithi189.stderr:10+0 records in 2020-11-16T22:29:35.336 INFO:tasks.workunit.client.0.smithi189.stderr:10+0 records out 2020-11-16T22:29:35.337 INFO:tasks.workunit.client.0.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0543523 s, 193 MB/s 2020-11-16T22:29:35.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: p=1 2020-11-16T22:29:35.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2020-11-16T22:29:35.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2020-11-16T22:29:35.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:35.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2020-11-16T22:29:35.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:35.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2020-11-16T22:29:36.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:36.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2020-11-16T22:29:37.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:37.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2020-11-16T22:29:37.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:37.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2020-11-16T22:29:38.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:38.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2020-11-16T22:29:39.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:39.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2020-11-16T22:29:40.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:40.391 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2020-11-16T22:29:40.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:40.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2020-11-16T22:29:41.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:41.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2020-11-16T22:29:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2020-11-16T22:29:43.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:43.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2020-11-16T22:29:43.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:43.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2020-11-16T22:29:44.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:44.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2020-11-16T22:29:45.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:45.277 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2020-11-16T22:29:45.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:45.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2020-11-16T22:29:46.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:46.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2020-11-16T22:29:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2020-11-16T22:29:48.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:48.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2020-11-16T22:29:48.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:48.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2020-11-16T22:29:49.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:49.721 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2020-11-16T22:29:50.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:50.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2020-11-16T22:29:51.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:51.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2020-11-16T22:29:51.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:51.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2020-11-16T22:29:53.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:53.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2020-11-16T22:29:54.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:54.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2020-11-16T22:29:54.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:54.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2020-11-16T22:29:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:55.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2020-11-16T22:29:56.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2020-11-16T22:29:56.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:56.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2020-11-16T22:29:57.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:57.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2020-11-16T22:29:58.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:58.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2020-11-16T22:29:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:58.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2020-11-16T22:29:59.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:29:59.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2020-11-16T22:30:00.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:00.400 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2020-11-16T22:30:00.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:00.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2020-11-16T22:30:01.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:01.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2020-11-16T22:30:02.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:02.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2020-11-16T22:30:03.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:03.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2020-11-16T22:30:03.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:03.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2020-11-16T22:30:04.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:04.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2020-11-16T22:30:05.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:05.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2020-11-16T22:30:06.341 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:06.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2020-11-16T22:30:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:06.868 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2020-11-16T22:30:07.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:07.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2020-11-16T22:30:08.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:08.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2020-11-16T22:30:09.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:09.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2020-11-16T22:30:11.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:11.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2020-11-16T22:30:11.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:11.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2020-11-16T22:30:12.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:12.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2020-11-16T22:30:13.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:13.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2020-11-16T22:30:13.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:13.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2020-11-16T22:30:14.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:14.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2020-11-16T22:30:15.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:15.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2020-11-16T22:30:15.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:15.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2020-11-16T22:30:16.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:16.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2020-11-16T22:30:17.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:17.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2020-11-16T22:30:17.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:17.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2020-11-16T22:30:18.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:18.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2020-11-16T22:30:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2020-11-16T22:30:20.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:20.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2020-11-16T22:30:20.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:20.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2020-11-16T22:30:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2020-11-16T22:30:22.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:22.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2020-11-16T22:30:23.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:23.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2020-11-16T22:30:23.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:23.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2020-11-16T22:30:24.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:24.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2020-11-16T22:30:25.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:25.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2020-11-16T22:30:26.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:26.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2020-11-16T22:30:26.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:26.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2020-11-16T22:30:27.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:27.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2020-11-16T22:30:29.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:29.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2020-11-16T22:30:29.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:29.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2020-11-16T22:30:30.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:30.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2020-11-16T22:30:31.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:31.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2020-11-16T22:30:32.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:32.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2020-11-16T22:30:32.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:32.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2020-11-16T22:30:33.524 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:33.525 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2020-11-16T22:30:34.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:34.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2020-11-16T22:30:34.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:34.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2020-11-16T22:30:35.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:35.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2020-11-16T22:30:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:36.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2020-11-16T22:30:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:36.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2020-11-16T22:30:37.859 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:37.860 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2020-11-16T22:30:38.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:38.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2020-11-16T22:30:38.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:38.936 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2020-11-16T22:30:39.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:39.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2020-11-16T22:30:40.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:40.706 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2020-11-16T22:30:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:41.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2020-11-16T22:30:41.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:41.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2020-11-16T22:30:42.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:42.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2020-11-16T22:30:43.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:43.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2020-11-16T22:30:44.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:44.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2020-11-16T22:30:44.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:44.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2020-11-16T22:30:46.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:46.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2020-11-16T22:30:48.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:48.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2020-11-16T22:30:50.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:50.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2020-11-16T22:30:52.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:52.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2020-11-16T22:30:52.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:52.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2020-11-16T22:30:53.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:53.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2020-11-16T22:30:53.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:53.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2020-11-16T22:30:54.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:54.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2020-11-16T22:30:55.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:55.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2020-11-16T22:30:56.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:56.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2020-11-16T22:30:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2020-11-16T22:30:58.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:58.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2020-11-16T22:30:59.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:59.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2020-11-16T22:30:59.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:30:59.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2020-11-16T22:31:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2020-11-16T22:31:01.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:01.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2020-11-16T22:31:01.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:01.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2020-11-16T22:31:02.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:02.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2020-11-16T22:31:03.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:03.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2020-11-16T22:31:03.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:03.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2020-11-16T22:31:04.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:04.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2020-11-16T22:31:05.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:05.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2020-11-16T22:31:05.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:05.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2020-11-16T22:31:06.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:06.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2020-11-16T22:31:08.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:08.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2020-11-16T22:31:09.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:09.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2020-11-16T22:31:09.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:09.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2020-11-16T22:31:10.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:10.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2020-11-16T22:31:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:11.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2020-11-16T22:31:12.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:12.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2020-11-16T22:31:12.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:12.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2020-11-16T22:31:13.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:13.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2020-11-16T22:31:14.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:14.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2020-11-16T22:31:14.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:14.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2020-11-16T22:31:15.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:15.785 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2020-11-16T22:31:16.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:16.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2020-11-16T22:31:17.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:17.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2020-11-16T22:31:17.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:17.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2020-11-16T22:31:18.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:18.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2020-11-16T22:31:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2020-11-16T22:31:20.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:20.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2020-11-16T22:31:20.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:20.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2020-11-16T22:31:21.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:21.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2020-11-16T22:31:22.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:22.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2020-11-16T22:31:23.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:23.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2020-11-16T22:31:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2020-11-16T22:31:24.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:24.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2020-11-16T22:31:26.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:26.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2020-11-16T22:31:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:26.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2020-11-16T22:31:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2020-11-16T22:31:28.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:28.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2020-11-16T22:31:28.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:28.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2020-11-16T22:31:29.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:29.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2020-11-16T22:31:30.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:30.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2020-11-16T22:31:30.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:30.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2020-11-16T22:31:31.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:31.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2020-11-16T22:31:32.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:32.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2020-11-16T22:31:32.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:32.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2020-11-16T22:31:33.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:33.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2020-11-16T22:31:34.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:34.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2020-11-16T22:31:35.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:35.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2020-11-16T22:31:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:35.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2020-11-16T22:31:36.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:36.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2020-11-16T22:31:37.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:37.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2020-11-16T22:31:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2020-11-16T22:31:38.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:38.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2020-11-16T22:31:39.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:39.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2020-11-16T22:31:40.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:40.378 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2020-11-16T22:31:41.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:41.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2020-11-16T22:31:41.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:41.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2020-11-16T22:31:42.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2020-11-16T22:31:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2020-11-16T22:31:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:44.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2020-11-16T22:31:45.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:45.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2020-11-16T22:31:46.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:46.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2020-11-16T22:31:46.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:46.829 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2020-11-16T22:31:47.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:47.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2020-11-16T22:31:48.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:48.417 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2020-11-16T22:31:49.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:49.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2020-11-16T22:31:49.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:49.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2020-11-16T22:31:50.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:50.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2020-11-16T22:31:51.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:51.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2020-11-16T22:31:52.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:52.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2020-11-16T22:31:52.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:52.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2020-11-16T22:31:53.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:53.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2020-11-16T22:31:54.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:54.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2020-11-16T22:31:55.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:55.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2020-11-16T22:31:55.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:55.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2020-11-16T22:31:56.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:56.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2020-11-16T22:31:57.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:57.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2020-11-16T22:31:57.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:57.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2020-11-16T22:31:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:58.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2020-11-16T22:31:59.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:31:59.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2020-11-16T22:32:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2020-11-16T22:32:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2020-11-16T22:32:03.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:03.892 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2020-11-16T22:32:06.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:06.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2020-11-16T22:32:07.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:07.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2020-11-16T22:32:07.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:07.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2020-11-16T22:32:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:08.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2020-11-16T22:32:09.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:09.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2020-11-16T22:32:10.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:10.226 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2020-11-16T22:32:10.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:10.774 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2020-11-16T22:32:11.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:11.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2020-11-16T22:32:12.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:12.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2020-11-16T22:32:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:13.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2020-11-16T22:32:13.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 1 + 1 2020-11-16T22:32:13.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=2 2020-11-16T22:32:13.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2020-11-16T22:32:13.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2020-11-16T22:32:13.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:13.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2020-11-16T22:32:13.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:13.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2020-11-16T22:32:14.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:14.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2020-11-16T22:32:14.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:14.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2020-11-16T22:32:15.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:15.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2020-11-16T22:32:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2020-11-16T22:32:16.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:16.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2020-11-16T22:32:17.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:17.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2020-11-16T22:32:17.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:17.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2020-11-16T22:32:17.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:17.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2020-11-16T22:32:18.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:18.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2020-11-16T22:32:18.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:18.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2020-11-16T22:32:20.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:20.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2020-11-16T22:32:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:20.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2020-11-16T22:32:20.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:20.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2020-11-16T22:32:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:20.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2020-11-16T22:32:21.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:21.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2020-11-16T22:32:22.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:22.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2020-11-16T22:32:22.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:22.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2020-11-16T22:32:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:22.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2020-11-16T22:32:23.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:23.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2020-11-16T22:32:23.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:23.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2020-11-16T22:32:23.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:23.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2020-11-16T22:32:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2020-11-16T22:32:24.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:24.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2020-11-16T22:32:25.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:25.419 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2020-11-16T22:32:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:25.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2020-11-16T22:32:26.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:26.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2020-11-16T22:32:26.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:26.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2020-11-16T22:32:26.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:26.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2020-11-16T22:32:27.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:27.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2020-11-16T22:32:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2020-11-16T22:32:28.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:28.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2020-11-16T22:32:28.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:28.626 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2020-11-16T22:32:28.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:28.943 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2020-11-16T22:32:29.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:29.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2020-11-16T22:32:30.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:30.071 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2020-11-16T22:32:30.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:30.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2020-11-16T22:32:31.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:31.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2020-11-16T22:32:31.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:31.616 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2020-11-16T22:32:31.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:31.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2020-11-16T22:32:32.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:32.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2020-11-16T22:32:32.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:32.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2020-11-16T22:32:32.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:32.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2020-11-16T22:32:33.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:33.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2020-11-16T22:32:33.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:33.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2020-11-16T22:32:34.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:34.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2020-11-16T22:32:34.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:34.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2020-11-16T22:32:35.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:35.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2020-11-16T22:32:35.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:35.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2020-11-16T22:32:35.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:35.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2020-11-16T22:32:36.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:36.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2020-11-16T22:32:36.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:36.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2020-11-16T22:32:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:37.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2020-11-16T22:32:37.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:37.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2020-11-16T22:32:38.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:38.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2020-11-16T22:32:38.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:38.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2020-11-16T22:32:38.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:38.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2020-11-16T22:32:39.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:39.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2020-11-16T22:32:40.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:40.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2020-11-16T22:32:40.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:40.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2020-11-16T22:32:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2020-11-16T22:32:41.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:41.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2020-11-16T22:32:41.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:41.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2020-11-16T22:32:42.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:42.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2020-11-16T22:32:42.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:42.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2020-11-16T22:32:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:43.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2020-11-16T22:32:43.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:43.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2020-11-16T22:32:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:44.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2020-11-16T22:32:44.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:44.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2020-11-16T22:32:44.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:44.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2020-11-16T22:32:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2020-11-16T22:32:45.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:45.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2020-11-16T22:32:46.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:46.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2020-11-16T22:32:46.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:46.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2020-11-16T22:32:47.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:47.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2020-11-16T22:32:47.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:47.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2020-11-16T22:32:47.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:47.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2020-11-16T22:32:48.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:48.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2020-11-16T22:32:48.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:48.413 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2020-11-16T22:32:49.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:49.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2020-11-16T22:32:49.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:49.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2020-11-16T22:32:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:50.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2020-11-16T22:32:51.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:51.753 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2020-11-16T22:32:52.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:52.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2020-11-16T22:32:53.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:53.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2020-11-16T22:32:53.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:53.622 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2020-11-16T22:32:53.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:53.957 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2020-11-16T22:32:54.264 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:54.265 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2020-11-16T22:32:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2020-11-16T22:32:54.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:54.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2020-11-16T22:32:55.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:55.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2020-11-16T22:32:56.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:56.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2020-11-16T22:32:56.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:56.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2020-11-16T22:32:56.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:56.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2020-11-16T22:32:57.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:57.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2020-11-16T22:32:57.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:57.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2020-11-16T22:32:57.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:57.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2020-11-16T22:32:58.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:58.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2020-11-16T22:32:59.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:59.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2020-11-16T22:32:59.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:59.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2020-11-16T22:32:59.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:32:59.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2020-11-16T22:33:00.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:00.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2020-11-16T22:33:00.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:00.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2020-11-16T22:33:00.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:00.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2020-11-16T22:33:00.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:00.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2020-11-16T22:33:02.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:02.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2020-11-16T22:33:03.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:03.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2020-11-16T22:33:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2020-11-16T22:33:04.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:04.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2020-11-16T22:33:04.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:04.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2020-11-16T22:33:05.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:05.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2020-11-16T22:33:05.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:05.508 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2020-11-16T22:33:05.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:05.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2020-11-16T22:33:06.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:06.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2020-11-16T22:33:06.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:06.495 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2020-11-16T22:33:06.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:06.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2020-11-16T22:33:07.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:07.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2020-11-16T22:33:08.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:08.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2020-11-16T22:33:08.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:08.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2020-11-16T22:33:08.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:08.767 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2020-11-16T22:33:09.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:09.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2020-11-16T22:33:09.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:09.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2020-11-16T22:33:09.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:09.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2020-11-16T22:33:10.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:10.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2020-11-16T22:33:11.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:11.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2020-11-16T22:33:11.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:11.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2020-11-16T22:33:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:11.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2020-11-16T22:33:12.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:12.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2020-11-16T22:33:12.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:12.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2020-11-16T22:33:13.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:13.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2020-11-16T22:33:14.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:14.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2020-11-16T22:33:14.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:14.406 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2020-11-16T22:33:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2020-11-16T22:33:15.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:15.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2020-11-16T22:33:15.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:15.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2020-11-16T22:33:15.769 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:15.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2020-11-16T22:33:16.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:16.654 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2020-11-16T22:33:16.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:16.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2020-11-16T22:33:17.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:17.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2020-11-16T22:33:17.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:17.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2020-11-16T22:33:17.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:17.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2020-11-16T22:33:18.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:18.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2020-11-16T22:33:18.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:18.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2020-11-16T22:33:18.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:18.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2020-11-16T22:33:19.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:19.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2020-11-16T22:33:20.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:20.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2020-11-16T22:33:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:20.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2020-11-16T22:33:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2020-11-16T22:33:21.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:21.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2020-11-16T22:33:21.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:21.556 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2020-11-16T22:33:21.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:21.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2020-11-16T22:33:22.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:22.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2020-11-16T22:33:23.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:23.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2020-11-16T22:33:25.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:25.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2020-11-16T22:33:26.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:26.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2020-11-16T22:33:27.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:27.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2020-11-16T22:33:27.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:27.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2020-11-16T22:33:28.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:28.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2020-11-16T22:33:28.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:28.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2020-11-16T22:33:29.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:29.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2020-11-16T22:33:29.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:29.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2020-11-16T22:33:29.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:29.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2020-11-16T22:33:30.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:30.181 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2020-11-16T22:33:30.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:30.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2020-11-16T22:33:30.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:30.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2020-11-16T22:33:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2020-11-16T22:33:32.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:32.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2020-11-16T22:33:32.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:32.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2020-11-16T22:33:32.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:32.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2020-11-16T22:33:33.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:33.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2020-11-16T22:33:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2020-11-16T22:33:33.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:33.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2020-11-16T22:33:34.707 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:34.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2020-11-16T22:33:35.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:35.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2020-11-16T22:33:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2020-11-16T22:33:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2020-11-16T22:33:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:36.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2020-11-16T22:33:37.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:37.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2020-11-16T22:33:37.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:37.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2020-11-16T22:33:38.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:38.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2020-11-16T22:33:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2020-11-16T22:33:38.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:38.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2020-11-16T22:33:39.067 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:39.068 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2020-11-16T22:33:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2020-11-16T22:33:39.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:39.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2020-11-16T22:33:40.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:40.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2020-11-16T22:33:41.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:41.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2020-11-16T22:33:41.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:41.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2020-11-16T22:33:41.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:41.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2020-11-16T22:33:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2020-11-16T22:33:42.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:42.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2020-11-16T22:33:42.759 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:42.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2020-11-16T22:33:43.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:43.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2020-11-16T22:33:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2020-11-16T22:33:44.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:44.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2020-11-16T22:33:44.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:44.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2020-11-16T22:33:44.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:44.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2020-11-16T22:33:45.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:45.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2020-11-16T22:33:45.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:45.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2020-11-16T22:33:47.193 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 2 + 1 2020-11-16T22:33:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=3 2020-11-16T22:33:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2020-11-16T22:33:47.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2020-11-16T22:33:47.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:47.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2020-11-16T22:33:47.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:47.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2020-11-16T22:33:47.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:47.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2020-11-16T22:33:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:48.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2020-11-16T22:33:48.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:48.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2020-11-16T22:33:48.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:48.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2020-11-16T22:33:49.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:49.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2020-11-16T22:33:50.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:50.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2020-11-16T22:33:50.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:50.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2020-11-16T22:33:50.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:50.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2020-11-16T22:33:51.122 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:51.123 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2020-11-16T22:33:51.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:51.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2020-11-16T22:33:52.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:52.131 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2020-11-16T22:33:52.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:52.875 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2020-11-16T22:33:53.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:53.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2020-11-16T22:33:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:53.602 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2020-11-16T22:33:53.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:53.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2020-11-16T22:33:54.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:54.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2020-11-16T22:33:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2020-11-16T22:33:54.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:54.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2020-11-16T22:33:55.215 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:55.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2020-11-16T22:33:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2020-11-16T22:33:56.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:56.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2020-11-16T22:33:57.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:57.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2020-11-16T22:33:58.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:58.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2020-11-16T22:33:59.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:33:59.946 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2020-11-16T22:34:00.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:00.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2020-11-16T22:34:00.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:00.615 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2020-11-16T22:34:00.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:00.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2020-11-16T22:34:01.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:01.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2020-11-16T22:34:02.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:02.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2020-11-16T22:34:02.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:02.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2020-11-16T22:34:02.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:02.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2020-11-16T22:34:02.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:02.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2020-11-16T22:34:03.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:03.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2020-11-16T22:34:03.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:03.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2020-11-16T22:34:04.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:04.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2020-11-16T22:34:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:04.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2020-11-16T22:34:05.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:05.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2020-11-16T22:34:05.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:05.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2020-11-16T22:34:05.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:05.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2020-11-16T22:34:06.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:06.324 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2020-11-16T22:34:06.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:06.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2020-11-16T22:34:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2020-11-16T22:34:07.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:07.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2020-11-16T22:34:08.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:08.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2020-11-16T22:34:08.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:08.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2020-11-16T22:34:09.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:09.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2020-11-16T22:34:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:09.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2020-11-16T22:34:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2020-11-16T22:34:10.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:10.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2020-11-16T22:34:11.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:11.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2020-11-16T22:34:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:11.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2020-11-16T22:34:11.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:11.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2020-11-16T22:34:12.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:12.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2020-11-16T22:34:12.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:12.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2020-11-16T22:34:12.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:12.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2020-11-16T22:34:13.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:13.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2020-11-16T22:34:13.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:13.387 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2020-11-16T22:34:14.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:14.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2020-11-16T22:34:14.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:14.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2020-11-16T22:34:14.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:14.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2020-11-16T22:34:15.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:15.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2020-11-16T22:34:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2020-11-16T22:34:16.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:16.101 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2020-11-16T22:34:16.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:16.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2020-11-16T22:34:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2020-11-16T22:34:17.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:17.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2020-11-16T22:34:17.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:17.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2020-11-16T22:34:18.206 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:18.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2020-11-16T22:34:19.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:19.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2020-11-16T22:34:19.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:19.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2020-11-16T22:34:20.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:20.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2020-11-16T22:34:20.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:20.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2020-11-16T22:34:20.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:20.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2020-11-16T22:34:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2020-11-16T22:34:21.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:21.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2020-11-16T22:34:21.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:21.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2020-11-16T22:34:22.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:22.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2020-11-16T22:34:23.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:23.084 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2020-11-16T22:34:23.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:23.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2020-11-16T22:34:23.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:23.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2020-11-16T22:34:24.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:24.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2020-11-16T22:34:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2020-11-16T22:34:24.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:24.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2020-11-16T22:34:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2020-11-16T22:34:26.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:26.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2020-11-16T22:34:26.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:26.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2020-11-16T22:34:26.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:26.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2020-11-16T22:34:27.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:27.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2020-11-16T22:34:27.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:27.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2020-11-16T22:34:27.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:27.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2020-11-16T22:34:28.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:28.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2020-11-16T22:34:28.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:28.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2020-11-16T22:34:30.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:30.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2020-11-16T22:34:30.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:30.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2020-11-16T22:34:30.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:30.811 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2020-11-16T22:34:31.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:31.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2020-11-16T22:34:31.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:31.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2020-11-16T22:34:32.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:32.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2020-11-16T22:34:32.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:32.531 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2020-11-16T22:34:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2020-11-16T22:34:33.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:33.140 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2020-11-16T22:34:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:33.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2020-11-16T22:34:33.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:33.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2020-11-16T22:34:34.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:34.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2020-11-16T22:34:34.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:34.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2020-11-16T22:34:35.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:35.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2020-11-16T22:34:35.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:35.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2020-11-16T22:34:36.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:36.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2020-11-16T22:34:36.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:36.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2020-11-16T22:34:36.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:36.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2020-11-16T22:34:37.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:37.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2020-11-16T22:34:38.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:38.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2020-11-16T22:34:38.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:38.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2020-11-16T22:34:39.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:39.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2020-11-16T22:34:39.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:39.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2020-11-16T22:34:40.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:40.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2020-11-16T22:34:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2020-11-16T22:34:42.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:42.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2020-11-16T22:34:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:43.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2020-11-16T22:34:44.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:44.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2020-11-16T22:34:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2020-11-16T22:34:44.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:44.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2020-11-16T22:34:45.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:45.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2020-11-16T22:34:45.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:45.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2020-11-16T22:34:46.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:46.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2020-11-16T22:34:46.344 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:46.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2020-11-16T22:34:47.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:47.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2020-11-16T22:34:47.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:47.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2020-11-16T22:34:47.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:47.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2020-11-16T22:34:48.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:48.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2020-11-16T22:34:48.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:48.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2020-11-16T22:34:48.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:48.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2020-11-16T22:34:49.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:49.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2020-11-16T22:34:49.561 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:49.562 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2020-11-16T22:34:50.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:50.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2020-11-16T22:34:50.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:50.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2020-11-16T22:34:51.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:51.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2020-11-16T22:34:51.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:51.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2020-11-16T22:34:51.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:51.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2020-11-16T22:34:52.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:52.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2020-11-16T22:34:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:53.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2020-11-16T22:34:53.812 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:53.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2020-11-16T22:34:54.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:54.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2020-11-16T22:34:54.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:54.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2020-11-16T22:34:54.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:54.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2020-11-16T22:34:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2020-11-16T22:34:55.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:55.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2020-11-16T22:34:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2020-11-16T22:34:56.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:56.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2020-11-16T22:34:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2020-11-16T22:34:57.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:57.363 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2020-11-16T22:34:57.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:57.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2020-11-16T22:34:58.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:58.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2020-11-16T22:34:58.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:58.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2020-11-16T22:34:58.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:58.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2020-11-16T22:34:59.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:34:59.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2020-11-16T22:35:00.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:00.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2020-11-16T22:35:00.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:00.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2020-11-16T22:35:00.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:00.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2020-11-16T22:35:01.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:01.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2020-11-16T22:35:01.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:01.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2020-11-16T22:35:01.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:01.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2020-11-16T22:35:03.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:03.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2020-11-16T22:35:03.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:03.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2020-11-16T22:35:04.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:04.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2020-11-16T22:35:04.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:04.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2020-11-16T22:35:04.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:04.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2020-11-16T22:35:05.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:05.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2020-11-16T22:35:05.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:05.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2020-11-16T22:35:06.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:06.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2020-11-16T22:35:06.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:06.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2020-11-16T22:35:06.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:06.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2020-11-16T22:35:07.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:07.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2020-11-16T22:35:07.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:07.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2020-11-16T22:35:08.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:08.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2020-11-16T22:35:08.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:08.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2020-11-16T22:35:09.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:09.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2020-11-16T22:35:09.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:09.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2020-11-16T22:35:10.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:10.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2020-11-16T22:35:10.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:10.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2020-11-16T22:35:10.679 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:10.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2020-11-16T22:35:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2020-11-16T22:35:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2020-11-16T22:35:12.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:12.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2020-11-16T22:35:12.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:12.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2020-11-16T22:35:12.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:12.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2020-11-16T22:35:13.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:13.803 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2020-11-16T22:35:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2020-11-16T22:35:14.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:14.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2020-11-16T22:35:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:15.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2020-11-16T22:35:15.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:15.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2020-11-16T22:35:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:16.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2020-11-16T22:35:16.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:16.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2020-11-16T22:35:16.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:16.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2020-11-16T22:35:17.564 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:17.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2020-11-16T22:35:17.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:17.879 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2020-11-16T22:35:18.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:18.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2020-11-16T22:35:18.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2020-11-16T22:35:18.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2020-11-16T22:35:18.856 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 3 + 1 2020-11-16T22:35:18.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=4 2020-11-16T22:35:18.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: get_not_primary test1 obj1-p1 2020-11-16T22:35:18.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test1 2020-11-16T22:35:18.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1-p1 2020-11-16T22:35:18.859 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test1 obj1-p1 2020-11-16T22:35:18.860 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test1 2020-11-16T22:35:18.860 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1-p1 2020-11-16T22:35:18.860 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test1 obj1-p1 2020-11-16T22:35:18.860 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-11-16T22:35:19.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2020-11-16T22:35:19.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-11-16T22:35:19.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-11-16T22:35:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: local otherosd=0 2020-11-16T22:35:19.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:145: TEST_recovery_priority: ceph pg dump pgs 2020-11-16T22:35:20.027 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T22:35:20.027 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 196 0 0 0 0 2048917504 0 0 586 586 active+clean 2020-11-16T22:29:29.813346+0000 91'586 91:629 [1] 1 [1] 1 0'0 2020-11-16T22:29:00.696185+0000 0'0 2020-11-16T22:29:00.696185+0000 0 2020-11-16T22:35:20.028 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2020-11-16T22:28:48.277918+0000 91'600 91:665 [1,0] 1 [1,0] 1 0'0 2020-11-16T22:28:45.457031+0000 0'0 2020-11-16T22:28:45.457031+0000 0 2020-11-16T22:35:20.028 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2020-11-16T22:29:28.659049+0000 91'600 91:665 [1] 1 [1] 1 0'0 2020-11-16T22:28:50.266785+0000 0'0 2020-11-16T22:28:50.266785+0000 0 2020-11-16T22:35:20.028 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T22:35:20.028 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T22:35:20.030 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T22:35:20.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: ERRORS=0 2020-11-16T22:35:20.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph osd set norecover 2020-11-16T22:35:20.735 INFO:tasks.workunit.client.0.smithi189.stderr:norecover is set 2020-11-16T22:35:20.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ceph osd set noout 2020-11-16T22:35:21.473 INFO:tasks.workunit.client.0.smithi189.stderr:noout is set 2020-11-16T22:35:21.484 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:153: TEST_recovery_priority: ceph osd pool set test7 size 2 2020-11-16T22:35:22.154 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 7 size to 2 2020-11-16T22:35:22.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:154: TEST_recovery_priority: sleep 2 2020-11-16T22:35:24.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: get_asok_path osd.1 2020-11-16T22:35:24.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T22:35:24.169 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T22:35:24.169 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:35:24.169 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:35:24.169 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:35:24.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.1.asok 2020-11-16T22:35:24.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: CEPH_ARGS= 2020-11-16T22:35:24.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:35:24.279 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:35:24.279 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:35:24.279 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:24.279 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:24.279 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:35:24.280 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:35:24.280 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:24.280 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:35:24.280 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:35:24.280 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:24.280 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:24.280 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:35:24.281 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:35:24.281 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:35:24.281 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:24.281 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:24.281 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:35:24.281 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:35:24.282 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:24.282 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:35:24.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: seq 1 10 2020-11-16T22:35:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2020-11-16T22:35:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2020-11-16T22:35:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2020-11-16T22:35:24.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: '[' 1 = 10 ']' 2020-11-16T22:35:24.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:167: TEST_recovery_priority: sleep 2 2020-11-16T22:35:26.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2020-11-16T22:35:26.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2020-11-16T22:35:26.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2020-11-16T22:35:27.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: '[' 2 = 10 ']' 2020-11-16T22:35:27.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:167: TEST_recovery_priority: sleep 2 2020-11-16T22:35:29.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2020-11-16T22:35:29.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2020-11-16T22:35:29.055 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2020-11-16T22:35:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: break 2020-11-16T22:35:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:169: TEST_recovery_priority: flush_pg_stats 2020-11-16T22:35:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:35:29.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:35:29.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:35:29.838 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:35:29.838 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:35:29.839 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:35:29.839 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T22:35:29.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:35:29.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:29.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:35:30.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836575 2020-11-16T22:35:30.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836575 2020-11-16T22:35:30.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836575' 2020-11-16T22:35:30.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:30.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:35:30.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673052 2020-11-16T22:35:30.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673052 2020-11-16T22:35:30.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836575 1-42949673052' 2020-11-16T22:35:30.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:30.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:35:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509531 2020-11-16T22:35:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509531 2020-11-16T22:35:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836575 1-42949673052 2-64424509531' 2020-11-16T22:35:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:30.327 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:35:30.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378713 2020-11-16T22:35:30.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378713 2020-11-16T22:35:30.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836575 1-42949673052 2-64424509531 3-81604378713' 2020-11-16T22:35:30.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:30.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:35:30.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247895 2020-11-16T22:35:30.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247895 2020-11-16T22:35:30.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836575 1-42949673052 2-64424509531 3-81604378713 4-98784247895' 2020-11-16T22:35:30.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:30.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836575 2020-11-16T22:35:30.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:30.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:35:30.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836575 2020-11-16T22:35:30.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:30.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836575 2020-11-16T22:35:30.646 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2020-11-16T22:35:30.646 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836575 2020-11-16T22:35:30.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:35:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836573 -lt 21474836575 2020-11-16T22:35:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:35:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:35:32.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:35:32.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836575 -lt 21474836575 2020-11-16T22:35:32.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:32.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673052 2020-11-16T22:35:32.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:32.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:35:32.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673052 2020-11-16T22:35:32.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:32.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673052 2020-11-16T22:35:32.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2020-11-16T22:35:32.444 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673052 2020-11-16T22:35:32.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:35:32.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673052 -lt 42949673052 2020-11-16T22:35:32.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:32.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509531 2020-11-16T22:35:32.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:32.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:35:32.842 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509531 2020-11-16T22:35:32.843 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:32.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509531 2020-11-16T22:35:32.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2020-11-16T22:35:32.844 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509531 2020-11-16T22:35:32.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:35:33.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509531 -lt 64424509531 2020-11-16T22:35:33.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:33.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378713 2020-11-16T22:35:33.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:33.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:35:33.230 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378713 2020-11-16T22:35:33.231 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:33.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378713 2020-11-16T22:35:33.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378713' 2020-11-16T22:35:33.233 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378713 2020-11-16T22:35:33.234 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:35:33.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378714 -lt 81604378713 2020-11-16T22:35:33.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:33.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247895 2020-11-16T22:35:33.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:33.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:35:33.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247895 2020-11-16T22:35:33.750 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:33.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247895 2020-11-16T22:35:33.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247895' 2020-11-16T22:35:33.752 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247895 2020-11-16T22:35:33.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:35:34.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247896 -lt 98784247895 2020-11-16T22:35:34.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: get_asok_path osd.1 2020-11-16T22:35:34.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T22:35:34.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T22:35:34.109 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:35:34.109 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:35:34.110 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:35:34.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.1.asok 2020-11-16T22:35:34.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: CEPH_ARGS= 2020-11-16T22:35:34.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:35:34.198 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:35:34.198 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:35:34.199 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:34.199 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:34.199 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:35:34.199 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:35:34.199 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:34.199 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:35:34.199 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:35:34.200 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:34.200 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:34.200 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:35:34.200 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:35:34.200 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:35:34.200 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:34.200 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:34.201 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:35:34.201 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:35:34.201 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:34.201 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:35:34.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: ceph osd out osd.0 2020-11-16T22:35:34.742 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 is already out. 2020-11-16T22:35:34.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: sleep 2 2020-11-16T22:35:36.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:174: TEST_recovery_priority: flush_pg_stats 2020-11-16T22:35:36.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:35:36.759 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:35:37.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:35:37.157 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:35:37.158 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:35:37.158 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:35:37.158 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T22:35:37.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:35:37.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:37.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:35:37.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836578 2020-11-16T22:35:37.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836578 2020-11-16T22:35:37.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836578' 2020-11-16T22:35:37.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:37.337 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:35:37.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673056 2020-11-16T22:35:37.497 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673056 2020-11-16T22:35:37.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836578 1-42949673056' 2020-11-16T22:35:37.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:37.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:35:37.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509534 2020-11-16T22:35:37.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509534 2020-11-16T22:35:37.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836578 1-42949673056 2-64424509534' 2020-11-16T22:35:37.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:37.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:35:37.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378717 2020-11-16T22:35:37.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378717 2020-11-16T22:35:37.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836578 1-42949673056 2-64424509534 3-81604378717' 2020-11-16T22:35:37.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:37.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:35:37.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247899 2020-11-16T22:35:37.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247899 2020-11-16T22:35:37.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836578 1-42949673056 2-64424509534 3-81604378717 4-98784247899' 2020-11-16T22:35:37.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:37.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836578 2020-11-16T22:35:37.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:37.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:35:37.984 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836578 2020-11-16T22:35:37.985 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:37.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836578 2020-11-16T22:35:37.986 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2020-11-16T22:35:37.986 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836578 2020-11-16T22:35:37.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:35:38.369 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836578 -lt 21474836578 2020-11-16T22:35:38.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:38.370 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673056 2020-11-16T22:35:38.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:38.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:35:38.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673056 2020-11-16T22:35:38.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:38.375 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673056 2020-11-16T22:35:38.376 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2020-11-16T22:35:38.376 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:35:38.376 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673056 2020-11-16T22:35:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673054 -lt 42949673056 2020-11-16T22:35:38.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:35:39.987 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:35:39.987 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:35:40.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673056 -lt 42949673056 2020-11-16T22:35:40.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:40.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509534 2020-11-16T22:35:40.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:40.388 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:35:40.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509534 2020-11-16T22:35:40.389 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:40.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509534 2020-11-16T22:35:40.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2020-11-16T22:35:40.390 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509534 2020-11-16T22:35:40.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:35:40.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509534 -lt 64424509534 2020-11-16T22:35:40.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:40.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:40.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378717 2020-11-16T22:35:40.775 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:35:40.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:40.775 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378717 2020-11-16T22:35:40.776 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378717 2020-11-16T22:35:40.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378717 2020-11-16T22:35:40.776 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378717' 2020-11-16T22:35:40.777 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:35:41.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378717 -lt 81604378717 2020-11-16T22:35:41.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:41.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247899 2020-11-16T22:35:41.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:41.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:35:41.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247899 2020-11-16T22:35:41.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:41.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247899 2020-11-16T22:35:41.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247899' 2020-11-16T22:35:41.174 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247899 2020-11-16T22:35:41.174 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:35:41.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247900 -lt 98784247899 2020-11-16T22:35:41.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: get_asok_path osd.1 2020-11-16T22:35:41.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T22:35:41.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T22:35:41.584 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:35:41.585 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:35:41.585 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:35:41.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.1.asok 2020-11-16T22:35:41.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: CEPH_ARGS= 2020-11-16T22:35:41.586 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:35:41.675 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:35:41.675 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:35:41.675 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:41.675 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:41.675 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:35:41.675 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:41.676 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:35:41.676 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:35:41.676 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:41.676 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:35:41.676 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:35:41.676 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:41.676 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:41.676 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:35:41.677 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:41.677 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:35:41.677 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:35:41.677 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:41.677 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:35:41.677 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:35:41.677 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:41.678 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:41.678 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:35:41.678 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:35:41.678 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:35:41.678 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:41.678 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:41.678 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:35:41.679 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:35:41.679 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:41.679 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:35:41.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: ceph pg dump pgs 2020-11-16T22:35:42.024 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T22:35:42.024 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+forced_recovery+undersized+degraded+remapped 2020-11-16T22:35:29.348534+0000 91'600 103:661 [1,4] 1 [1] 1 0'0 2020-11-16T22:29:00.696185+0000 0'0 2020-11-16T22:29:00.696185+0000 0 2020-11-16T22:35:42.024 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovery_wait+undersized+degraded+remapped 2020-11-16T22:35:35.986724+0000 91'600 104:681 [1,2] 1 [1] 1 0'0 2020-11-16T22:28:45.457031+0000 0'0 2020-11-16T22:28:45.457031+0000 0 2020-11-16T22:35:42.025 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2020-11-16T22:29:28.659049+0000 91'600 103:674 [1] 1 [1] 1 0'0 2020-11-16T22:28:50.266785+0000 0'0 2020-11-16T22:28:50.266785+0000 0 2020-11-16T22:35:42.025 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T22:35:42.025 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T22:35:42.026 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T22:35:42.035 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph osd pool set test3 size 2 2020-11-16T22:35:42.684 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 size to 2 2020-11-16T22:35:42.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: sleep 2 2020-11-16T22:35:44.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:180: TEST_recovery_priority: flush_pg_stats 2020-11-16T22:35:44.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:35:44.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:35:45.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:35:45.166 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:35:45.166 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:35:45.166 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:35:45.166 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T22:35:45.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:35:45.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:45.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:35:45.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836582 2020-11-16T22:35:45.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836582 2020-11-16T22:35:45.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836582' 2020-11-16T22:35:45.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:45.318 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:35:45.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673059 2020-11-16T22:35:45.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673059 2020-11-16T22:35:45.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836582 1-42949673059' 2020-11-16T22:35:45.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:45.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:35:45.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509538 2020-11-16T22:35:45.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509538 2020-11-16T22:35:45.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836582 1-42949673059 2-64424509538' 2020-11-16T22:35:45.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:45.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:35:45.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378720 2020-11-16T22:35:45.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378720 2020-11-16T22:35:45.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836582 1-42949673059 2-64424509538 3-81604378720' 2020-11-16T22:35:45.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:45.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:35:46.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247903 2020-11-16T22:35:46.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247903 2020-11-16T22:35:46.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836582 1-42949673059 2-64424509538 3-81604378720 4-98784247903' 2020-11-16T22:35:46.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:46.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836582 2020-11-16T22:35:46.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:46.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:35:46.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836582 2020-11-16T22:35:46.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:46.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836582 2020-11-16T22:35:46.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2020-11-16T22:35:46.047 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836582 2020-11-16T22:35:46.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:35:46.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836582 -lt 21474836582 2020-11-16T22:35:46.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:46.438 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673059 2020-11-16T22:35:46.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:46.440 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:35:46.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673059 2020-11-16T22:35:46.441 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:46.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673059 2020-11-16T22:35:46.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2020-11-16T22:35:46.443 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673059 2020-11-16T22:35:46.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:35:46.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673057 -lt 42949673059 2020-11-16T22:35:46.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:35:47.849 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:35:47.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:35:48.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673060 -lt 42949673059 2020-11-16T22:35:48.251 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:48.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:48.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509538 2020-11-16T22:35:48.253 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:35:48.253 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509538 2020-11-16T22:35:48.254 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:48.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509538 2020-11-16T22:35:48.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509538' 2020-11-16T22:35:48.255 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509538 2020-11-16T22:35:48.255 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:35:48.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509538 -lt 64424509538 2020-11-16T22:35:48.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:48.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378720 2020-11-16T22:35:48.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:48.621 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:35:48.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378720 2020-11-16T22:35:48.623 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:48.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378720 2020-11-16T22:35:48.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378720' 2020-11-16T22:35:48.625 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378720 2020-11-16T22:35:48.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:35:49.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378720 -lt 81604378720 2020-11-16T22:35:49.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:49.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247903 2020-11-16T22:35:49.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:49.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:35:49.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247903 2020-11-16T22:35:49.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247903 2020-11-16T22:35:49.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247903' 2020-11-16T22:35:49.021 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247903 2020-11-16T22:35:49.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:35:49.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247903 -lt 98784247903 2020-11-16T22:35:49.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: get_asok_path osd.1 2020-11-16T22:35:49.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T22:35:49.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T22:35:49.417 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:35:49.417 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:35:49.417 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:35:49.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.1.asok 2020-11-16T22:35:49.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: CEPH_ARGS= 2020-11-16T22:35:49.418 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:35:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:35:49.529 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:35:49.529 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:35:49.529 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:49.529 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:49.530 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:35:49.530 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:49.530 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:35:49.530 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:35:49.530 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:49.530 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:35:49.530 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:35:49.531 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:49.531 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:35:49.531 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:49.531 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:35:49.531 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:35:49.531 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:49.531 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:49.532 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:35:49.532 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:49.532 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:35:49.532 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:35:49.532 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:49.532 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:35:49.532 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:35:49.532 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:49.533 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:49.533 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:35:49.533 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:35:49.533 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:35:49.533 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:49.533 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:49.533 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:35:49.534 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:35:49.534 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:49.534 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:35:49.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: ceph pg dump pgs 2020-11-16T22:35:49.869 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T22:35:49.869 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+forced_recovery+undersized+degraded+remapped 2020-11-16T22:35:29.348534+0000 91'600 107:664 [1,4] 1 [1] 1 0'0 2020-11-16T22:29:00.696185+0000 0'0 2020-11-16T22:29:00.696185+0000 0 2020-11-16T22:35:49.869 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovery_wait+undersized+degraded+remapped 2020-11-16T22:35:35.986724+0000 91'600 107:684 [1,2] 1 [1] 1 0'0 2020-11-16T22:28:45.457031+0000 0'0 2020-11-16T22:28:45.457031+0000 0 2020-11-16T22:35:49.870 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovery_wait+undersized+degraded+remapped 2020-11-16T22:35:45.855302+0000 91'600 108:684 [1,2] 1 [1] 1 0'0 2020-11-16T22:28:50.266785+0000 0'0 2020-11-16T22:28:50.266785+0000 0 2020-11-16T22:35:49.870 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T22:35:49.870 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T22:35:49.871 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T22:35:49.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:35:49.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2020-11-16T22:35:49.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: PRIO=190 2020-11-16T22:35:49.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2020-11-16T22:35:49.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:35:49.886 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2020-11-16T22:35:49.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: eval 'ITEM="7.0"' 2020-11-16T22:35:49.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: ITEM=7.0 2020-11-16T22:35:49.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:194: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2020-11-16T22:35:49.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:199: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:35:49.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:199: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2020-11-16T22:35:49.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:199: TEST_recovery_priority: PRIO=255 2020-11-16T22:35:49.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:200: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2020-11-16T22:35:49.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:208: TEST_recovery_priority: seq 1 10 2020-11-16T22:35:49.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:208: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2020-11-16T22:35:49.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:210: TEST_recovery_priority: ceph pg force-recovery 3.0 2020-11-16T22:35:49.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:210: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2020-11-16T22:35:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: break 2020-11-16T22:35:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:219: TEST_recovery_priority: sleep 2 2020-11-16T22:35:52.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: get_asok_path osd.1 2020-11-16T22:35:52.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T22:35:52.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T22:35:52.274 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:35:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:35:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:35:52.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.1.asok 2020-11-16T22:35:52.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: CEPH_ARGS= 2020-11-16T22:35:52.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:35:52.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:221: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:35:52.394 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:35:52.395 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:35:52.395 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:52.395 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:52.395 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:35:52.395 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:52.395 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:35:52.395 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:35:52.396 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:52.396 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:35:52.396 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:35:52.396 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:52.396 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:52.396 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:35:52.396 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:35:52.397 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:52.397 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 255, 2020-11-16T22:35:52.397 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:35:52.397 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:52.397 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:35:52.397 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:35:52.397 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:52.397 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:52.398 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:35:52.398 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:52.398 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:35:52.398 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:35:52.398 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:52.398 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:35:52.398 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:35:52.399 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:52.399 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:52.399 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:35:52.399 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:35:52.399 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:35:52.399 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:52.399 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:52.400 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:35:52.400 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:35:52.400 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:52.400 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:35:52.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:35:52.401 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-11-16T22:35:52.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: PRIO=255 2020-11-16T22:35:52.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2020-11-16T22:35:52.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:228: TEST_recovery_priority: flush_pg_stats 2020-11-16T22:35:52.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:35:52.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:35:52.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:35:52.793 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:35:52.794 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:35:52.794 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:35:52.794 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T22:35:52.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:35:52.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:52.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:35:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836585 2020-11-16T22:35:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836585 2020-11-16T22:35:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836585' 2020-11-16T22:35:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:52.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:35:53.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673063 2020-11-16T22:35:53.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673063 2020-11-16T22:35:53.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836585 1-42949673063' 2020-11-16T22:35:53.134 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:53.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:35:53.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509541 2020-11-16T22:35:53.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509541 2020-11-16T22:35:53.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836585 1-42949673063 2-64424509541' 2020-11-16T22:35:53.292 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:53.293 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:35:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378724 2020-11-16T22:35:53.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378724 2020-11-16T22:35:53.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836585 1-42949673063 2-64424509541 3-81604378724' 2020-11-16T22:35:53.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:35:53.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:35:53.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247906 2020-11-16T22:35:53.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247906 2020-11-16T22:35:53.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836585 1-42949673063 2-64424509541 3-81604378724 4-98784247906' 2020-11-16T22:35:53.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:53.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836585 2020-11-16T22:35:53.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:53.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:35:53.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836585 2020-11-16T22:35:53.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:53.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836585 2020-11-16T22:35:53.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2020-11-16T22:35:53.614 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836585 2020-11-16T22:35:53.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:35:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836585 -lt 21474836585 2020-11-16T22:35:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:54.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673063 2020-11-16T22:35:54.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:35:54.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673063 2020-11-16T22:35:54.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673063 2020-11-16T22:35:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2020-11-16T22:35:54.016 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673063 2020-11-16T22:35:54.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:35:54.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673063 -lt 42949673063 2020-11-16T22:35:54.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:54.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509541 2020-11-16T22:35:54.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:54.396 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:35:54.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509541 2020-11-16T22:35:54.397 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:54.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509541 2020-11-16T22:35:54.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509541' 2020-11-16T22:35:54.399 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509541 2020-11-16T22:35:54.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:35:54.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509541 -lt 64424509541 2020-11-16T22:35:54.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:54.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:54.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378724 2020-11-16T22:35:54.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:35:54.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378724 2020-11-16T22:35:54.793 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:54.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378724 2020-11-16T22:35:54.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378724' 2020-11-16T22:35:54.795 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378724 2020-11-16T22:35:54.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:35:55.191 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378724 -lt 81604378724 2020-11-16T22:35:55.192 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:35:55.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247906 2020-11-16T22:35:55.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:35:55.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:35:55.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247906 2020-11-16T22:35:55.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:35:55.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247906 2020-11-16T22:35:55.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247906' 2020-11-16T22:35:55.197 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247906 2020-11-16T22:35:55.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:35:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247906 -lt 98784247906 2020-11-16T22:35:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:232: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2020-11-16T22:35:55.966 INFO:tasks.workunit.client.0.smithi189.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2020-11-16T22:35:55.966 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T22:35:55.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:233: TEST_recovery_priority: sleep 2 2020-11-16T22:35:57.977 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: get_asok_path osd.1 2020-11-16T22:35:57.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T22:35:57.978 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T22:35:57.979 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:35:57.979 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:35:57.979 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:35:57.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.1.asok 2020-11-16T22:35:57.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: CEPH_ARGS= 2020-11-16T22:35:57.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:35:58.097 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:35:58.098 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:35:58.099 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:35:58.099 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:58.099 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:58.099 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:35:58.099 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:58.099 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:35:58.100 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:35:58.100 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:58.100 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:35:58.100 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:35:58.100 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:58.100 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:35:58.100 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:58.101 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:35:58.101 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:35:58.101 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:58.101 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:58.101 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:35:58.101 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:58.101 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:35:58.101 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:35:58.102 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:35:58.102 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:35:58.102 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:35:58.102 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:35:58.102 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:58.102 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:35:58.102 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:35:58.103 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:35:58.103 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:35:58.103 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:35:58.103 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:35:58.103 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:35:58.103 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:35:58.103 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:35:58.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:35:58.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2020-11-16T22:35:58.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: PRIO=190 2020-11-16T22:35:58.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2020-11-16T22:35:58.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:35:58.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2020-11-16T22:35:58.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: eval 'ITEM="3.0"' 2020-11-16T22:35:58.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: ITEM=3.0 2020-11-16T22:35:58.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:245: TEST_recovery_priority: '[' 3.0 '!=' 3.0 ']' 2020-11-16T22:35:58.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:250: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:35:58.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:250: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2020-11-16T22:35:58.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:250: TEST_recovery_priority: PRIO=255 2020-11-16T22:35:58.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:251: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2020-11-16T22:35:58.117 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:258: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2020-11-16T22:35:58.498 INFO:tasks.workunit.client.0.smithi189.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2020-11-16T22:35:58.498 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T22:35:58.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:259: TEST_recovery_priority: sleep 5 2020-11-16T22:36:03.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: get_asok_path osd.1 2020-11-16T22:36:03.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T22:36:03.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T22:36:03.512 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:36:03.512 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:36:03.512 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:36:03.512 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.1.asok 2020-11-16T22:36:03.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: CEPH_ARGS= 2020-11-16T22:36:03.513 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:03.625 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:03.625 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:03.625 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:03.625 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:03.625 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:03.626 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:03.626 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:03.626 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:03.626 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:03.626 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:03.626 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:03.626 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:03.627 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:03.627 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:03.627 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:03.627 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:03.627 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:03.627 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:03.627 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:03.627 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:03.628 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:03.628 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:03.628 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:03.628 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:03.628 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:03.628 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:03.628 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:03.629 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:03.629 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:03.629 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:03.629 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:03.629 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:03.629 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:03.629 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:03.629 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:03.630 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:03.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: flush_pg_stats 2020-11-16T22:36:03.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:36:03.635 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:36:04.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:36:04.026 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:36:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:36:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:36:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T22:36:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:36:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:36:04.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:36:04.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836590 2020-11-16T22:36:04.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836590 2020-11-16T22:36:04.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836590' 2020-11-16T22:36:04.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:36:04.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:36:04.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673067 2020-11-16T22:36:04.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673067 2020-11-16T22:36:04.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836590 1-42949673067' 2020-11-16T22:36:04.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:36:04.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:36:04.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509545 2020-11-16T22:36:04.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509545 2020-11-16T22:36:04.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836590 1-42949673067 2-64424509545' 2020-11-16T22:36:04.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:36:04.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:36:04.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378728 2020-11-16T22:36:04.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378728 2020-11-16T22:36:04.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836590 1-42949673067 2-64424509545 3-81604378728' 2020-11-16T22:36:04.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:36:04.670 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:36:04.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247910 2020-11-16T22:36:04.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247910 2020-11-16T22:36:04.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836590 1-42949673067 2-64424509545 3-81604378728 4-98784247910' 2020-11-16T22:36:04.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:36:04.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836590 2020-11-16T22:36:04.836 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:36:04.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:36:04.838 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:36:04.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836590 2020-11-16T22:36:04.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836590 2020-11-16T22:36:04.840 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2020-11-16T22:36:04.840 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836590 2020-11-16T22:36:04.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:36:05.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836587 -lt 21474836590 2020-11-16T22:36:05.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:36:06.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:36:06.243 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:36:06.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836590 -lt 21474836590 2020-11-16T22:36:06.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:36:06.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673067 2020-11-16T22:36:06.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:36:06.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:36:06.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673067 2020-11-16T22:36:06.652 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:36:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673067 2020-11-16T22:36:06.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2020-11-16T22:36:06.653 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673067 2020-11-16T22:36:06.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:36:07.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673067 -lt 42949673067 2020-11-16T22:36:07.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:36:07.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:36:07.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509545 2020-11-16T22:36:07.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:36:07.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509545 2020-11-16T22:36:07.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:36:07.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509545 2020-11-16T22:36:07.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509545' 2020-11-16T22:36:07.044 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509545 2020-11-16T22:36:07.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:36:07.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509546 -lt 64424509545 2020-11-16T22:36:07.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:36:07.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378728 2020-11-16T22:36:07.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:36:07.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:36:07.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378728 2020-11-16T22:36:07.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:36:07.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378728 2020-11-16T22:36:07.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378728' 2020-11-16T22:36:07.433 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378728 2020-11-16T22:36:07.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:36:07.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378728 -lt 81604378728 2020-11-16T22:36:07.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:36:07.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247910 2020-11-16T22:36:07.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:36:07.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:36:07.822 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247910 2020-11-16T22:36:07.822 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:36:07.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247910 2020-11-16T22:36:07.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247910' 2020-11-16T22:36:07.824 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247910 2020-11-16T22:36:07.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:36:08.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247911 -lt 98784247910 2020-11-16T22:36:08.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:264: TEST_recovery_priority: ceph pg force-recovery 7.0 2020-11-16T22:36:08.565 INFO:tasks.workunit.client.0.smithi189.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2020-11-16T22:36:08.565 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-16T22:36:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:265: TEST_recovery_priority: sleep 2 2020-11-16T22:36:10.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: get_asok_path osd.1 2020-11-16T22:36:10.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T22:36:10.583 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T22:36:10.583 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:36:10.584 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:36:10.584 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:36:10.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.1.asok 2020-11-16T22:36:10.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: CEPH_ARGS= 2020-11-16T22:36:10.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:10.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:36:10.703 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:10.703 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:10.703 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:10.703 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:10.704 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:10.704 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:10.704 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:10.704 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:10.704 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:10.704 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:10.705 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:10.705 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:10.705 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:10.705 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:10.705 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:10.705 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:10.705 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:10.706 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:10.706 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:10.706 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:10.706 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:10.706 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:10.706 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:10.706 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:10.707 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:10.707 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:10.707 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:10.707 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:10.707 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:10.707 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:10.707 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:10.708 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:10.708 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:10.708 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:10.708 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:10.708 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:10.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:36:10.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-11-16T22:36:10.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: PRIO=190 2020-11-16T22:36:10.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2020-11-16T22:36:10.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:36:10.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2020-11-16T22:36:10.718 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: eval 'ITEM="7.0"' 2020-11-16T22:36:10.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: ITEM=7.0 2020-11-16T22:36:10.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:277: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2020-11-16T22:36:10.719 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:282: TEST_recovery_priority: cat td/osd-recovery-prio/out 2020-11-16T22:36:10.720 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:282: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2020-11-16T22:36:10.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:282: TEST_recovery_priority: PRIO=255 2020-11-16T22:36:10.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:283: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2020-11-16T22:36:10.724 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:290: TEST_recovery_priority: ceph osd unset noout 2020-11-16T22:36:11.437 INFO:tasks.workunit.client.0.smithi189.stderr:noout is unset 2020-11-16T22:36:11.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:291: TEST_recovery_priority: ceph osd unset norecover 2020-11-16T22:36:12.314 INFO:tasks.workunit.client.0.smithi189.stderr:norecover is unset 2020-11-16T22:36:12.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: get_asok_path osd.1 2020-11-16T22:36:12.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T22:36:12.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T22:36:12.329 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:36:12.330 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:36:12.330 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:36:12.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.1.asok 2020-11-16T22:36:12.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations' 2020-11-16T22:36:12.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations' 2020-11-16T22:36:12.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:36:12.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:36:12.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:36:12.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:36:12.332 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:36:12.332 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:36:12.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:36:12.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:36:12.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:36:12.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:36:12.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:36:12.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:36:12.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:36:12.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:36:12.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:36:12.932 INFO:tasks.workunit.client.0.smithi189.stderr:1 2020-11-16T22:36:12.932 INFO:tasks.workunit.client.0.smithi189.stderr:2 2020-11-16T22:36:12.932 INFO:tasks.workunit.client.0.smithi189.stderr:3 2020-11-16T22:36:12.932 INFO:tasks.workunit.client.0.smithi189.stderr:4' 2020-11-16T22:36:12.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:36:12.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:36:12.933 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:36:13.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836593 2020-11-16T22:36:13.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836593 2020-11-16T22:36:13.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836593' 2020-11-16T22:36:13.112 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:36:13.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:36:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673071 2020-11-16T22:36:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673071 2020-11-16T22:36:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836593 1-42949673071' 2020-11-16T22:36:13.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:36:13.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-11-16T22:36:13.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509549 2020-11-16T22:36:13.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509549 2020-11-16T22:36:13.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509549' 2020-11-16T22:36:13.442 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:36:13.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-11-16T22:36:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378732 2020-11-16T22:36:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378732 2020-11-16T22:36:13.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509549 3-81604378732' 2020-11-16T22:36:13.597 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:36:13.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-11-16T22:36:13.757 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247914 2020-11-16T22:36:13.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247914 2020-11-16T22:36:13.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509549 3-81604378732 4-98784247914' 2020-11-16T22:36:13.758 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:36:13.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836593 2020-11-16T22:36:13.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:36:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:36:13.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836593 2020-11-16T22:36:13.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:36:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836593 2020-11-16T22:36:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2020-11-16T22:36:13.764 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:36:13.764 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836593 2020-11-16T22:36:14.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836593 -lt 21474836593 2020-11-16T22:36:14.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:36:14.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673071 2020-11-16T22:36:14.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:36:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:36:14.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673071 2020-11-16T22:36:14.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:36:14.188 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 42949673071 2020-11-16T22:36:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673071 2020-11-16T22:36:14.189 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673071' 2020-11-16T22:36:14.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:36:14.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673071 -lt 42949673071 2020-11-16T22:36:14.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:36:14.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509549 2020-11-16T22:36:14.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:36:14.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-11-16T22:36:14.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509549 2020-11-16T22:36:14.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:36:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509549 2020-11-16T22:36:14.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509549' 2020-11-16T22:36:14.668 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.2 seq 64424509549 2020-11-16T22:36:14.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-11-16T22:36:15.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509549 -lt 64424509549 2020-11-16T22:36:15.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:36:15.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378732 2020-11-16T22:36:15.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:36:15.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-11-16T22:36:15.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378732 2020-11-16T22:36:15.056 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:36:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378732 2020-11-16T22:36:15.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378732' 2020-11-16T22:36:15.057 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.3 seq 81604378732 2020-11-16T22:36:15.057 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:36:15.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378730 -lt 81604378732 2020-11-16T22:36:15.488 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-11-16T22:36:16.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-11-16T22:36:16.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-11-16T22:36:16.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378732 -lt 81604378732 2020-11-16T22:36:16.924 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:36:16.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247914 2020-11-16T22:36:16.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:36:16.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-11-16T22:36:16.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247914 2020-11-16T22:36:16.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:36:16.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247914 2020-11-16T22:36:16.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247914' 2020-11-16T22:36:16.929 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.4 seq 98784247914 2020-11-16T22:36:16.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-11-16T22:36:17.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247914 -lt 98784247914 2020-11-16T22:36:17.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:36:17.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:17.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:17.905 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2020-11-16T22:36:17.906 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:17.906 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:17.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:17.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:17.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:17.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:17.907 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:18.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:18.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:18.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:18.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:18.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:18.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2020-11-16T22:36:18.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T22:36:18.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2020-11-16T22:36:18.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:18.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:18.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:19.049 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:19.049 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:19.049 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:19.050 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:19.050 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:19.050 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:19.050 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:19.050 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:19.050 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:19.050 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:19.051 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:19.051 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:19.051 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:19.051 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:19.051 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:19.051 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:19.051 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:19.051 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:19.052 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:19.052 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:19.052 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:19.052 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:19.052 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:19.052 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:19.052 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:19.053 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:19.053 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:19.053 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:19.053 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:19.053 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:19.053 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:19.053 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:19.053 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:19.054 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:19.054 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:19.054 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:19.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:19.158 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:19.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:19.159 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:19.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:19.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:19.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:19.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:19.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:19.526 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:19.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:19.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:19.527 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:19.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:19.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:19.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:19.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:19.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:19.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:19.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:19.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:19.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:20.501 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7861943 2020-11-16T22:36:20.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7861943 '!=' null 2020-11-16T22:36:20.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:20.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:20.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:20.502 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:20.598 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:20.599 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:20.599 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:20.599 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:20.599 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:20.599 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:20.599 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:20.599 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:20.600 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:20.600 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:20.600 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:20.600 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:20.600 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:20.600 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:20.600 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:20.600 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:20.601 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:20.601 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:20.601 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:20.601 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:20.601 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:20.601 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:20.601 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:20.602 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:20.602 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:20.602 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:20.602 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:20.602 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:20.602 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:20.602 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:20.602 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:20.603 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:20.603 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:20.603 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:20.603 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:20.603 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:20.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:20.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:20.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:20.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:20.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:20.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:20.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:20.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:20.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:21.093 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:21.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:21.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:21.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:21.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:21.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:21.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:21.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:21.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:21.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:21.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:21.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:21.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:22.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7861943 2020-11-16T22:36:22.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7861943 '!=' null 2020-11-16T22:36:22.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:22.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:22.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:22.060 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:22.160 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:22.161 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:22.161 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:22.161 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:22.161 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:22.161 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:22.161 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:22.161 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:22.162 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:22.162 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:22.162 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:22.162 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:22.162 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:22.162 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:22.162 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:22.163 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:22.163 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:22.163 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:22.163 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:22.163 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:22.163 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:22.163 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:22.163 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:22.164 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:22.164 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:22.164 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:22.164 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:22.164 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:22.164 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:22.164 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:22.165 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:22.165 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:22.165 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:22.165 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:22.165 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:22.165 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:22.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:22.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:22.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:22.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:22.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:22.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:22.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:22.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:22.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:22.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:22.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:23.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:23.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:23.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:23.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:23.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:23.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:23.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:23.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:23.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:23.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6664003 2020-11-16T22:36:23.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6664003 '!=' null 2020-11-16T22:36:23.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:23.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:23.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:23.637 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:23.732 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:23.732 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:23.733 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:23.733 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:23.733 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:23.733 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:23.733 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:23.733 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:23.733 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:23.734 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:23.734 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:23.734 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:23.734 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:23.734 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:23.734 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:23.734 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:23.735 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:23.735 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:23.735 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:23.735 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:23.735 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:23.735 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:23.735 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:23.736 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:23.736 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:23.736 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:23.736 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:23.736 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:23.736 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:23.736 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:23.736 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:23.737 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:23.737 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:23.737 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:23.737 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:23.737 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:23.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:23.843 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:23.844 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:23.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:23.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:23.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:23.844 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:23.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:23.845 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:24.218 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:24.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:24.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:24.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:25.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:25.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:25.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:25.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14118064 2020-11-16T22:36:25.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14118064 '!=' null 2020-11-16T22:36:25.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:25.572 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:25.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:25.572 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:25.660 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:25.661 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:25.661 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:25.661 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:25.661 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:25.661 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:25.661 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:25.661 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:25.662 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:25.662 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:25.662 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:25.662 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:25.662 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:25.662 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:25.662 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:25.663 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:25.663 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:25.663 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:25.663 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:25.663 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:25.663 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:25.663 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:25.664 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:25.664 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:25.664 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:25.664 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:25.664 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:25.664 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:25.664 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:25.664 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:25.665 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:25.665 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:25.665 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:25.665 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:25.665 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:25.665 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:25.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:25.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:25.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:25.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:25.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:25.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:25.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:25.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:25.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:26.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:26.135 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:26.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:26.136 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:26.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:26.791 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:26.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:26.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:26.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:26.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:26.792 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:26.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:26.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:27.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13103101 2020-11-16T22:36:27.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13103101 '!=' null 2020-11-16T22:36:27.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:27.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:27.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:27.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:27.376 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:27.377 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:27.377 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:27.377 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:27.377 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:27.377 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:27.378 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:27.378 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:27.378 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:27.378 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:27.378 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:27.378 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:27.378 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:27.378 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:27.379 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:27.379 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:27.379 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:27.379 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:27.379 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:27.379 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:27.379 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:27.380 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:27.380 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:27.380 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:27.380 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:27.380 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:27.380 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:27.380 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:27.380 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:27.381 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:27.381 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:27.381 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:27.381 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:27.381 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:27.381 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:27.381 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:27.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:27.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:27.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:27.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:27.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:27.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:27.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:27.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:27.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:28.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:28.078 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:28.078 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:28.078 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:28.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:28.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:28.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:28.565 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:28.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:28.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:28.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:28.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:28.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:29.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21838091 2020-11-16T22:36:29.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21838091 '!=' null 2020-11-16T22:36:29.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:29.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:29.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:29.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:29.228 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:29.229 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:29.229 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:29.229 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:29.229 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:29.229 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:29.229 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:29.229 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:29.230 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:29.230 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:29.230 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:29.230 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:29.230 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:29.230 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:29.230 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:29.231 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:29.231 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:29.231 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:29.231 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:29.231 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:29.231 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:29.231 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:29.232 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:29.232 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:29.232 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:29.232 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:29.232 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:29.232 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:29.232 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:29.232 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:29.233 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:29.233 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:29.233 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:29.233 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:29.233 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:29.233 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:29.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:29.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:29.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:29.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:29.339 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:29.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:29.340 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:29.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:29.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:29.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:29.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:30.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:30.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:30.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597050 2020-11-16T22:36:30.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597050 '!=' null 2020-11-16T22:36:30.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:30.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:30.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:30.689 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:30.789 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:30.789 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:30.789 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:30.790 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:30.790 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:30.790 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:30.790 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:30.790 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:30.790 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:30.790 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:30.791 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:30.791 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:30.791 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:30.791 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:30.791 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:30.791 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:30.791 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:30.792 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:30.792 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:30.792 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:30.792 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:30.792 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:30.792 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:30.792 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:30.793 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:30.793 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:30.793 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:30.793 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:30.793 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:30.793 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:30.793 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:30.793 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:30.794 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:30.794 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:30.794 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:30.794 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:30.798 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:30.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:30.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:30.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:30.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:30.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:30.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:30.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:30.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:31.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:31.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:31.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:31.268 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:31.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:31.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:31.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:31.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:31.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:31.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:31.796 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:31.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:31.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597508 2020-11-16T22:36:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597508 '!=' null 2020-11-16T22:36:32.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:32.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:32.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:32.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:32.380 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:32.381 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:32.381 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:32.381 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:32.381 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:32.381 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:32.382 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:32.382 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:32.382 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:32.382 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:32.382 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:32.382 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:32.382 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:32.383 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:32.383 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:32.383 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:32.383 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:32.383 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:32.383 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:32.383 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:32.384 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:32.384 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:32.384 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:32.384 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:32.384 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:32.384 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:32.384 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:32.384 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:32.385 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:32.385 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:32.385 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:32.385 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:32.385 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:32.385 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:32.385 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:32.385 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:32.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:32.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:32.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:32.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:32.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:32.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:32.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:32.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:32.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:32.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:32.869 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:32.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:32.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:33.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:33.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:33.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:33.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:33.356 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:33.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:33.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:33.357 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:33.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24459752 2020-11-16T22:36:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24459752 '!=' null 2020-11-16T22:36:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:33.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:33.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:33.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:33.951 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:33.951 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:33.951 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:33.952 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:33.952 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:33.952 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:33.952 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:33.952 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:33.952 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:33.952 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:33.953 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:33.953 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:33.953 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:33.953 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:33.953 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:33.953 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:33.953 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:33.954 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:33.954 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:33.954 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:33.954 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:33.954 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:33.954 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:33.954 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:33.954 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:33.955 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:33.955 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:33.955 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:33.955 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:33.955 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:33.955 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:33.955 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:33.956 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:33.956 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:33.956 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:33.956 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:33.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:34.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:34.061 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:34.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:34.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:34.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:34.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:34.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:34.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:34.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:34.424 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:34.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:34.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:34.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:34.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:34.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:34.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:34.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:34.896 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:34.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24459752 2020-11-16T22:36:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24459752 '!=' null 2020-11-16T22:36:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:35.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:35.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:35.484 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:35.485 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:35.485 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:35.485 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:35.485 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:35.485 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:35.485 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:35.485 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:35.486 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:35.486 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:35.486 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:35.486 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:35.486 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:35.486 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:35.486 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:35.486 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:35.487 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:35.487 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:35.487 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:35.487 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:35.487 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:35.487 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:35.487 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:35.488 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:35.488 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:35.488 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:35.488 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:35.488 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:35.488 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:35.488 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:35.488 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:35.489 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:35.489 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:35.489 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:35.489 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:35.489 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:35.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:35.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:35.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:35.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:35.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:35.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:35.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:35.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:35.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:35.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:35.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:35.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:35.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:36.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:36.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:36.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:36.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:36.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:36.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:36.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:36.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:36.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:37.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597053 2020-11-16T22:36:37.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597053 '!=' null 2020-11-16T22:36:37.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:37.259 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:37.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:37.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:37.358 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:37.358 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:37.359 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:37.359 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:37.359 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:37.359 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:37.359 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:37.359 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:37.359 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:37.360 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:37.360 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:37.360 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:37.360 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:37.360 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:37.360 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:37.360 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:37.360 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:37.361 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:37.361 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:37.361 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:37.361 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:37.361 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:37.361 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:37.361 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:37.362 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:37.362 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:37.362 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:37.362 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:37.362 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:37.362 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:37.362 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:37.363 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:37.363 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:37.363 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:37.363 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:37.363 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:37.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:37.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:37.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:37.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:37.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:37.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:37.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:37.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:37.470 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:37.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:37.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:37.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:37.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:38.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:38.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:38.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:38.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:38.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:38.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:38.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:38.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:38.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:38.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25332967 2020-11-16T22:36:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25332967 '!=' null 2020-11-16T22:36:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:38.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:38.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:38.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:39.030 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:39.031 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:39.031 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:39.031 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:39.031 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:39.031 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:39.031 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:39.032 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:39.032 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:39.032 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:39.032 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:39.032 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:39.032 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:39.032 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:39.033 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:39.033 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:39.033 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:39.033 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:39.033 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:39.033 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:39.033 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:39.034 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:39.034 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:39.034 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:39.034 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:39.034 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:39.034 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:39.034 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:39.035 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:39.035 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:39.035 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:39.035 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:39.035 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:39.035 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:39.035 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:39.036 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:39.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:39.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:39.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:39.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:39.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:39.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:39.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:39.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:39.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:39.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:39.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:39.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:39.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:40.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:40.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597591 2020-11-16T22:36:40.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597591 '!=' null 2020-11-16T22:36:40.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:40.931 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:40.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:40.931 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:41.033 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:41.033 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:41.033 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:41.033 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:41.034 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:41.034 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:41.034 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:41.034 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:41.034 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:41.034 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:41.034 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:41.035 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:41.035 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:41.035 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:41.035 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:41.035 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:41.035 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:41.035 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:41.035 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:41.036 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:41.036 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:41.036 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:41.036 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:41.036 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:41.036 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:41.036 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:41.037 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:41.037 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:41.037 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:41.037 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:41.037 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:41.037 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:41.037 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:41.037 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:41.038 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:41.038 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:41.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:41.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:41.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:41.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:41.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:41.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:41.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:41.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:41.144 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:41.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:41.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:41.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:41.522 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:42.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:42.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:42.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:42.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:42.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:42.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:42.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:42.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:42.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:42.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597625 2020-11-16T22:36:42.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597625 '!=' null 2020-11-16T22:36:42.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:42.571 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:42.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:42.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:42.660 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:42.660 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:42.660 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:42.660 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:42.660 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:42.660 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:42.661 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:42.661 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:42.661 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:42.661 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:42.661 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:42.661 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:42.661 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:42.662 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:42.662 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:42.662 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:42.662 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:42.662 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:42.662 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:42.662 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:42.663 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:42.663 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:42.663 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:42.663 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:42.663 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:42.663 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:42.663 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:42.663 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:42.664 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:42.664 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:42.664 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:42.664 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:42.664 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:42.664 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:42.664 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:42.664 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:42.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:42.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:42.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:42.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:42.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:42.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:42.771 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:42.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:42.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:43.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:43.489 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:43.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:43.490 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:43.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:43.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:43.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:43.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:44.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24459760 2020-11-16T22:36:44.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24459760 '!=' null 2020-11-16T22:36:44.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:44.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:44.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:44.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:44.532 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:44.533 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:44.533 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:44.534 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:44.534 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:44.534 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:44.534 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:44.535 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:44.535 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:44.535 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:44.535 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:44.536 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:44.536 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:44.536 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:44.536 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:44.537 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:44.537 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:44.537 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:44.537 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:44.538 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:44.538 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:44.538 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:44.538 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:44.539 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:44.539 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:44.539 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:44.540 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:44.540 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:44.540 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:44.540 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:44.541 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:44.541 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:44.541 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:44.541 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:44.541 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:44.542 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:44.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:44.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:44.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:44.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:44.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:44.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:44.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:44.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:44.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:45.038 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:45.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:45.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:45.039 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:45.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:45.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:45.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:45.543 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:45.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16598004 2020-11-16T22:36:45.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16598004 '!=' null 2020-11-16T22:36:45.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:45.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:45.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:45.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:46.091 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:46.091 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:46.091 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:46.091 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:46.092 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:46.092 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:46.092 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:46.092 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:46.092 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:46.092 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:46.092 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:46.092 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:46.093 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:46.093 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:46.093 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:46.093 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:46.093 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:46.093 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:46.093 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:46.094 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:46.094 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:46.094 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:46.094 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:46.094 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:46.094 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:46.094 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:46.095 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:46.095 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:46.095 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:46.095 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:46.095 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:46.095 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:46.095 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:46.095 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:46.096 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:46.096 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:46.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:46.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:46.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:46.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:46.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:46.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:46.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:46.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:46.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:46.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:46.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:46.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:46.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:47.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:47.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:47.279 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:47.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:47.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:47.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:47.280 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:47.280 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:47.281 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:47.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24461233 2020-11-16T22:36:47.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24461233 '!=' null 2020-11-16T22:36:47.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:47.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:47.757 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:47.757 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:47.855 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:47.856 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:47.856 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:47.856 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:47.856 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:47.856 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:47.856 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:47.856 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:47.857 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:47.857 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:47.857 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:47.857 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:47.857 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:47.857 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:47.857 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:47.857 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:47.858 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:47.858 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:47.858 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:47.858 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:47.858 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:47.858 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:47.858 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:47.859 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:47.859 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:47.859 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:47.859 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:47.859 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:47.859 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:47.859 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:47.859 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:47.860 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:47.860 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:47.860 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:47.860 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:47.860 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:47.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:47.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:47.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:47.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:47.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:47.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:47.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:47.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:47.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:48.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:48.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:48.326 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:48.832 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:48.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:48.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:48.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:49.293 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24461233 2020-11-16T22:36:49.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24461233 '!=' null 2020-11-16T22:36:49.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:49.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:49.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:49.294 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:49.387 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:49.387 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:49.387 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:49.387 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:49.387 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:49.388 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:49.388 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:49.388 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:49.388 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:49.388 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:49.388 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:49.388 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:49.389 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:49.389 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:49.389 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:49.389 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:49.389 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:49.389 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:49.389 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:49.389 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:49.390 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:49.390 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:49.390 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:49.390 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:49.390 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:49.390 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:49.390 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:49.391 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:49.391 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:49.391 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:49.391 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:49.391 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:49.391 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:49.391 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:49.392 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:49.392 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:49.395 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:49.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:49.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:49.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:49.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:49.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:49.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:49.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:49.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:49.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:49.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:49.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:49.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:50.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:50.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:50.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:50.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:50.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724913 2020-11-16T22:36:50.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724913 '!=' null 2020-11-16T22:36:50.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:50.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:50.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:50.888 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:50.984 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:50.984 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:50.984 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:50.985 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:50.985 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:50.985 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:50.985 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:50.985 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:50.985 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:50.985 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:50.985 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:50.986 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:50.986 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:50.986 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:50.986 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:50.986 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:50.986 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:50.986 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:50.987 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:50.987 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:50.987 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:50.987 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:50.987 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:50.987 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:50.987 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:50.988 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:50.988 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:50.988 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:50.988 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:50.988 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:50.988 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:50.988 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:50.988 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:50.989 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:50.989 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:50.989 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:50.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:51.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:51.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:51.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:51.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:51.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:51.095 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:51.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:51.096 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:51.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:51.465 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:51.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:51.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:52.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:52.430 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725115 2020-11-16T22:36:52.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725115 '!=' null 2020-11-16T22:36:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:52.992 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:52.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:52.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:53.079 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:53.079 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:53.079 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:53.079 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:53.079 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:53.079 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:53.080 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:53.080 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:53.080 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:53.080 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:53.080 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:53.080 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:53.080 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:53.080 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:53.081 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:53.081 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:53.081 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:53.081 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:53.081 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:53.081 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:53.081 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:53.082 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:53.082 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:53.082 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:53.082 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:53.082 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:53.082 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:53.082 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:53.083 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:53.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:53.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:53.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:53.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:53.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:53.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:53.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:53.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:53.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:53.549 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:53.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:53.550 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:53.551 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:54.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:54.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:54.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:54.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:54.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:54.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:54.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:54.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:54.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:54.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24461100 2020-11-16T22:36:54.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24461100 '!=' null 2020-11-16T22:36:54.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:54.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:54.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:54.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:54.642 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:54.643 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:54.643 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:54.643 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:54.643 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:54.643 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:54.643 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:54.644 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:54.644 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:54.644 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:54.644 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:54.644 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:54.644 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:54.644 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:54.645 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:54.645 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:54.645 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:54.645 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:54.645 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:54.645 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:54.645 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:54.646 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:54.646 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:54.646 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:54.646 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:54.646 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:54.646 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:54.646 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:54.646 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:54.647 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:54.647 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:54.647 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:54.647 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:54.647 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:54.647 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:54.647 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:54.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:54.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:54.752 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:54.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:54.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:54.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:54.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:54.753 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:54.754 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:55.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:55.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:55.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:55.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:55.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:55.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:55.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:55.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:55.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:55.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:55.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:55.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:55.611 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:56.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16598144 2020-11-16T22:36:56.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16598144 '!=' null 2020-11-16T22:36:56.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:56.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:56.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:56.079 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:56.176 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:56.176 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:56.176 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:56.176 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:56.176 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:56.177 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:56.177 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:56.177 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:56.177 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:56.177 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:56.177 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:56.177 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:56.178 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:56.178 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:56.178 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:56.178 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:56.178 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:56.178 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:56.178 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:56.179 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:56.179 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:56.179 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:56.179 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:56.179 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:56.179 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:56.179 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:56.179 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:56.180 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:56.180 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:56.180 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:56.180 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:56.180 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:56.180 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:56.180 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:56.181 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:56.181 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:56.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:56.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:56.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:56.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:56.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:56.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:56.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:56.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:56.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:56.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:56.647 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:56.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:56.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:57.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:57.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:57.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:57.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16598144 2020-11-16T22:36:57.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16598144 '!=' null 2020-11-16T22:36:57.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:57.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:57.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:57.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:57.737 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:57.737 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:57.737 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:57.737 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:57.738 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:57.738 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:57.738 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:57.738 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:57.738 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:57.738 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:57.738 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:57.739 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:57.739 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:57.739 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:57.739 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:57.739 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:57.739 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:57.739 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:57.740 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:57.740 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:57.740 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:57.740 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:57.740 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:57.740 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:57.740 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:57.740 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:57.741 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:57.741 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:57.741 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:57.741 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:57.741 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:57.741 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:57.741 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:57.742 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:57.742 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:57.742 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:57.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:57.846 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:57.847 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:57.847 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:57.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:57.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:57.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:57.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:57.848 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:36:58.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:36:58.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:36:58.628 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:36:58.629 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:36:59.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:36:59.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:36:59.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:36:59.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:36:59.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:36:59.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:36:59.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:36:59.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:36:59.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:36:59.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24459749 2020-11-16T22:36:59.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24459749 '!=' null 2020-11-16T22:36:59.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:36:59.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:59.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:36:59.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:36:59.670 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:36:59.671 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:36:59.671 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:59.671 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:59.671 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:36:59.671 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:59.671 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:36:59.672 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:36:59.672 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:59.672 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:36:59.672 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:59.672 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:59.672 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:59.672 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:59.672 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:36:59.673 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:36:59.673 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:59.673 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:59.673 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:59.673 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:59.673 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:36:59.673 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:36:59.674 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:36:59.674 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:36:59.674 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:36:59.674 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:36:59.674 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:59.674 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:36:59.674 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:36:59.675 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:36:59.675 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:36:59.675 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:36:59.675 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:36:59.675 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:36:59.675 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:36:59.675 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:36:59.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:36:59.782 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:36:59.783 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:36:59.783 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:36:59.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:36:59.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:36:59.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:36:59.784 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:36:59.785 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:00.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:00.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:00.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:00.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:00.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:00.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:00.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:00.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:00.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:00.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:00.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:00.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:00.697 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:01.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597394 2020-11-16T22:37:01.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597394 '!=' null 2020-11-16T22:37:01.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:01.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:01.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:01.199 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:01.293 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:01.293 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:01.293 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:01.293 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:01.293 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:01.293 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:01.294 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:01.294 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:01.294 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:01.294 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:01.294 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:01.294 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:01.294 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:01.295 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:01.295 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:01.295 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:01.295 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:01.295 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:01.295 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:01.295 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:01.295 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:01.296 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:01.296 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:01.296 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:01.296 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:01.296 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:01.296 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:01.296 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:01.297 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:01.297 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:01.297 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:01.297 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:01.297 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:01.297 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:01.297 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:01.297 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:01.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:01.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:01.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:01.403 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:01.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:01.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:01.404 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:01.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:01.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:01.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:01.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:02.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:02.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:02.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:02.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:02.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:02.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:02.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:02.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:02.284 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:02.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597267 2020-11-16T22:37:02.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597267 '!=' null 2020-11-16T22:37:02.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:02.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:02.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:02.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:02.858 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:02.858 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:02.858 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:02.858 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:02.858 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:02.859 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:02.859 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:02.859 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:02.859 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:02.859 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:02.859 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:02.859 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:02.860 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:02.860 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:02.860 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:02.860 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:02.860 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:02.860 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:02.860 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:02.861 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:02.861 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:02.861 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:02.861 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:02.861 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:02.861 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:02.861 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:02.861 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:02.862 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:02.862 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:02.862 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:02.862 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:02.862 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:02.862 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:02.862 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:02.863 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:02.863 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:02.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:02.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:02.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:02.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:02.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:02.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:02.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:02.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:02.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:03.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:03.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:03.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:03.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:03.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:03.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:03.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:03.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:04.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25331938 2020-11-16T22:37:04.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25331938 '!=' null 2020-11-16T22:37:04.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:04.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:04.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:04.437 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:04.532 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:04.533 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:04.533 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:04.533 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:04.533 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:04.533 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:04.533 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:04.533 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:04.534 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:04.534 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:04.534 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:04.534 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:04.534 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:04.534 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:04.534 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:04.535 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:04.535 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:04.535 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:04.535 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:04.535 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:04.535 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:04.535 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:04.536 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:04.536 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:04.536 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:04.536 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:04.536 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:04.536 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:04.536 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:04.537 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:04.537 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:04.537 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:04.537 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:04.537 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:04.537 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:04.537 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:04.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:04.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:04.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:04.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:04.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:04.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:04.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:04.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:04.644 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:05.030 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:05.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:05.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:05.688 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:05.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:05.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:05.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:05.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:05.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:05.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:05.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:05.690 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:06.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25331938 2020-11-16T22:37:06.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25331938 '!=' null 2020-11-16T22:37:06.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:06.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:06.177 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:06.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:06.279 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:06.279 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:06.279 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:06.279 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:06.279 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:06.279 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:06.280 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:06.280 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:06.280 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:06.280 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:06.280 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:06.280 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:06.280 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:06.281 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:06.281 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:06.281 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:06.281 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:06.281 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:06.281 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:06.281 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:06.281 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:06.282 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:06.282 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:06.282 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:06.282 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:06.282 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:06.282 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:06.282 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:06.283 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:06.283 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:06.283 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:06.283 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:06.283 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:06.283 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:06.283 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:06.283 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:06.288 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:06.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:06.390 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:06.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:06.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:06.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:06.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:06.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:06.392 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:06.778 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:06.779 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:06.779 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:07.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:07.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:07.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:07.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:07.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:07.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:07.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:07.359 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:07.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:07.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22711794 2020-11-16T22:37:07.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22711794 '!=' null 2020-11-16T22:37:07.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:07.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:07.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:07.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:08.015 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:08.016 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:08.016 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:08.016 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:08.016 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:08.016 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:08.016 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:08.016 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:08.017 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:08.017 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:08.017 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:08.017 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:08.017 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:08.017 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:08.017 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:08.018 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:08.018 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:08.018 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:08.018 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:08.018 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:08.018 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:08.018 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:08.019 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:08.019 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:08.019 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:08.019 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:08.019 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:08.019 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:08.019 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:08.019 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:08.020 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:08.020 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:08.020 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:08.020 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:08.020 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:08.020 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:08.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:08.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:08.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:08.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:08.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:08.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:08.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:08.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:08.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:08.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:08.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:08.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:08.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:08.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:08.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:08.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:08.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:08.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:08.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:08.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:08.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:09.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22711794 2020-11-16T22:37:09.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22711794 '!=' null 2020-11-16T22:37:09.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:09.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:09.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:09.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:09.528 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:09.528 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:09.528 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:09.528 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:09.529 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:09.529 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:09.529 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:09.529 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:09.529 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:09.529 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:09.529 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:09.530 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:09.530 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:09.530 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:09.530 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:09.530 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:09.530 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:09.530 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:09.530 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:09.531 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:09.531 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:09.531 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:09.531 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:09.531 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:09.531 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:09.531 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:09.532 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:09.532 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:09.532 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:09.532 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:09.532 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:09.532 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:09.532 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:09.533 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:09.533 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:09.533 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:09.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:09.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:09.637 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:09.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:09.638 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:09.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:09.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:09.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:09.639 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:10.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:10.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:10.081 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:10.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:10.587 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:10.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:10.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:10.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:10.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:10.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:10.588 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:10.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:10.589 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:11.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850381 2020-11-16T22:37:11.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850381 '!=' null 2020-11-16T22:37:11.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:11.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:11.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:11.064 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:11.158 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:11.159 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:11.159 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:11.159 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:11.159 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:11.159 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:11.159 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:11.159 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:11.160 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:11.160 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:11.160 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:11.160 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:11.160 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:11.160 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:11.160 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:11.161 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:11.161 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:11.161 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:11.161 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:11.161 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:11.161 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:11.161 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:11.162 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:11.162 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:11.162 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:11.162 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:11.162 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:11.162 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:11.162 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:11.162 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:11.163 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:11.163 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:11.163 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:11.163 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:11.163 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:11.163 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:11.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:11.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:11.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:11.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:11.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:11.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:11.269 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:11.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:11.270 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:11.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:11.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:11.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:11.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:12.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:12.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:12.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:12.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:12.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:12.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:12.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:12.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:12.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:12.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850585 2020-11-16T22:37:12.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850585 '!=' null 2020-11-16T22:37:12.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:12.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:12.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:12.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:12.719 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:12.719 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:12.720 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:12.720 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:12.720 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:12.720 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:12.720 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:12.720 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:12.720 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:12.721 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:12.721 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:12.721 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:12.721 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:12.721 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:12.721 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:12.721 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:12.722 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:12.722 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:12.722 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:12.722 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:12.722 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:12.722 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:12.722 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:12.722 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:12.723 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:12.723 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:12.723 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:12.723 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:12.723 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:12.723 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:12.723 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:12.724 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:12.724 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:12.724 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:12.724 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:12.724 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:12.728 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:12.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:12.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:12.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:12.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:12.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:12.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:12.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:12.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:13.200 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:13.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:13.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:13.201 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:13.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:13.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:13.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:13.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:13.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:13.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:13.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:13.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:13.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:14.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712748 2020-11-16T22:37:14.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712748 '!=' null 2020-11-16T22:37:14.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:14.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:14.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:14.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:14.424 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:14.425 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:14.425 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:14.425 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:14.425 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:14.425 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:14.425 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:14.425 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:14.426 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:14.426 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:14.426 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:14.426 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:14.426 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:14.426 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:14.426 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:14.427 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:14.427 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:14.427 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:14.427 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:14.427 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:14.427 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:14.427 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:14.428 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:14.428 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:14.428 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:14.428 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:14.428 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:14.428 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:14.428 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:14.428 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:14.429 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:14.429 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:14.429 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:14.429 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:14.429 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:14.429 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:14.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:14.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:14.535 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:14.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:14.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:14.536 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:14.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:14.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:14.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:14.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:14.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:14.920 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:14.921 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:15.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:15.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:15.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:15.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:15.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:15.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:15.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:15.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:15.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:15.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13977324 2020-11-16T22:37:15.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13977324 '!=' null 2020-11-16T22:37:15.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:15.885 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:15.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:15.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:15.982 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:15.982 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:15.982 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:15.982 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:15.983 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:15.983 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:15.983 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:15.983 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:15.983 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:15.983 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:15.983 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:15.983 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:15.984 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:15.984 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:15.984 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:15.984 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:15.984 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:15.984 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:15.984 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:15.985 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:15.985 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:15.985 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:15.985 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:15.985 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:15.985 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:15.985 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:15.986 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:15.986 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:15.986 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:15.986 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:15.986 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:15.986 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:15.986 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:15.986 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:15.987 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:15.987 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:15.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:16.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:16.092 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:16.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:16.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:16.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:16.093 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:16.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:16.094 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:16.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:16.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:16.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:16.459 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:16.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:16.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:16.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:16.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:16.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:16.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:16.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:16.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:16.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:17.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13977324 2020-11-16T22:37:17.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13977324 '!=' null 2020-11-16T22:37:17.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:17.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:17.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:17.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:17.536 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:17.537 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:17.538 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:17.539 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:17.540 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:17.541 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:17.541 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:17.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:17.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:17.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:17.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:17.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:17.648 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:17.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:17.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:17.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:18.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:18.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:18.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:18.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:18.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:18.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:18.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:18.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:18.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:18.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:18.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:18.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:18.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:19.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21839526 2020-11-16T22:37:19.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21839526 '!=' null 2020-11-16T22:37:19.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:19.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:19.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:19.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:19.309 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:19.310 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:19.310 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:19.310 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:19.310 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:19.310 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:19.311 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:19.311 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:19.311 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:19.311 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:19.311 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:19.311 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:19.311 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:19.311 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:19.312 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:19.312 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:19.312 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:19.312 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:19.312 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:19.312 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:19.312 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:19.313 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:19.313 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:19.313 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:19.313 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:19.313 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:19.313 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:19.313 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:19.314 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:19.314 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:19.314 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:19.314 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:19.314 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:19.314 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:19.314 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:19.314 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:19.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:19.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:19.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:19.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:19.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:19.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:19.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:19.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:19.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:20.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:20.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:20.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:20.034 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:20.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:20.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:20.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:20.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:20.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:20.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:20.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:20.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:20.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724412 2020-11-16T22:37:20.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724412 '!=' null 2020-11-16T22:37:20.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:20.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:20.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:20.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:21.091 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:21.092 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:21.092 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:21.092 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:21.092 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:21.092 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:21.092 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:21.092 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:21.093 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:21.093 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:21.093 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:21.093 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:21.093 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:21.093 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:21.093 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:21.094 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:21.094 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:21.094 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:21.094 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:21.094 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:21.094 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:21.094 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:21.095 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:21.095 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:21.095 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:21.095 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:21.095 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:21.095 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:21.095 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:21.096 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:21.096 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:21.096 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:21.096 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:21.096 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:21.096 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:21.096 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:21.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:21.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:21.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:21.202 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:21.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:21.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:21.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:21.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:21.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:21.567 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:21.568 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:21.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:21.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:22.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:22.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:22.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:22.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:22.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:22.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:22.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:22.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:22.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:22.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15723529 2020-11-16T22:37:22.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15723529 '!=' null 2020-11-16T22:37:22.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:22.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:22.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:22.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:22.791 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:22.791 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:22.791 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:22.791 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:22.791 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:22.791 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:22.792 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:22.792 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:22.792 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:22.792 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:22.792 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:22.792 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:22.792 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:22.793 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:22.793 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:22.793 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:22.793 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:22.793 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:22.793 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:22.793 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:22.793 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:22.794 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:22.794 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:22.794 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:22.794 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:22.794 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:22.794 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:22.794 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:22.795 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:22.795 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:22.795 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:22.795 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:22.795 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:22.795 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:22.795 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:22.795 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:22.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:22.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:22.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:22.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:22.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:22.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:22.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:22.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:22.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:23.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:23.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:23.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:23.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:23.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:23.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:23.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:23.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:23.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:23.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:23.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:23.767 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23584938 2020-11-16T22:37:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23584938 '!=' null 2020-11-16T22:37:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:24.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:24.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:24.374 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:24.374 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:24.374 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:24.375 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:24.375 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:24.375 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:24.375 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:24.375 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:24.375 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:24.375 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:24.376 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:24.376 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:24.376 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:24.376 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:24.376 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:24.376 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:24.376 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:24.376 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:24.377 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:24.377 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:24.377 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:24.377 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:24.377 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:24.377 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:24.377 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:24.378 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:24.378 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:24.378 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:24.378 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:24.378 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:24.378 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:24.378 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:24.378 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:24.379 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:24.379 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:24.379 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:24.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:24.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:24.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:24.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:24.486 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:24.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:24.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:24.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:24.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:24.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:24.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:25.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:25.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:25.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:25.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:25.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:25.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:25.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:25.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:25.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:26.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15723203 2020-11-16T22:37:26.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15723203 '!=' null 2020-11-16T22:37:26.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:26.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:26.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:26.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:26.339 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:26.339 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:26.339 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:26.339 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:26.340 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:26.340 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:26.340 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:26.340 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:26.340 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:26.340 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:26.340 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:26.340 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:26.341 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:26.341 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:26.341 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:26.341 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:26.341 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:26.341 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:26.341 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:26.342 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:26.342 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:26.342 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:26.342 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:26.342 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:26.342 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:26.342 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:26.343 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:26.343 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:26.343 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:26.343 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:26.343 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:26.343 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:26.343 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:26.343 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:26.344 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:26.344 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:26.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:26.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:26.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:26.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:26.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:26.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:26.450 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:26.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:26.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:26.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:26.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:26.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:26.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:27.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:27.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:27.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:27.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:27.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:27.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:27.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:27.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:27.385 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:27.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24458102 2020-11-16T22:37:27.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24458102 '!=' null 2020-11-16T22:37:27.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:27.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:27.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:27.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:28.001 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:28.002 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:28.002 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:28.002 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:28.002 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:28.002 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:28.003 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:28.003 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:28.003 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:28.003 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:28.003 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:28.003 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:28.003 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:28.003 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:28.004 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:28.004 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:28.004 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:28.004 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:28.004 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:28.004 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:28.004 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:28.005 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:28.005 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:28.005 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:28.005 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:28.005 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:28.005 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:28.005 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:28.005 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:28.006 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:28.006 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:28.006 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:28.006 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:28.006 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:28.006 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:28.006 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:28.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:28.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:28.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:28.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:28.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:28.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:28.114 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:28.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:28.115 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:28.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:28.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:28.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:28.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:28.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:28.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:28.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:28.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:28.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:28.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:28.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:28.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:28.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:29.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24458102 2020-11-16T22:37:29.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24458102 '!=' null 2020-11-16T22:37:29.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:29.436 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:29.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:29.436 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:29.534 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:29.534 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:29.535 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:29.535 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:29.535 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:29.535 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:29.535 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:29.535 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:29.535 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:29.536 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:29.536 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:29.536 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:29.536 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:29.536 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:29.536 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:29.536 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:29.537 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:29.537 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:29.537 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:29.537 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:29.537 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:29.538 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:29.538 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:29.538 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:29.538 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:29.539 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:29.539 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:29.539 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:29.539 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:29.540 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:29.540 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:29.540 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:29.540 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:29.540 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:29.541 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:29.541 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:29.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:29.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:29.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:29.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:29.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:29.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:29.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:29.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:29.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:30.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:30.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:30.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:30.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:30.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:30.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:30.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:30.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:30.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:30.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:30.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:30.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:30.493 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:31.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16596644 2020-11-16T22:37:31.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16596644 '!=' null 2020-11-16T22:37:31.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:31.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:31.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:31.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:31.109 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:31.110 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:31.110 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:31.110 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:31.110 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:31.110 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:31.110 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:31.111 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:31.111 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:31.111 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:31.111 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:31.111 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:31.111 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:31.111 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:31.111 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:31.112 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:31.112 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:31.112 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:31.112 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:31.112 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:31.112 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:31.112 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:31.112 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:31.113 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:31.113 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:31.113 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:31.113 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:31.113 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:31.113 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:31.113 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:31.114 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:31.114 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:31.114 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:31.114 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:31.114 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:31.114 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:31.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:31.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:31.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:31.221 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:31.221 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:31.221 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:31.221 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:31.221 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:31.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:31.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:31.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:31.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:31.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:32.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:32.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:32.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:32.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:32.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:32.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:32.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:32.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:32.090 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:32.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16596926 2020-11-16T22:37:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16596926 '!=' null 2020-11-16T22:37:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:32.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:32.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:32.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:32.975 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:32.975 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:32.975 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:32.975 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:32.975 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:32.976 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:32.976 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:32.976 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:32.976 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:32.976 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:32.976 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:32.976 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:32.977 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:32.977 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:32.977 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:32.977 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:32.977 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:32.977 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:32.977 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:32.977 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:32.978 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:32.978 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:32.978 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:32.978 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:32.978 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:32.978 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:32.978 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:32.979 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:32.979 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:32.979 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:32.979 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:32.979 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:32.979 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:32.979 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:32.980 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:32.980 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:32.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:33.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:33.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:33.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:33.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:33.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:33.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:33.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:33.088 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:33.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:33.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:33.475 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:33.476 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:33.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:33.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:33.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:33.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:33.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:33.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:33.951 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:33.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:33.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:34.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24460033 2020-11-16T22:37:34.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24460033 '!=' null 2020-11-16T22:37:34.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:34.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:34.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:34.542 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:34.640 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:34.640 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:34.641 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:34.641 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:34.641 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:34.641 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:34.641 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:34.641 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:34.641 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:34.642 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:34.642 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:34.642 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:34.642 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:34.642 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:34.642 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:34.642 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:34.643 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:34.643 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:34.643 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:34.643 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:34.643 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:34.643 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:34.643 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:34.643 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:34.644 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:34.644 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:34.644 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:34.644 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:34.644 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:34.644 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:34.644 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:34.645 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:34.645 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:34.645 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:34.645 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:34.645 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:34.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:34.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:34.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:34.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:34.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:34.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:34.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:34.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:34.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:35.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:35.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:35.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:35.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:35.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:35.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:35.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:35.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:35.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:36.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597910 2020-11-16T22:37:36.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597910 '!=' null 2020-11-16T22:37:36.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:36.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:36.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:36.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:36.241 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:36.241 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:36.241 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:36.242 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:36.242 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:36.242 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:36.242 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:36.242 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:36.242 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:36.242 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:36.243 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:36.243 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:36.243 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:36.243 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:36.243 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:36.243 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:36.243 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:36.244 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:36.244 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:36.244 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:36.244 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:36.244 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:36.244 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:36.244 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:36.244 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:36.245 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:36.245 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:36.245 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:36.245 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:36.245 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:36.245 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:36.245 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:36.246 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:36.246 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:36.246 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:36.246 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:36.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:36.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:36.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:36.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:36.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:36.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:36.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:36.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:36.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:36.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:36.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:36.747 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:36.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:37.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:37.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:37.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:37.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:37.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:37.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:37.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:37.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:37.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:38.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586823 2020-11-16T22:37:38.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586823 '!=' null 2020-11-16T22:37:38.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:38.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:38.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:38.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:38.107 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:38.107 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:38.107 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:38.108 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:38.108 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:38.108 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:38.108 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:38.108 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:38.108 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:38.108 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:38.109 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:38.109 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:38.109 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:38.109 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:38.109 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:38.109 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:38.109 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:38.109 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:38.110 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:38.110 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:38.110 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:38.110 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:38.110 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:38.110 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:38.110 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:38.111 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:38.111 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:38.111 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:38.111 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:38.111 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:38.111 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:38.111 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:38.111 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:38.112 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:38.112 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:38.112 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:38.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:38.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:38.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:38.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:38.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:38.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:38.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:38.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:38.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:38.574 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:38.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:38.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:38.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:39.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:39.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:39.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:39.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:39.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:39.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:39.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:39.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:39.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:39.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586823 2020-11-16T22:37:39.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586823 '!=' null 2020-11-16T22:37:39.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:39.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:39.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:39.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:39.801 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:39.802 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:39.802 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:39.802 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:39.802 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:39.802 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:39.802 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:39.802 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:39.803 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:39.803 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:39.803 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:39.803 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:39.803 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:39.803 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:39.803 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:39.804 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:39.804 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:39.804 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:39.804 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:39.804 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:39.804 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:39.804 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:39.804 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:39.805 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:39.805 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:39.805 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:39.805 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:39.805 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:39.805 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:39.805 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:39.806 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:39.806 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:39.806 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:39.806 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:39.806 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:39.806 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:39.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:39.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:39.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:39.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:39.913 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:39.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:39.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:39.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:39.914 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:40.296 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:40.296 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:40.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:40.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:40.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:40.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:40.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:40.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:41.294 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850848 2020-11-16T22:37:41.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850848 '!=' null 2020-11-16T22:37:41.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:41.295 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:41.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:41.295 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:41.384 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:41.384 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:41.385 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:41.385 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:41.385 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:41.385 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:41.385 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:41.385 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:41.385 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:41.386 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:41.386 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:41.386 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:41.386 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:41.386 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:41.386 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:41.386 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:41.387 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:41.387 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:41.387 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:41.387 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:41.387 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:41.387 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:41.387 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:41.388 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:41.388 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:41.388 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:41.388 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:41.388 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:41.388 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:41.388 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:41.388 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:41.389 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:41.389 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:41.389 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:41.389 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:41.389 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:41.392 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:41.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:41.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:41.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:41.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:41.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:41.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:41.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:41.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:42.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:42.111 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:42.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:42.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:42.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:42.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:42.580 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:42.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:42.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:42.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:42.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:42.581 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:42.582 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:43.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850713 2020-11-16T22:37:43.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850713 '!=' null 2020-11-16T22:37:43.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:43.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:43.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:43.062 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:43.161 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:43.162 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:43.162 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:43.162 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:43.162 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:43.162 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:43.162 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:43.163 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:43.163 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:43.163 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:43.163 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:43.163 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:43.163 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:43.163 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:43.164 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:43.164 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:43.164 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:43.164 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:43.164 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:43.164 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:43.164 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:43.164 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:43.165 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:43.165 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:43.165 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:43.165 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:43.165 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:43.165 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:43.165 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:43.166 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:43.166 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:43.166 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:43.166 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:43.166 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:43.166 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:43.166 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:43.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:43.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:43.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:43.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:43.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:43.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:43.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:43.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:43.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:43.652 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:43.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:43.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:43.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:44.168 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:44.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:44.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:44.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:44.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:44.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:44.169 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:44.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:44.170 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712488 2020-11-16T22:37:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712488 '!=' null 2020-11-16T22:37:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:44.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:44.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:44.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:44.746 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:44.746 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:44.746 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:44.746 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:44.746 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:44.747 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:44.747 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:44.747 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:44.747 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:44.747 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:44.747 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:44.747 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:44.747 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:44.748 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:44.748 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:44.748 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:44.748 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:44.748 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:44.748 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:44.748 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:44.749 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:44.749 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:44.749 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:44.749 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:44.749 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:44.749 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:44.749 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:44.750 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:44.750 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:44.750 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:44.750 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:44.750 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:44.750 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:44.750 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:44.750 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:44.751 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:44.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:44.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:44.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:44.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:44.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:44.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:44.857 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:44.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:44.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:45.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:45.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:45.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:45.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:45.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:45.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:45.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:45.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:45.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:45.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:45.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:45.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:45.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:46.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850026 2020-11-16T22:37:46.422 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850026 '!=' null 2020-11-16T22:37:46.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:46.423 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:46.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:46.423 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:46.511 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:46.511 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:46.511 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:46.512 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:46.512 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:46.512 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:46.512 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:46.512 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:46.512 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:46.513 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:46.513 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:46.513 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:46.513 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:46.513 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:46.513 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:46.513 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:46.513 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:46.514 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:46.514 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:46.514 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:46.514 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:46.514 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:46.514 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:46.514 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:46.515 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:46.515 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:46.515 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:46.515 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:46.515 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:46.515 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:46.515 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:46.516 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:46.516 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:46.516 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:46.516 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:46.516 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:46.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:46.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:46.620 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:46.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:46.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:46.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:46.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:46.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:46.622 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:47.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:47.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:47.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:47.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:47.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:47.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:47.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:47.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:47.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:47.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:47.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:47.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:47.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:48.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712245 2020-11-16T22:37:48.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712245 '!=' null 2020-11-16T22:37:48.127 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:48.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:48.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:48.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:48.229 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:48.230 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:48.230 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:48.230 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:48.230 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:48.230 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:48.230 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:48.230 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:48.231 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:48.231 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:48.231 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:48.231 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:48.231 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:48.231 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:48.231 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:48.231 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:48.232 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:48.232 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:48.232 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:48.232 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:48.232 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:48.232 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:48.232 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:48.233 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:48.233 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:48.233 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:48.233 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:48.233 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:48.233 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:48.233 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:48.233 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:48.234 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:48.234 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:48.234 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:48.234 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:48.234 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:48.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:48.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:48.340 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:48.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:48.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:48.341 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:48.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:48.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:48.342 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:48.713 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:48.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:48.714 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:48.715 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:49.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:49.220 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:49.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:49.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:49.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:49.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:49.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:49.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:49.222 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:49.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712245 2020-11-16T22:37:49.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712245 '!=' null 2020-11-16T22:37:49.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:49.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:49.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:49.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:49.791 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:49.791 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:49.791 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:49.792 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:49.792 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:49.792 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:49.792 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:49.792 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:49.792 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:49.792 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:49.793 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:49.793 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:49.793 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:49.793 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:49.793 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:49.793 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:49.793 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:49.793 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:49.794 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:49.794 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:49.794 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:49.794 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:49.794 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:49.794 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:49.794 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:49.795 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:49.795 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:49.795 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:49.795 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:49.795 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:49.795 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:49.795 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:49.795 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:49.796 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:49.796 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:49.796 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:49.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:49.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:49.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:49.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:49.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:49.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:49.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:49.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:49.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:50.281 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:50.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:50.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:50.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:50.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:50.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:50.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:50.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:50.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:50.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:50.730 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:50.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:50.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:51.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15723950 2020-11-16T22:37:51.194 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15723950 '!=' null 2020-11-16T22:37:51.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:51.195 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:51.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:51.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:51.299 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:51.300 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:51.300 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:51.300 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:51.300 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:51.300 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:51.300 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:51.301 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:51.301 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:51.301 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:51.301 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:51.301 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:51.301 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:51.301 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:51.304 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:51.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:51.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:51.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:51.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:51.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:51.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:51.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:51.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:51.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:51.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:51.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:51.769 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:51.770 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:52.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:52.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:52.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:52.249 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:53.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724104 2020-11-16T22:37:53.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724104 '!=' null 2020-11-16T22:37:53.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:53.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:53.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:53.092 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:53.192 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:53.192 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:53.192 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:53.192 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:53.192 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:53.193 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:53.193 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:53.193 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:53.193 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:53.193 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:53.193 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:53.193 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:53.194 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:53.194 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:53.194 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:53.194 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:53.194 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:53.194 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:53.194 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:53.195 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:53.195 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:53.195 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:53.195 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:53.195 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:53.195 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:53.195 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:53.195 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:53.196 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:53.196 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:53.196 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:53.196 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:53.196 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:53.196 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:53.196 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:53.197 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:53.197 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:53.201 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:53.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:53.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:53.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:53.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:53.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:53.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:53.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:53.697 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:53.698 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:53.698 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:53.698 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:54.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:54.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:54.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:54.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:54.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:54.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:54.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:54.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:54.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:54.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24459613 2020-11-16T22:37:54.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24459613 '!=' null 2020-11-16T22:37:54.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:54.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:54.687 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:54.688 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:54.790 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:54.790 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:54.790 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:54.791 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:54.791 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:54.791 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:54.791 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:54.791 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:54.791 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:54.791 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:54.792 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:54.792 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:54.792 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:54.792 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:54.792 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:54.792 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:54.792 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:54.792 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:54.793 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:54.793 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:54.793 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:54.793 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:54.793 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:54.793 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:54.793 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:54.794 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:54.794 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:54.794 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:54.794 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:54.794 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:54.794 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:54.794 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:54.794 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:54.795 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:54.795 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:54.795 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:54.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:54.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:54.902 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:54.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:54.902 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:54.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:54.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:54.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:54.903 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:55.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:55.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:55.277 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:55.278 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:56.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:56.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:56.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:56.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:56.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:56.115 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:56.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:56.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:56.116 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:56.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597489 2020-11-16T22:37:56.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597489 '!=' null 2020-11-16T22:37:56.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:56.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:56.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:56.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:56.706 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:56.706 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:56.707 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:56.707 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:56.707 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:56.707 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:56.707 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:56.707 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:56.707 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:56.708 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:56.708 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:56.708 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:56.708 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:56.708 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:56.708 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:56.708 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:56.709 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:56.709 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:56.709 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:56.709 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:56.709 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:56.709 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:56.709 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:56.710 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:56.710 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:56.710 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:56.710 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:56.710 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:56.710 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:56.710 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:56.710 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:56.711 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:56.711 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:56.711 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:56.711 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:56.711 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:56.714 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:56.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:56.817 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:56.817 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:56.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:56.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:56.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:56.818 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:56.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:57.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:57.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:57.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:57.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:57.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:57.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:57.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:57.719 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:57.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:57.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:57.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:57.720 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:57.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:58.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25333225 2020-11-16T22:37:58.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25333225 '!=' null 2020-11-16T22:37:58.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:58.184 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:58.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:58.184 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:58.282 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:37:58.282 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:37:58.282 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:58.283 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:58.283 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:37:58.283 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:58.283 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:37:58.283 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:37:58.283 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:58.283 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:37:58.284 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:58.284 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:58.284 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:58.284 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:58.284 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:37:58.284 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:37:58.284 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:58.284 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:58.285 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:58.285 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:58.285 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:37:58.285 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:37:58.285 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:37:58.285 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:37:58.285 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:37:58.286 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:37:58.286 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:58.286 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:37:58.286 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:37:58.286 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:37:58.286 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:37:58.286 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:37:58.286 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:37:58.287 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:37:58.287 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:37:58.287 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:37:58.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:37:58.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:37:58.393 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:37:58.393 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:37:58.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:37:58.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:37:58.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:37:58.394 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:37:58.395 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:37:58.773 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:37:58.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:37:58.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:37:58.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:37:59.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:37:59.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:37:59.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:37:59.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:37:59.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:37:59.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:37:59.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:37:59.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:37:59.274 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:37:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17470792 2020-11-16T22:37:59.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17470792 '!=' null 2020-11-16T22:37:59.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:37:59.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:37:59.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:37:59.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:00.078 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:00.079 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:00.079 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:00.079 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:00.079 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:00.079 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:00.079 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:00.080 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:00.080 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:00.080 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:00.080 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:00.080 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:00.080 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:00.080 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:00.080 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:00.081 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:00.081 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:00.081 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:00.081 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:00.081 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:00.081 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:00.081 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:00.082 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:00.082 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:38:00.082 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:38:00.082 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:00.082 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:00.082 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:00.082 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:00.083 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:00.083 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:00.083 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:00.083 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:00.083 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:00.083 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:00.083 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:00.088 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:00.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:00.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:00.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:00.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:00.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:00.191 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:00.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:00.192 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:00.813 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:38:00.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:00.814 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:00.815 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:01.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:38:01.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:38:01.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:01.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:01.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:01.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:01.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:01.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:01.508 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:02.470 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17470821 2020-11-16T22:38:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17470821 '!=' null 2020-11-16T22:38:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:02.471 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:02.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:02.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:02.572 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:02.573 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:02.574 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:02.575 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:38:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:38:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:02.576 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:02.577 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:02.578 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:02.581 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:02.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:02.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:02.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:02.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:02.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:02.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:02.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:02.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:03.203 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:38:03.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:03.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:03.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:03.732 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:38:03.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:38:03.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:03.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:03.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:03.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:03.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:03.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:03.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:04.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21838432 2020-11-16T22:38:04.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21838432 '!=' null 2020-11-16T22:38:04.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:04.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:04.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:04.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:04.320 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:04.320 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:04.320 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:04.320 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:04.321 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:04.321 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:04.321 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:04.321 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:04.321 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:04.321 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:04.321 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:04.322 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:04.322 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:04.322 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:04.322 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:04.322 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:04.322 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:04.322 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:04.323 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:04.323 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:04.323 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:04.323 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:04.323 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:04.323 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "7.0", 2020-11-16T22:38:04.323 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 255, 2020-11-16T22:38:04.323 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:04.324 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:04.324 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:04.324 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:04.324 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:04.324 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:04.324 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:04.324 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:04.325 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:04.325 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:04.325 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:04.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:04.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:04.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:04.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:04.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:04.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:04.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:04.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:04.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:04.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:38:04.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:04.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:04.807 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:05.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:38:05.331 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:38:05.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:05.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:05.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:05.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:05.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:05.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:05.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21838432 2020-11-16T22:38:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21838432 '!=' null 2020-11-16T22:38:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:05.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:05.994 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:06.097 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:06.098 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:06.098 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:06.098 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:06.098 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:06.099 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:06.099 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:06.099 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:06.099 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:06.099 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:06.099 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:06.099 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:06.100 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:06.100 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:06.100 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:06.100 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:06.100 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:06.100 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:06.100 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:06.101 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:06.101 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:06.101 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:06.101 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:06.101 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:06.101 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:06.102 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:06.102 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:06.102 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:06.102 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:06.102 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:06.102 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:06.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:06.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:06.209 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:06.209 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:06.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:06.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:06.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:06.210 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:06.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:06.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2020-11-16T22:38:06.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:06.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:06.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:07.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2020-11-16T22:38:07.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2020-11-16T22:38:07.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:07.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:07.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:07.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:07.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:07.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:07.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:07.681 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15723479 2020-11-16T22:38:07.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15723479 '!=' null 2020-11-16T22:38:07.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:07.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:07.682 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:07.682 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:07.788 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:07.788 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:07.788 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:07.788 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:07.788 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:07.789 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:07.789 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:07.789 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:07.789 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:07.789 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:07.789 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:07.789 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:07.790 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:07.790 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:07.790 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:07.790 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:07.790 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:07.790 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:07.791 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:07.791 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:07.791 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:07.791 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:07.791 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:07.791 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:07.791 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:07.791 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:07.792 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:07.792 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:07.792 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:07.792 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:07.792 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:07.797 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:07.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:07.899 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:07.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:07.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:07.900 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:07.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:07.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:07.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:08.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:08.282 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:08.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:08.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:08.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:08.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 0 2020-11-16T22:38:08.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T22:38:08.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=1 2020-11-16T22:38:08.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:08.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:09.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:09.107 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:09.107 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:09.107 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:09.107 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:09.110 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:09.110 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:09.110 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:09.110 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:09.110 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:09.110 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:09.110 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:09.111 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:09.111 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:09.111 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:09.111 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:09.111 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:09.111 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:09.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:09.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:09.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:09.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:09.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:09.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:09.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:09.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:09.219 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:09.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:09.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:09.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:09.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:10.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:10.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:10.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:10.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:10.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:10.148 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15723330 2020-11-16T22:38:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15723330 '!=' null 2020-11-16T22:38:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:10.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:10.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:10.714 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:10.714 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:10.714 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:10.714 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:10.714 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:10.715 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:10.715 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:10.715 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:10.715 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:10.715 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:10.715 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:10.715 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:10.716 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:10.716 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:10.716 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:10.716 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:10.716 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:10.716 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:10.716 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:10.716 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:10.717 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:10.717 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:10.717 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:10.717 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:10.717 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:10.717 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:10.717 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:10.718 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:10.718 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:10.718 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:10.718 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:10.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:10.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:10.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:10.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:10.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:10.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:10.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:10.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:10.827 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:11.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:11.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:11.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:11.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:11.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:11.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=9171909 2020-11-16T22:38:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 9171909 '!=' null 2020-11-16T22:38:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:12.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:12.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:12.812 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:12.812 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:12.813 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:12.813 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:12.813 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:12.813 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:12.813 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:12.813 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:12.813 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:12.814 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:12.814 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:12.814 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:12.814 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:12.814 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:12.814 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:12.815 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:12.815 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:12.815 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:12.815 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:12.815 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:12.815 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:12.815 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:12.815 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:12.816 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:12.816 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:12.816 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:12.816 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:12.816 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:12.816 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:12.816 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:12.817 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:12.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:12.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:12.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:12.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:12.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:12.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:12.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:12.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:12.925 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:13.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:13.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:13.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:13.305 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:13.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:13.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:13.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:13.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:13.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:13.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:13.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:13.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:13.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:14.385 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26422748 2020-11-16T22:38:14.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26422748 '!=' null 2020-11-16T22:38:14.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:14.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:14.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:14.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:14.484 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:14.484 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:14.484 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:14.485 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:14.485 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:14.485 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:14.485 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:14.485 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:14.485 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:14.485 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:14.486 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:14.486 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:14.486 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:14.486 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:14.486 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:14.486 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:14.486 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:14.486 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:14.487 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:14.487 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:14.487 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:14.487 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:14.487 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:14.487 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:14.487 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:14.488 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:14.488 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:14.488 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:14.488 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:14.488 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:14.488 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:14.492 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:14.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:14.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:14.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:14.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:14.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:14.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:14.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:14.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:14.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:14.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:14.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:14.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:15.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:15.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:15.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:15.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:15.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:15.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:15.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:15.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:16.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22012689 2020-11-16T22:38:16.118 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22012689 '!=' null 2020-11-16T22:38:16.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:16.119 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:16.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:16.119 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:16.219 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:16.220 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:16.220 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:16.220 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:16.220 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:16.220 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:16.220 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:16.220 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:16.221 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:16.221 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:16.221 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:16.221 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:16.221 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:16.221 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:16.221 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:16.222 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:16.222 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:16.222 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:16.222 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:16.222 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:16.222 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:16.222 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:16.223 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:16.223 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:16.223 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:16.223 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:16.223 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:16.223 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:16.223 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:16.223 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:16.224 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:16.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:16.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:16.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:16.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:16.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:16.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:16.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:16.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:16.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:16.720 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:16.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:16.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:16.721 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:17.221 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:17.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:17.223 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:17.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18924109 2020-11-16T22:38:17.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18924109 '!=' null 2020-11-16T22:38:17.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:17.821 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:17.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:17.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:17.925 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:17.925 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:17.926 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:17.926 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:17.926 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:17.926 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:17.926 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:17.926 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:17.926 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:17.927 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:17.927 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:17.927 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:17.927 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:17.927 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:17.927 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:17.927 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:17.928 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:17.928 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:17.928 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:17.928 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:17.928 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:17.928 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:17.928 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:17.929 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:17.929 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:17.929 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:17.929 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:17.929 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:17.929 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:17.929 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:17.930 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:17.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:18.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:18.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:18.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:18.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:18.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:18.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:18.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:18.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:18.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:18.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:18.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:18.422 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:18.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:18.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:18.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:18.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:18.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:18.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:18.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:18.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:18.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:19.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18924109 2020-11-16T22:38:19.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18924109 '!=' null 2020-11-16T22:38:19.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:19.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:19.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:19.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:19.539 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:19.540 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:19.540 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:19.540 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:19.540 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:19.540 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:19.540 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:19.541 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:19.541 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:19.541 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:19.541 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:19.541 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:19.541 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:19.541 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:19.541 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:19.542 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:19.542 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:19.542 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:19.542 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:19.542 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:19.542 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:19.543 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:19.543 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:19.543 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:19.543 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:19.543 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:19.543 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:19.543 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:19.544 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:19.544 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:19.544 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:19.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:19.649 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:19.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:19.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:19.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:19.650 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:19.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:20.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:20.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:20.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:20.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:20.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:20.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:20.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:20.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:20.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:20.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:20.500 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:20.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:20.501 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:21.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22711397 2020-11-16T22:38:21.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22711397 '!=' null 2020-11-16T22:38:21.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:21.278 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:21.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:21.279 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:21.385 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:21.385 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:21.385 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:21.385 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:21.386 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:21.386 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:21.386 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:21.386 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:21.386 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:21.386 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:21.386 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:21.387 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:21.387 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:21.387 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:21.387 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:21.387 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:21.387 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:21.387 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:21.387 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:21.388 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:21.388 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:21.388 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:21.388 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:21.388 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:21.388 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:21.388 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:21.389 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:21.389 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:21.389 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:21.389 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:21.389 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:21.394 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:21.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:21.496 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:21.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:21.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:21.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:21.497 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:21.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:21.498 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:21.881 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:21.881 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:21.882 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:21.882 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:22.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:22.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:22.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:22.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:22.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:22.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:22.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:22.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:22.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:22.852 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22711698 2020-11-16T22:38:22.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22711698 '!=' null 2020-11-16T22:38:22.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:22.853 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:22.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:22.853 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:22.947 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:22.947 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:22.948 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:22.948 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:22.948 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:22.948 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:22.948 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:22.948 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:22.948 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:22.949 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:22.949 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:22.949 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:22.949 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:22.949 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:22.949 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:22.949 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:22.949 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:22.950 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:22.950 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:22.950 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:22.950 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:22.950 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:22.950 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:22.950 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:22.951 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:22.951 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:22.951 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:22.951 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:22.951 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:22.951 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:22.951 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:22.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:23.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:23.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:23.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:23.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:23.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:23.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:23.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:23.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:23.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:23.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:24.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:24.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:24.275 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:24.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=31447769 2020-11-16T22:38:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 31447769 '!=' null 2020-11-16T22:38:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:24.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:24.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:24.745 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:24.846 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:24.846 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:24.846 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:24.846 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:24.846 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:24.847 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:24.847 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:24.847 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:24.847 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:24.847 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:24.847 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:24.848 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:24.848 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:24.848 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:24.848 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:24.848 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:24.848 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:24.848 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:24.849 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:24.849 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:24.849 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:24.849 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:24.849 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:24.849 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:24.849 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:24.849 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:24.850 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:24.850 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:24.850 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:24.850 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:24.850 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:24.854 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:24.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:24.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:24.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:24.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:24.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:24.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:24.957 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:24.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:25.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:25.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:25.334 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:25.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:25.794 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:25.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:25.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:25.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:25.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:25.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:25.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:25.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:25.796 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19218281 2020-11-16T22:38:26.282 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19218281 '!=' null 2020-11-16T22:38:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:26.283 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:26.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:26.283 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:26.373 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:26.374 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:26.374 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:26.374 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:26.374 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:26.374 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:26.374 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:26.374 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:26.375 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:26.375 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:26.375 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:26.375 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:26.375 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:26.375 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:26.375 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:26.376 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:26.376 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:26.376 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:26.376 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:26.376 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:26.376 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:26.376 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:26.377 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:26.377 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:26.377 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:26.377 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:26.377 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:26.377 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:26.377 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:26.377 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:26.378 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:26.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:26.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:26.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:26.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:26.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:26.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:26.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:26.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:26.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:26.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:26.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:26.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:26.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:27.749 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:27.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:27.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:27.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:27.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:27.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:27.751 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:27.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:27.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:28.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19218936 2020-11-16T22:38:28.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19218936 '!=' null 2020-11-16T22:38:28.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:28.242 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:28.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:28.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:28.334 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:28.334 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:28.334 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:28.335 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:28.335 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:28.335 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:28.335 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:28.335 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:28.335 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:28.335 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:28.336 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:28.336 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:28.336 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:28.336 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:28.336 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:28.336 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:28.336 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:28.337 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:28.337 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:28.337 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:28.337 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:28.337 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:28.337 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:28.337 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:28.338 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:28.338 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:28.338 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:28.338 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:28.338 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:28.338 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:28.338 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:28.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:28.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:28.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:28.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:28.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:28.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:28.445 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:28.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:28.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:28.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:28.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:28.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:29.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:29.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:29.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:29.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:29.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:29.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:29.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:29.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:29.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:29.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27081618 2020-11-16T22:38:29.786 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27081618 '!=' null 2020-11-16T22:38:29.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:29.787 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:29.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:29.787 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:29.888 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:29.888 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:29.889 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:29.889 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:29.889 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:29.889 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:29.889 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:29.889 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:29.890 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:29.890 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:29.890 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:29.890 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:29.890 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:29.890 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:29.890 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:29.891 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:29.891 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:29.891 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:29.891 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:29.891 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:29.891 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:29.891 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:29.892 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:29.892 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:29.892 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:29.892 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:29.892 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:29.892 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:29.892 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:29.892 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:29.893 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:29.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:29.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:29.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:29.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:29.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:30.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:30.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:30.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:30.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:30.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:30.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:30.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:30.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:30.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:30.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:30.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:30.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:30.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:30.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:30.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:30.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:30.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:31.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27081618 2020-11-16T22:38:31.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27081618 '!=' null 2020-11-16T22:38:31.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:31.352 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:31.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:31.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:31.450 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:31.450 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:31.450 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:31.450 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:31.451 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:31.451 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:31.451 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:31.451 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:31.451 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:31.451 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:31.451 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:31.452 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:31.452 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:31.452 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:31.452 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:31.452 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:31.452 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:31.452 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:31.452 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:31.453 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:31.453 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:31.453 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:31.453 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:31.453 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:31.453 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:31.453 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:31.454 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:31.454 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:31.454 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:31.454 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:31.454 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:31.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:31.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:31.560 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:31.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:31.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:31.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:31.561 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:31.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:31.562 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:31.969 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:31.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:31.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:31.970 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:32.456 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:32.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:32.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:32.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:32.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:32.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:32.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:32.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:32.458 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:33.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16598117 2020-11-16T22:38:33.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16598117 '!=' null 2020-11-16T22:38:33.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:33.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:33.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:33.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:33.241 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:33.241 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:33.241 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:33.241 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:33.241 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:33.241 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:33.242 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:33.242 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:33.242 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:33.242 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:33.242 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:33.242 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:33.242 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:33.243 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:33.243 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:33.243 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:33.243 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:33.243 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:33.243 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:33.243 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:33.243 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:33.244 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:33.244 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:33.244 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:33.244 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:33.244 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:33.244 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:33.244 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:33.245 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:33.245 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:33.245 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:33.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:33.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:33.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:33.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:33.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:33.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:33.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:33.352 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:33.353 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:33.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:33.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:33.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:33.717 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:34.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:34.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:34.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:34.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:34.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:34.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:34.199 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:34.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:34.200 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:34.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24460410 2020-11-16T22:38:34.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24460410 '!=' null 2020-11-16T22:38:34.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:34.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:34.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:34.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:34.760 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:34.761 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:34.761 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:34.761 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:34.761 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:34.761 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:34.761 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:34.762 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:34.762 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:34.762 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:34.762 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:34.762 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:34.762 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:34.762 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:34.763 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:34.763 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:34.763 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:34.763 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:34.763 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:34.763 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:34.763 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:34.763 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:34.764 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:34.764 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:34.764 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:34.764 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:34.764 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:34.764 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:34.764 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:34.765 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:34.765 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:34.768 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:34.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:34.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:34.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:34.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:34.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:34.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:34.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:34.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:35.241 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:35.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:35.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:35.242 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:35.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:35.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:35.754 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:35.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:35.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:35.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:35.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:35.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:35.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:36.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724316 2020-11-16T22:38:36.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724316 '!=' null 2020-11-16T22:38:36.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:36.322 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:36.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:36.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:36.424 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:36.425 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:36.425 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:36.425 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:36.425 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:36.425 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:36.426 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:36.426 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:36.426 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:36.426 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:36.426 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:36.426 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:36.427 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:36.427 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:36.427 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:36.427 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:36.427 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:36.427 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:36.427 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:36.428 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:36.428 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:36.428 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:36.428 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:36.428 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:36.428 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:36.428 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:36.428 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:36.429 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:36.429 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:36.429 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:36.429 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:36.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:36.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:36.536 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:36.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:36.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:36.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:36.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:36.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:36.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:36.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:36.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:36.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:36.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:37.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:37.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:37.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:37.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:37.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724611 2020-11-16T22:38:38.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724611 '!=' null 2020-11-16T22:38:38.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:38.089 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:38.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:38.089 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:38.090 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:38.090 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:38.090 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:38.091 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:38.091 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:38.091 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:38.091 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:38.091 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:38.091 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:38.091 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:38.092 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:38.092 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:38.092 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:38.092 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:38.092 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:38.092 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:38.092 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:38.093 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:38.093 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:38.093 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:38.093 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:38.093 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:38.093 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:38.093 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:38.093 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:38.094 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:38.094 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:38.094 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:38.094 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:38.094 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:38.094 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:38.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:38.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:38.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:38.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:38.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:38.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:38.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:38.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:38.106 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:38.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:38.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:38.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:38.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:38.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:38.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:38.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:38.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:38.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:38.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:38.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:38.967 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:38.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:39.896 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586710 2020-11-16T22:38:39.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586710 '!=' null 2020-11-16T22:38:39.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:39.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:39.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:39.897 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:39.998 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:39.998 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:39.999 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:39.999 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:39.999 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:39.999 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:39.999 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:39.999 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:39.999 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:40.000 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:40.000 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:40.000 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:40.000 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:40.000 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:40.000 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:40.000 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:40.001 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:40.001 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:40.001 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:40.001 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:40.001 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:40.001 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:40.001 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:40.001 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:40.002 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:40.002 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:40.002 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:40.002 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:40.002 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:40.002 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:40.002 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:40.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:40.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:40.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:40.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:40.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:40.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:40.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:40.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:40.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:40.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:40.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:40.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:40.495 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:40.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:40.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:40.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:40.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:40.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:40.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:40.976 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:40.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:40.976 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586710 2020-11-16T22:38:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586710 '!=' null 2020-11-16T22:38:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:41.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:41.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:41.564 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:41.565 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:41.565 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:41.565 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:41.565 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:41.565 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:41.565 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:41.565 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:41.566 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:41.566 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:41.566 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:41.566 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:41.566 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:41.566 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:41.566 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:41.567 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:41.567 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:41.567 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:41.567 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:41.567 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:41.567 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:41.567 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:41.567 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:41.568 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:41.568 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:41.568 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:41.568 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:41.568 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:41.568 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:41.568 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:41.569 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:41.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:41.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:41.675 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:41.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:41.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:41.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:41.676 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:41.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:41.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:42.054 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:42.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:42.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:42.055 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:42.694 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:42.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:42.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:42.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:42.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:42.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:42.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:42.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:42.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:43.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724207 2020-11-16T22:38:43.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724207 '!=' null 2020-11-16T22:38:43.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:43.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:43.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:43.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:43.938 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:43.939 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:43.939 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:43.939 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:43.939 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:43.939 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:43.939 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:43.939 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:43.940 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:43.940 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:43.940 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:43.940 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:43.940 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:43.940 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:43.940 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:43.941 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:43.941 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:43.941 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:43.941 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:43.941 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:43.941 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:43.941 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:43.942 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:43.942 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:43.942 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:43.942 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:43.942 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:43.942 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:43.942 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:43.942 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:43.943 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:43.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:44.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:44.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:44.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:44.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:44.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:44.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:44.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:44.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:44.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:44.440 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:44.904 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:44.904 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:44.905 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:45.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712990 2020-11-16T22:38:45.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712990 '!=' null 2020-11-16T22:38:45.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:45.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:45.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:45.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:45.465 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:45.466 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:45.467 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:45.468 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:45.469 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:45.470 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:45.474 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:45.576 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:45.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:45.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:45.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:45.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:45.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:45.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:45.958 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:45.958 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:45.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:45.959 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:46.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:46.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:46.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:46.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:46.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:46.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:46.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:46.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:46.439 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:46.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850572 2020-11-16T22:38:46.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850572 '!=' null 2020-11-16T22:38:46.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:46.909 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:46.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:46.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:46.999 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:46.999 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:46.999 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:46.999 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:46.999 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:47.000 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:47.000 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:47.000 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:47.000 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:47.000 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:47.000 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:47.000 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:47.001 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:47.001 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:47.001 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:47.001 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:47.001 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:47.001 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:47.001 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:47.001 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:47.002 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:47.002 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:47.002 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:47.002 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:47.002 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:47.002 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:47.002 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:47.003 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:47.003 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:47.003 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:47.003 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:47.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:47.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:47.109 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:47.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:47.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:47.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:47.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:47.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:47.110 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:47.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:47.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:47.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:47.469 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:47.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:47.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:47.970 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:47.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:47.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:47.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:47.971 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:47.971 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:47.972 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:48.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850494 2020-11-16T22:38:48.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850494 '!=' null 2020-11-16T22:38:48.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:48.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:48.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:48.808 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:48.911 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:48.911 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:48.911 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:48.912 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:48.912 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:48.912 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:48.912 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:48.912 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:48.912 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:48.912 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:48.913 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:48.913 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:48.913 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:48.913 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:48.913 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:48.913 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:48.913 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:48.913 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:48.914 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:48.914 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:48.914 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:48.914 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:48.914 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:48.914 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:48.914 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:48.915 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:48.915 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:48.915 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:48.915 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:48.915 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:48.915 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:48.920 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:49.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:49.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:49.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:49.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:49.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:49.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:49.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:49.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:49.399 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:49.399 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:49.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:49.400 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:49.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:49.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:49.883 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:49.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:49.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:49.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:49.884 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:49.884 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:49.885 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:50.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20964957 2020-11-16T22:38:50.373 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20964957 '!=' null 2020-11-16T22:38:50.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:50.374 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:50.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:50.374 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:50.467 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:50.467 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:50.467 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:50.467 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:50.467 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:50.468 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:50.469 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:50.470 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:50.471 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:50.471 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:50.471 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:50.471 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:50.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:50.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:50.577 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:50.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:50.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:50.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:50.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:50.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:50.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:50.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:50.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:50.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:51.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:51.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:51.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:51.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:51.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:51.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:51.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:51.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:51.428 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:51.939 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13103007 2020-11-16T22:38:51.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13103007 '!=' null 2020-11-16T22:38:51.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:51.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:51.940 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:51.940 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:52.039 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:52.039 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:52.039 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:52.039 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:52.039 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:52.039 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:52.040 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:52.040 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:52.040 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:52.040 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:52.040 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:52.040 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:52.040 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:52.040 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:52.041 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:52.041 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:52.041 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:52.041 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:52.041 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:52.041 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:52.041 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:52.042 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:52.042 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:52.042 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:52.042 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:52.042 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:52.042 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:52.042 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:52.043 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:52.043 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:52.043 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:52.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:52.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:52.149 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:52.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:52.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:52.150 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:52.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:52.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:52.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:52.537 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:52.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:52.538 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:53.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:53.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:53.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:53.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:53.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13103007 2020-11-16T22:38:53.480 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13103007 '!=' null 2020-11-16T22:38:53.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:53.481 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:53.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:53.481 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:53.582 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:53.583 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:53.583 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:53.583 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:53.583 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:53.583 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:53.583 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:53.583 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:53.584 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:53.584 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:53.584 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:53.584 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:53.584 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:53.584 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:53.584 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:53.584 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:53.585 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:53.585 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:53.585 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:53.585 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:53.585 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:53.585 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:53.585 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:53.586 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:53.586 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:53.586 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:53.586 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:53.586 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:53.586 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:53.586 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:53.586 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:53.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:53.693 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:53.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:53.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:53.694 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:53.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:53.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:53.695 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:54.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:54.073 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:54.074 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:54.074 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:54.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:54.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:54.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:54.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:54.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:54.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:54.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:54.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:54.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:55.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21839029 2020-11-16T22:38:55.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21839029 '!=' null 2020-11-16T22:38:55.136 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:55.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:55.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:55.137 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:55.236 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:55.236 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:55.236 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:55.237 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:55.237 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:55.237 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:55.237 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:55.237 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:55.237 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:55.237 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:55.238 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:55.238 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:55.238 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:55.238 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:55.238 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:55.238 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:55.238 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:55.239 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:55.239 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:55.239 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:55.239 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:55.239 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:55.239 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:55.239 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:55.239 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:55.240 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:55.240 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:55.240 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:55.240 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:55.240 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:55.240 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:55.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:55.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:55.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:55.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:55.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:55.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:55.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:55.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:55.348 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:55.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:55.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:55.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:55.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:56.223 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:56.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:56.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:56.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:56.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:56.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:56.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:56.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:56.225 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:56.760 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850780 2020-11-16T22:38:56.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850780 '!=' null 2020-11-16T22:38:56.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:56.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:56.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:56.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:56.858 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:56.858 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:56.858 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:56.859 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:56.859 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:56.859 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:56.859 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:56.859 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:56.859 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:56.859 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:56.859 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:56.860 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:56.860 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:56.860 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:56.860 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:56.860 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:56.860 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:56.860 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:56.861 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:56.861 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:56.861 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:56.861 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:56.861 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:56.861 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:56.861 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:56.861 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:56.862 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:56.862 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:56.862 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:56.862 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:56.862 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:56.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:56.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:56.968 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:56.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:56.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:56.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:56.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:56.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:56.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:57.566 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:57.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:57.567 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:58.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:58.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:58.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:58.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:58.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:58.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:58.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:58.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:58.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:38:58.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851178 2020-11-16T22:38:58.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851178 '!=' null 2020-11-16T22:38:58.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:38:58.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:58.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:38:58.520 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:38:58.626 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:38:58.627 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:38:58.627 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:58.627 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:58.627 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:38:58.627 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:58.627 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:38:58.628 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:38:58.628 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:58.628 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:38:58.628 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:58.628 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:58.628 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:58.628 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:58.629 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:58.629 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:38:58.629 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:38:58.629 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:38:58.629 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:38:58.629 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:38:58.629 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:38:58.629 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:58.630 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:38:58.630 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:38:58.630 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:38:58.630 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:38:58.630 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:38:58.630 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:38:58.630 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:38:58.631 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:38:58.631 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:38:58.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:38:58.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:38:58.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:38:58.737 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:38:58.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:38:58.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:38:58.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:38:58.738 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:38:58.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:38:59.111 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:38:59.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:38:59.112 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:38:59.113 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:38:59.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:38:59.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:38:59.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:38:59.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:38:59.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:38:59.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:38:59.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:38:59.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:38:59.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21840078 2020-11-16T22:39:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21840078 '!=' null 2020-11-16T22:39:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:00.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:00.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:00.149 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:00.149 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:00.150 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:00.150 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:00.150 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:00.150 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:00.150 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:00.150 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:00.150 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:00.151 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:00.151 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:00.151 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:00.151 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:00.151 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:00.151 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:00.151 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:00.152 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:00.152 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:00.152 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:00.152 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:00.152 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:00.152 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:00.152 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:00.152 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:00.153 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:00.153 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:00.153 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:00.153 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:00.153 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:00.153 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:00.153 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:00.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:00.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:00.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:00.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:00.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:00.259 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:00.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:00.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:00.260 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:00.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:00.640 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:00.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:00.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:01.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:01.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:01.210 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:01.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:01.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:01.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:01.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:01.211 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:01.212 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724715 2020-11-16T22:39:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724715 '!=' null 2020-11-16T22:39:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:01.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:01.716 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:01.816 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:01.816 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:01.816 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:01.817 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:01.817 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:01.817 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:01.817 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:01.817 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:01.817 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:01.817 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:01.818 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:01.818 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:01.818 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:01.818 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:01.818 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:01.818 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:01.818 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:01.818 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:01.819 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:01.819 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:01.819 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:01.819 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:01.819 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:01.819 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:01.819 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:01.820 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:01.820 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:01.820 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:01.820 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:01.820 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:01.820 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:01.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:01.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:01.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:01.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:01.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:01.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:01.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:01.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:01.928 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:02.332 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:02.332 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:02.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:02.333 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:02.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:02.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:02.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:02.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:02.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:02.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:02.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:02.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:02.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:03.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724715 2020-11-16T22:39:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724715 '!=' null 2020-11-16T22:39:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:03.555 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:03.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:03.555 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:03.661 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:03.661 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:03.661 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:03.661 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:03.661 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:03.662 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:03.662 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:03.662 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:03.662 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:03.662 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:03.662 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:03.663 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:03.663 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:03.663 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:03.663 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:03.663 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:03.663 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:03.663 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:03.664 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:03.664 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:03.664 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:03.664 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:03.664 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:03.664 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:03.664 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:03.665 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:03.665 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:03.665 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:03.665 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:03.665 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:03.665 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:03.669 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:03.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:03.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:03.772 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:03.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:03.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:03.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:03.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:03.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:04.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:04.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:04.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:04.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:04.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:04.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:04.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:04.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:04.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:04.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:04.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:04.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:04.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586977 2020-11-16T22:39:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586977 '!=' null 2020-11-16T22:39:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:05.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:05.138 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:05.237 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:05.238 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:05.238 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:05.238 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:05.238 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:05.238 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:05.238 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:05.238 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:05.239 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:05.239 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:05.239 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:05.239 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:05.239 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:05.239 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:05.239 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:05.240 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:05.240 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:05.240 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:05.240 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:05.240 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:05.240 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:05.240 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:05.241 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:05.241 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:05.241 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:05.241 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:05.241 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:05.241 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:05.241 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:05.241 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:05.242 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:05.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:05.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:05.348 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:05.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:05.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:05.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:05.349 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:05.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:05.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:05.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:05.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:05.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:05.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:06.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:06.173 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:06.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:06.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:06.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:06.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:06.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:06.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:06.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:07.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850575 2020-11-16T22:39:07.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850575 '!=' null 2020-11-16T22:39:07.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:07.037 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:07.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:07.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:07.139 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:07.139 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:07.139 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:07.139 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:07.139 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:07.139 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:07.140 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:07.140 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:07.140 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:07.140 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:07.140 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:07.140 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:07.140 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:07.141 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:07.141 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:07.141 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:07.141 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:07.141 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:07.141 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:07.141 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:07.141 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:07.142 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:07.142 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:07.142 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:07.142 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:07.142 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:07.142 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:07.142 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:07.143 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:07.143 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:07.143 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:07.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:07.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:07.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:07.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:07.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:07.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:07.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:07.251 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:07.252 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:07.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:07.617 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:07.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:07.618 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:08.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:08.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:08.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:08.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:08.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:08.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:08.233 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:08.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:08.233 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:08.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850498 2020-11-16T22:39:08.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850498 '!=' null 2020-11-16T22:39:08.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:08.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:08.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:08.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:08.811 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:08.811 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:08.811 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:08.812 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:08.812 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:08.812 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:08.812 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:08.812 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:08.812 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:08.812 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:08.813 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:08.813 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:08.813 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:08.813 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:08.813 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:08.813 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:08.813 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:08.813 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:08.814 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:08.814 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:08.814 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:08.814 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:08.814 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:08.814 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:08.814 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:08.815 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:08.815 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:08.815 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:08.815 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:08.815 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:08.815 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:08.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:08.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:08.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:08.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:08.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:08.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:08.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:08.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:08.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:09.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:09.297 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:09.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:09.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:09.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:09.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:09.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:09.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:09.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:09.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:09.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:09.819 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:09.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:10.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23585840 2020-11-16T22:39:10.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23585840 '!=' null 2020-11-16T22:39:10.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:10.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:10.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:10.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:10.543 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:10.543 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:10.543 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:10.543 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:10.544 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:10.544 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:10.544 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:10.544 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:10.544 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:10.544 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:10.544 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:10.545 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:10.545 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:10.545 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:10.545 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:10.545 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:10.545 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:10.545 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:10.546 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:10.546 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:10.546 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:10.546 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:10.546 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:10.546 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:10.546 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:10.547 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:10.547 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:10.547 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:10.547 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:10.547 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:10.547 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:10.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:10.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:10.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:10.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:10.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:10.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:10.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:10.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:10.655 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:11.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:11.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:11.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:11.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:11.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:11.494 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:12.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597208 2020-11-16T22:39:12.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597208 '!=' null 2020-11-16T22:39:12.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:12.029 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:12.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:12.029 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:12.127 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:12.127 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:12.128 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:12.128 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:12.128 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:12.128 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:12.128 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:12.128 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:12.128 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:12.129 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:12.129 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:12.129 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:12.129 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:12.129 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:12.129 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:12.129 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:12.129 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:12.130 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:12.130 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:12.130 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:12.130 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:12.130 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:12.130 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:12.130 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:12.131 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:12.131 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:12.131 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:12.131 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:12.131 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:12.131 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:12.131 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:12.135 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:12.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:12.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:12.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:12.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:12.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:12.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:12.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:12.238 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:12.595 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:12.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:12.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:12.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:13.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:13.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:13.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:13.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:13.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:13.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:13.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:13.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:13.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:13.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597208 2020-11-16T22:39:13.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597208 '!=' null 2020-11-16T22:39:13.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:13.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:13.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:13.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:13.738 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:13.738 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:13.738 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:13.738 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:13.738 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:13.739 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:13.739 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:13.739 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:13.739 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:13.739 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:13.739 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:13.739 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:13.740 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:13.740 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:13.740 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:13.740 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:13.740 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:13.740 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:13.740 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:13.740 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:13.741 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:13.741 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:13.741 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:13.741 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:13.741 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:13.741 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:13.741 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:13.742 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:13.742 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:13.742 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:13.742 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:13.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:13.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:13.848 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:13.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:13.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:13.849 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:13.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:13.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:13.850 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:14.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:14.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:14.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:14.271 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:14.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:14.740 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:15.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22711968 2020-11-16T22:39:15.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22711968 '!=' null 2020-11-16T22:39:15.236 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:15.237 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:15.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:15.237 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:15.331 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:15.331 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:15.332 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:15.332 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:15.332 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:15.332 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:15.332 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:15.332 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:15.332 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:15.333 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:15.333 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:15.333 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:15.333 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:15.333 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:15.333 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:15.333 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:15.334 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:15.334 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:15.334 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:15.334 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:15.334 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:15.334 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:15.334 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:15.335 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:15.335 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:15.335 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:15.335 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:15.335 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:15.335 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:15.335 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:15.335 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:15.339 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:15.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:15.441 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:15.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:15.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:15.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:15.442 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:15.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:15.443 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:15.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:15.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:15.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:15.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:16.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:16.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:16.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:16.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:16.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:16.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:16.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:16.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:16.355 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:16.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850141 2020-11-16T22:39:16.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850141 '!=' null 2020-11-16T22:39:16.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:16.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:16.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:16.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:16.927 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:16.927 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:16.928 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:16.928 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:16.928 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:16.928 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:16.928 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:16.928 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:16.928 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:16.929 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:16.929 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:16.929 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:16.929 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:16.929 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:16.929 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:16.929 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:16.930 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:16.930 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:16.930 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:16.930 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:16.930 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:16.930 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:16.930 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:16.930 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:16.931 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:16.931 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:16.931 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:16.931 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:16.931 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:16.931 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:16.931 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:16.935 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:17.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:17.036 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:17.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:17.037 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:17.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:17.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:17.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:17.038 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:17.389 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:17.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:17.390 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:17.391 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:17.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:17.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:17.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:17.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:17.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:17.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:17.874 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:17.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:17.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850269 2020-11-16T22:39:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850269 '!=' null 2020-11-16T22:39:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:18.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:18.677 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:18.768 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:18.769 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:18.769 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:18.769 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:18.769 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:18.769 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:18.769 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:18.769 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:18.770 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:18.770 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:18.770 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:18.770 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:18.770 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:18.770 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:18.770 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:18.771 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:18.771 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:18.771 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:18.771 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:18.771 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:18.771 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:18.771 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:18.772 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:18.772 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:18.772 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:18.772 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:18.772 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:18.772 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:18.772 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:18.772 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:18.773 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:18.777 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:18.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:18.878 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:18.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:18.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:18.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:18.879 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:18.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:18.880 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:19.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:19.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:19.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:19.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:19.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:19.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:19.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:19.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:19.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:19.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:19.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:19.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:19.825 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:20.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22711900 2020-11-16T22:39:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22711900 '!=' null 2020-11-16T22:39:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:20.338 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:20.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:20.338 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:20.437 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:20.437 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:20.437 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:20.437 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:20.437 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:20.437 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:20.438 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:20.438 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:20.438 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:20.438 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:20.438 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:20.438 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:20.439 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:20.439 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:20.439 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:20.439 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:20.439 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:20.439 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:20.439 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:20.439 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:20.440 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:20.440 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:20.440 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:20.440 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:20.440 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:20.440 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:20.440 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:20.440 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:20.441 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:20.441 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:20.441 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:20.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:20.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:20.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:20.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:20.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:20.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:20.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:20.549 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:20.917 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:20.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:20.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:21.722 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:21.723 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:21.724 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:21.724 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13976556 2020-11-16T22:39:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13976556 '!=' null 2020-11-16T22:39:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:22.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:22.272 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:22.376 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:22.377 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:22.377 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:22.377 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:22.377 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:22.377 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:22.377 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:22.377 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:22.378 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:22.378 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:22.378 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:22.378 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:22.378 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:22.378 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:22.378 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:22.379 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:22.379 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:22.379 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:22.379 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:22.379 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:22.379 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:22.379 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:22.379 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:22.380 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:22.380 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:22.380 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:22.380 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:22.380 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:22.380 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:22.380 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:22.381 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:22.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:22.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:22.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:22.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:22.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:22.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:22.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:22.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:22.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:22.869 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:22.870 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:22.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:22.871 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:23.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:23.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:23.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:23.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:23.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:23.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:23.334 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:23.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:23.335 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:23.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712027 2020-11-16T22:39:23.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712027 '!=' null 2020-11-16T22:39:23.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:23.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:23.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:23.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:23.916 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:23.917 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:23.917 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:23.917 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:23.917 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:23.917 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:23.917 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:23.918 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:23.918 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:23.918 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:23.918 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:23.918 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:23.918 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:23.918 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:23.919 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:23.919 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:23.919 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:23.919 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:23.919 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:23.919 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:23.919 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:23.920 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:23.920 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:23.920 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:23.920 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:23.920 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:23.920 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:23.920 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:23.921 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:23.921 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:23.921 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:23.925 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:24.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:24.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:24.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:24.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:24.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:24.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:24.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:24.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:24.420 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:24.420 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:24.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:24.421 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:24.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:24.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:24.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:24.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:24.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:24.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:24.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:24.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:24.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:25.401 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712027 2020-11-16T22:39:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712027 '!=' null 2020-11-16T22:39:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:25.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:25.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:25.402 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:25.502 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:25.503 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:25.503 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:25.503 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:25.503 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:25.503 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:25.503 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:25.504 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:25.504 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:25.504 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:25.504 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:25.504 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:25.504 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:25.504 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:25.505 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:25.505 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:25.505 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:25.505 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:25.505 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:25.505 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:25.505 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:25.505 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:25.506 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:25.506 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:25.506 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:25.506 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:25.506 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:25.506 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:25.506 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:25.507 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:25.507 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:25.511 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:25.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:25.614 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:25.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:25.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:25.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:25.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:25.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:25.616 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:26.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:26.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:26.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:26.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:26.490 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:26.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:26.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:26.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:26.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:26.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:26.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:26.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:26.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:26.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597452 2020-11-16T22:39:26.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597452 '!=' null 2020-11-16T22:39:26.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:26.953 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:26.953 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:26.954 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:27.053 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:27.053 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:27.054 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:27.054 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:27.054 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:27.054 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:27.054 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:27.054 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:27.055 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:27.055 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:27.055 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:27.055 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:27.055 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:27.055 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:27.055 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:27.056 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:27.056 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:27.056 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:27.056 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:27.056 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:27.056 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:27.056 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:27.057 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:27.057 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:27.057 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:27.057 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:27.057 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:27.057 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:27.057 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:27.058 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:27.058 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:27.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:27.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:27.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:27.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:27.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:27.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:27.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:27.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:27.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:27.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:27.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:27.560 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:28.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:28.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:28.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:28.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:28.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:28.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:28.198 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:28.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:28.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:28.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597672 2020-11-16T22:39:28.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597672 '!=' null 2020-11-16T22:39:28.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:28.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:28.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:28.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:28.761 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:28.761 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:28.762 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:28.762 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:28.762 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:28.762 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:28.762 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:28.762 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:28.762 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:28.763 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:28.763 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:28.763 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:28.763 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:28.763 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:28.763 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:28.763 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:28.764 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:28.764 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:28.764 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:28.764 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:28.764 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:28.764 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:28.764 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:28.764 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:28.765 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:28.765 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:28.765 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:28.765 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:28.765 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:28.765 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:28.765 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:28.770 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:28.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:28.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:28.872 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:28.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:28.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:28.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:28.873 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:28.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:29.386 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:29.386 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:29.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:29.387 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:29.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:29.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:29.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:29.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:29.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:29.888 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:29.889 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:29.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:29.889 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:30.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586261 2020-11-16T22:39:30.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586261 '!=' null 2020-11-16T22:39:30.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:30.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:30.380 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:30.381 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:30.473 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:30.473 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:30.473 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:30.474 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:30.474 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:30.474 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:30.474 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:30.474 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:30.474 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:30.474 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:30.475 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:30.475 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:30.475 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:30.475 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:30.475 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:30.475 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:30.475 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:30.476 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:30.476 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:30.476 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:30.476 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:30.476 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:30.476 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:30.476 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:30.476 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:30.477 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:30.477 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:30.477 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:30.477 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:30.477 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:30.477 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:30.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:30.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:30.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:30.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:30.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:30.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:30.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:30.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:30.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:30.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:30.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:30.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:30.966 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:31.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:31.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:31.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724022 2020-11-16T22:39:31.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724022 '!=' null 2020-11-16T22:39:31.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:31.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:31.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:31.996 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:32.104 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:32.104 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:32.105 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:32.105 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:32.105 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:32.105 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:32.105 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:32.105 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:32.105 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:32.106 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:32.106 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:32.106 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:32.106 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:32.106 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:32.106 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:32.106 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:32.107 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:32.107 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:32.107 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:32.107 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:32.107 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:32.107 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:32.107 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:32.107 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:32.108 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:32.108 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:32.108 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:32.108 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:32.108 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:32.108 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:32.108 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:32.113 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:32.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:32.214 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:32.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:32.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:32.215 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:32.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:32.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:32.216 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:32.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:32.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:32.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:32.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:33.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:33.170 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:33.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:33.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:33.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:33.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:33.171 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:33.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:33.172 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:33.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724022 2020-11-16T22:39:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724022 '!=' null 2020-11-16T22:39:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:33.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:33.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:33.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:33.971 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:33.971 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:33.971 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:33.971 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:33.971 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:33.972 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:33.972 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:33.972 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:33.972 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:33.972 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:33.972 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:33.972 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:33.973 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:33.973 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:33.973 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:33.973 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:33.973 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:33.973 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:33.973 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:33.974 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:33.974 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:33.974 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:33.974 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:33.974 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:33.974 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:33.974 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:33.975 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:33.975 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:33.975 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:33.975 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:33.975 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:33.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:34.081 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:34.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:34.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:34.082 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:34.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:34.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:34.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:34.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:34.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:34.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:34.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:34.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:34.948 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:34.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:34.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:34.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:34.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:34.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:34.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:34.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:34.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23585785 2020-11-16T22:39:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23585785 '!=' null 2020-11-16T22:39:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:35.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:35.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:35.546 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:35.546 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:35.546 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:35.547 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:35.547 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:35.547 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:35.547 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:35.547 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:35.547 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:35.547 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:35.547 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:35.548 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:35.548 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:35.548 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:35.548 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:35.548 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:35.548 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:35.548 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:35.549 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:35.549 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:35.549 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:35.549 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:35.549 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:35.549 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:35.549 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:35.549 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:35.550 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:35.550 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:35.550 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:35.550 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:35.550 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:35.553 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:35.655 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:35.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:35.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:35.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:35.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:35.656 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:35.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:35.657 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:36.030 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:36.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:36.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:36.031 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:36.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:36.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:36.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:36.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:36.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:36.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:36.824 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:36.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:36.824 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:37.297 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850185 2020-11-16T22:39:37.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850185 '!=' null 2020-11-16T22:39:37.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:37.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:37.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:37.298 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:37.399 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:37.399 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:37.399 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:37.399 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:37.399 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:37.400 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:37.400 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:37.400 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:37.400 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:37.400 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:37.400 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:37.400 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:37.401 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:37.401 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:37.401 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:37.401 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:37.401 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:37.401 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:37.401 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:37.402 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:37.402 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:37.402 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:37.402 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:37.402 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:37.402 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:37.402 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:37.402 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:37.403 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:37.403 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:37.403 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:37.403 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:37.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:37.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:37.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:37.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:37.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:37.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:37.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:37.511 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:38.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:38.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:38.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:38.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:38.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:38.818 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:38.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:38.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:38.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:38.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:38.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:38.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:38.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:39.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850279 2020-11-16T22:39:39.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850279 '!=' null 2020-11-16T22:39:39.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:39.504 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:39.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:39.505 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:39.599 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:39.600 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:39.600 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:39.600 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:39.600 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:39.600 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:39.600 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:39.600 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:39.601 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:39.601 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:39.601 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:39.601 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:39.601 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:39.601 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:39.601 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:39.601 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:39.602 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:39.602 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:39.602 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:39.602 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:39.602 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:39.602 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:39.602 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:39.603 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:39.603 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:39.603 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:39.603 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:39.603 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:39.603 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:39.603 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:39.604 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:39.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:39.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:39.710 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:40.082 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:40.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:40.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:40.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:40.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:40.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:40.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:40.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:40.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:40.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:40.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:40.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:40.666 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:41.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712332 2020-11-16T22:39:41.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712332 '!=' null 2020-11-16T22:39:41.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:41.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:41.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:41.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:41.394 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:41.394 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:41.394 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:41.395 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:41.395 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:41.395 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:41.395 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:41.395 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:41.395 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:41.395 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:41.396 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:41.396 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:41.396 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:41.396 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:41.396 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:41.396 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:41.396 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:41.397 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:41.397 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:41.397 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:41.397 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:41.397 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:41.397 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:41.397 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:41.398 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:41.398 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:41.398 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:41.398 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:41.398 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:41.398 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:41.398 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:41.403 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:41.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:41.505 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:41.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:41.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:41.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:41.506 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:41.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:41.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:41.891 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:41.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:41.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:41.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:42.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:42.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:42.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:42.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:42.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:42.367 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:42.368 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:42.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:42.368 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:43.070 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15723969 2020-11-16T22:39:43.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15723969 '!=' null 2020-11-16T22:39:43.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:43.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:43.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:43.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:43.172 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:43.172 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:43.172 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:43.172 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:43.172 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:43.173 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:43.173 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:43.173 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:43.173 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:43.173 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:43.173 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:43.173 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:43.174 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:43.174 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:43.174 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:43.174 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:43.174 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:43.174 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:43.174 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:43.175 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:43.175 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:43.175 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:43.175 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:43.175 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:43.175 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:43.175 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:43.175 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:43.176 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:43.176 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:43.176 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:43.176 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:43.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:43.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:43.284 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:43.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:43.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:43.285 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:43.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:43.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:43.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:43.668 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:43.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:43.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:43.669 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:44.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:44.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:44.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:44.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:44.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:44.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:44.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:44.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:44.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:44.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20965540 2020-11-16T22:39:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20965540 '!=' null 2020-11-16T22:39:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:44.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:44.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:44.608 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:44.708 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:44.708 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:44.709 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:44.709 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:44.709 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:44.709 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:44.709 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:44.709 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:44.709 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:44.710 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:44.710 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:44.710 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:44.710 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:44.710 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:44.710 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:44.710 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:44.711 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:44.711 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:44.711 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:44.711 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:44.711 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:44.711 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:44.711 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:44.712 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:44.712 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:44.712 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:44.712 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:44.712 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:44.712 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:44.712 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:44.712 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:44.717 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:44.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:44.819 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:44.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:44.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:44.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:44.820 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:44.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:44.821 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:45.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:45.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:45.186 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:45.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:46.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:46.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:46.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:46.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:46.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:46.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:46.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:46.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:46.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:46.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13103636 2020-11-16T22:39:46.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13103636 '!=' null 2020-11-16T22:39:46.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:46.507 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:46.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:46.507 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:46.601 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:46.602 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:46.602 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:46.602 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:46.602 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:46.602 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:46.603 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:46.603 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:46.603 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:46.603 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:46.603 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:46.603 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:46.603 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:46.604 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:46.604 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:46.604 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:46.604 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:46.604 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:46.604 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:46.604 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:46.605 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:46.605 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:46.605 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:46.605 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:46.605 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:46.605 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:46.605 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:46.606 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:46.606 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:46.606 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:46.606 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:46.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:46.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:46.712 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:46.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:46.712 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:46.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:46.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:46.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:46.713 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:47.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:47.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:47.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:47.087 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:47.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:47.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:47.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:47.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:47.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:47.551 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:47.552 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:47.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:47.552 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:48.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13103960 2020-11-16T22:39:48.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13103960 '!=' null 2020-11-16T22:39:48.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:48.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:48.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:48.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:48.094 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:48.095 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:48.095 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:48.095 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:48.095 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:48.095 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:48.095 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:48.096 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:48.096 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:48.096 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:48.096 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:48.096 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:48.096 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:48.096 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:48.097 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:48.097 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:48.097 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:48.097 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:48.097 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:48.097 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:48.097 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:48.097 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:48.098 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:48.098 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:48.098 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:48.098 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:48.098 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:48.098 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:48.098 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:48.099 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:48.099 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:48.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:48.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:48.204 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:48.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:48.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:48.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:48.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:48.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:48.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:48.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:48.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:48.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:48.587 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:49.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:49.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:49.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:49.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:49.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:49.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:49.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:49.053 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:49.054 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13103960 2020-11-16T22:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13103960 '!=' null 2020-11-16T22:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:49.681 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:49.681 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:49.681 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:49.682 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:49.682 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:49.682 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:49.682 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:49.682 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:49.682 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:49.683 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:49.683 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:49.683 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:49.683 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:49.683 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:49.683 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:49.683 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:49.683 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:49.684 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:49.684 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:49.684 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:49.684 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:49.684 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:49.684 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:49.684 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:49.685 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:49.685 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:49.685 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:49.685 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:49.685 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:49.685 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:49.685 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:49.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:49.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:49.790 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:49.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:49.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:49.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:49.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:49.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:49.792 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:50.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:50.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:50.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:50.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:50.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:50.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:50.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:50.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:50.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:50.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:50.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:50.658 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:50.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:51.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21840027 2020-11-16T22:39:51.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21840027 '!=' null 2020-11-16T22:39:51.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:51.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:51.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:51.153 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:51.254 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:51.254 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:51.254 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:51.254 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:51.254 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:51.255 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:51.255 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:51.255 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:51.255 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:51.255 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:51.255 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:51.255 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:51.256 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:51.256 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:51.256 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:51.256 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:51.256 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:51.256 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:51.256 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:51.257 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:51.257 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:51.257 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:51.257 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:51.257 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:51.257 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:51.257 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:51.257 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:51.258 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:51.258 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:51.258 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:51.258 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:51.262 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:51.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:51.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:51.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:51.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:51.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:51.365 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:51.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:51.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:51.748 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:51.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:51.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:51.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:52.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:52.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:52.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:52.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:52.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:52.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:52.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:52.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:52.447 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13977585 2020-11-16T22:39:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13977585 '!=' null 2020-11-16T22:39:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:52.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:52.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:53.082 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:53.082 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:53.082 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:53.082 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:53.083 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:53.084 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:53.084 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:53.084 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:53.084 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:53.084 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:53.084 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:53.084 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:53.084 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:53.085 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:53.085 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:53.085 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:53.085 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:53.085 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:53.085 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:53.085 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:53.086 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:53.086 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:53.086 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:53.086 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:53.086 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:53.091 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:53.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:53.193 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:53.194 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:53.194 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:53.194 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:53.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:53.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:53.195 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:53.682 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:53.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:53.683 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:53.684 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:54.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:54.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:54.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:54.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:54.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:54.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:54.311 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:54.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:54.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:54.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20966492 2020-11-16T22:39:54.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20966492 '!=' null 2020-11-16T22:39:54.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:55.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:55.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:55.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:55.102 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:55.102 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:55.102 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:55.102 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:55.102 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:55.103 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:55.104 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:55.105 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:55.106 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:55.106 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:55.106 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:55.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:55.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:55.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:55.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:55.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:55.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:55.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:55.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:55.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:55.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:55.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:55.597 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:56.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:56.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:56.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:56.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:56.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:56.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:56.240 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:56.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:56.241 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:56.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725012 2020-11-16T22:39:56.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725012 '!=' null 2020-11-16T22:39:56.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:56.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:56.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:56.710 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:56.811 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:56.811 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:56.811 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:56.812 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:56.812 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [ 2020-11-16T22:39:56.812 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:56.812 INFO:tasks.workunit.client.0.smithi189.stdout: "priority": 190, 2020-11-16T22:39:56.812 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2020-11-16T22:39:56.812 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:56.812 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:56.813 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:56.813 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:56.813 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:56.813 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:56.813 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:56.813 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2020-11-16T22:39:56.813 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:56.814 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:56.814 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "1.0", 2020-11-16T22:39:56.814 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:56.814 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:56.814 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:56.814 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:56.814 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:56.814 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:56.815 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:56.815 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:56.815 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:56.815 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:56.815 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:56.815 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:56.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:56.921 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:56.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:56.922 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:56.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:56.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:56.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:56.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:56.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:57.330 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:57.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:57.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:57.331 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:39:58.543 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:39:58.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:39:58.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:39:58.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:39:58.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:39:58.544 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:39:58.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:39:58.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:39:58.545 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:39:59.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18870028 2020-11-16T22:39:59.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18870028 '!=' null 2020-11-16T22:39:59.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:39:59.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:59.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:39:59.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:39:59.199 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:39:59.200 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:39:59.200 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:59.200 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:59.200 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:59.200 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:39:59.200 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:39:59.200 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:39:59.201 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:39:59.201 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:39:59.201 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:59.201 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:39:59.201 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:39:59.201 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:39:59.201 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:39:59.202 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:39:59.202 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:39:59.202 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:39:59.202 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:39:59.202 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:39:59.208 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:39:59.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:39:59.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:39:59.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:39:59.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:39:59.311 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:39:59.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:39:59.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:39:59.312 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:39:59.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:39:59.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:39:59.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:39:59.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:00.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:40:00.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:40:00.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:00.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:00.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:00.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:00.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:00.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:00.372 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:01.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19656205 2020-11-16T22:40:01.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19656205 '!=' null 2020-11-16T22:40:01.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:01.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:01.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:01.091 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:01.202 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:01.203 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:01.203 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:01.203 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:01.203 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:01.204 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:01.204 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:01.204 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:01.204 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:01.204 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:01.204 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:01.204 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:01.204 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:01.205 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:01.205 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:01.205 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:01.205 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:01.205 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:01.205 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:01.205 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:01.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:01.313 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:01.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:01.314 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:01.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:01.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:01.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:01.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:01.316 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:01.700 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:40:01.701 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:01.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:01.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:02.185 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:40:02.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:40:02.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:02.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:02.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:02.186 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:02.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:02.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:02.187 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:02.635 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=9172850 2020-11-16T22:40:02.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 9172850 '!=' null 2020-11-16T22:40:02.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:02.636 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:02.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:02.636 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:02.729 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:02.729 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:02.729 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:02.729 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:02.729 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:02.729 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:02.730 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:02.730 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:02.730 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:02.730 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:02.730 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:02.730 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:02.730 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:02.731 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:02.731 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:02.731 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:02.731 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:02.731 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:02.731 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:02.731 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:02.736 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:02.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:02.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:02.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:02.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:02.839 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:02.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:02.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:02.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:03.227 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:40:03.227 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:03.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:03.228 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:03.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2020-11-16T22:40:03.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2020-11-16T22:40:03.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:03.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:03.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:03.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:03.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:03.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:03.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:04.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586491 2020-11-16T22:40:04.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586491 '!=' null 2020-11-16T22:40:04.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:04.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:04.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:04.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:04.503 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:04.503 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:04.503 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:04.504 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:04.504 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:04.504 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:04.504 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:04.504 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:04.504 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:04.505 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:04.505 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:04.505 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:04.505 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:04.505 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:04.505 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:04.505 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:04.506 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:04.506 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:04.506 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:04.506 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:04.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:04.613 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:04.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:04.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:04.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:04.614 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:04.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:04.615 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:05.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:05.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:05.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:05.028 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:05.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:05.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 1 2020-11-16T22:40:05.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-11-16T22:40:05.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=2 2020-11-16T22:40:05.487 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:05.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:05.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:05.588 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:05.589 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:05.589 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:05.590 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:05.590 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:05.590 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:05.590 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:05.590 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:05.590 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:05.590 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:05.591 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:05.591 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:05.591 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:05.591 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:05.591 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:05.591 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:05.591 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:05.591 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:05.592 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:05.592 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:05.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:05.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:05.699 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:05.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:05.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:05.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:05.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:05.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:05.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:06.083 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:06.083 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:06.084 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:06.084 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:06.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:06.583 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:06.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:06.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:06.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:06.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:06.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:06.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:06.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:07.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22475376 2020-11-16T22:40:07.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22475376 '!=' null 2020-11-16T22:40:07.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:07.354 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:07.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:07.354 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:07.456 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:07.456 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:07.456 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:07.456 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:07.456 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:07.456 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:07.457 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:07.457 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:07.457 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:07.457 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:07.457 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:07.457 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:07.457 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:07.458 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:07.458 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:07.458 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:07.458 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:07.458 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:07.458 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:07.458 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:07.466 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:07.568 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:07.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:07.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:07.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:07.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:07.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:07.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:08.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:08.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:08.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:08.033 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:08.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:08.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:08.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:08.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:08.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:08.520 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:08.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:08.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:08.521 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:09.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18868754 2020-11-16T22:40:09.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18868754 '!=' null 2020-11-16T22:40:09.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:09.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:09.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:09.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:09.107 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:09.107 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:09.107 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:09.107 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:09.108 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:09.109 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:09.110 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:09.114 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:09.216 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:09.217 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:09.217 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:09.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:09.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:09.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:09.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:09.218 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:09.605 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:09.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:09.606 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:09.607 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:10.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:10.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:10.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:10.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:10.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:10.275 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:10.276 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:10.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:10.276 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:10.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23896271 2020-11-16T22:40:10.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23896271 '!=' null 2020-11-16T22:40:10.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:10.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:10.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:10.763 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:10.862 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:10.862 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:10.862 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:10.862 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:10.862 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:10.863 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:10.863 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:10.863 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:10.863 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:10.863 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:10.863 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:10.863 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:10.864 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:10.864 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:10.864 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:10.864 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:10.864 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:10.864 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:10.869 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:10.869 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:10.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:10.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:10.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:10.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:10.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:10.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:10.973 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:10.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:10.974 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:11.329 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:11.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:11.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:11.330 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:11.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:11.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:11.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:11.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:11.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:11.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:11.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:11.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:11.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:12.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597662 2020-11-16T22:40:12.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597662 '!=' null 2020-11-16T22:40:12.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:12.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:12.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:12.320 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:12.422 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:12.422 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:12.423 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:12.423 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:12.423 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:12.423 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:12.423 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:12.423 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:12.423 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:12.424 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:12.424 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:12.424 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:12.424 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:12.424 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:12.424 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:12.424 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:12.424 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:12.425 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:12.425 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:12.425 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:12.431 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:12.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:12.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:12.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:12.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:12.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:12.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:12.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:12.535 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:12.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:12.890 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:12.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:12.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:13.538 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:13.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:13.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:13.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:13.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:13.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:13.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:13.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:13.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:14.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24459717 2020-11-16T22:40:27.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24459717 '!=' null 2020-11-16T22:40:27.959 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:27.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:27.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:27.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:27.972 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:27.973 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:27.973 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.973 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.973 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.973 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:27.973 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:27.973 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:27.974 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:27.974 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:27.974 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.974 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:27.974 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:27.974 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:27.974 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.975 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.975 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.975 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:27.975 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.975 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:27.975 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:27.975 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:27.975 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.976 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.976 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.976 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:27.976 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:27.976 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:27.976 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:27.976 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:27.977 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.977 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:27.977 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:27.977 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:27.977 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.977 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.977 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.977 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:27.978 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.978 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:27.978 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:27.978 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:27.978 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.978 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.978 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.979 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:27.979 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:27.979 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:27.979 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:27.979 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:27.979 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.979 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:27.980 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:27.980 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:27.980 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.980 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.980 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.980 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:27.980 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.981 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:27.981 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:27.981 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:27.981 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.981 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.981 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.981 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:27.981 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:27.982 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:27.982 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:27.982 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:27.982 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.982 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:27.982 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:27.982 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:27.983 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.983 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.983 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.983 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:27.983 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.983 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:27.983 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:27.983 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:27.984 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.984 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.984 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.984 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:27.984 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:27.984 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:27.984 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:27.985 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:27.985 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.985 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:27.985 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:27.985 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:27.985 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.985 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.985 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.986 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:27.986 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.986 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:27.986 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:27.986 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:27.986 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.986 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.987 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.987 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:27.987 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:27.987 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:27.987 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:27.987 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:27.987 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.987 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:27.988 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:27.988 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:27.988 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.988 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.988 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.988 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:27.988 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.989 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:27.989 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:27.989 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:27.989 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.989 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.989 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.989 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:27.990 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:27.990 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:27.990 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:27.990 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:27.990 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.990 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:27.990 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:27.990 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:27.991 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.991 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.991 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.991 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:27.991 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.991 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:27.991 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:27.991 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:27.992 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.992 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.992 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.992 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:27.992 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:27.992 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:27.992 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:27.993 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:27.993 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.993 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:27.993 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:27.993 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:27.993 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.993 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.993 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.994 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:27.994 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.994 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:27.994 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:27.994 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:27.994 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.994 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.995 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.995 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:27.995 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:27.995 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:27.995 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:27.995 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:27.995 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.995 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:27.996 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:27.996 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:27.996 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:27.996 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:27.996 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:27.996 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:27.996 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:27.996 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24459717 2020-11-16T22:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24459717 '!=' null 2020-11-16T22:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:28.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:28.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:28.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:28.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:28.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:28.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:28.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:28.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:28.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:28.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:28.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:28.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:28.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:28.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:28.003 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:28.004 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:28.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850743 2020-11-16T22:40:28.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850743 '!=' null 2020-11-16T22:40:28.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:28.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.004 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:28.004 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:28.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:28.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:28.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:28.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:28.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:28.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:28.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:28.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:28.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:28.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:28.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:28.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:28.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:28.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:28.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:28.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:28.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:28.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:28.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:28.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:28.007 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:28.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850832 2020-11-16T22:40:28.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850832 '!=' null 2020-11-16T22:40:28.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:28.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:28.008 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:28.008 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:28.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:28.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:28.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:28.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:28.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:28.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:28.009 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:28.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:28.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:28.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:28.010 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:28.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:28.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:28.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:28.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:28.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:28.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:28.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:28.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:28.011 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:28.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22713138 2020-11-16T22:40:28.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22713138 '!=' null 2020-11-16T22:40:28.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:28.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:28.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:28.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:28.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:28.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:28.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:28.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:28.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:28.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:28.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:28.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:28.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:28.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:28.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:28.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:28.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:28.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:28.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:28.014 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:28.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:28.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:28.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:28.015 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:28.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724488 2020-11-16T22:40:28.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724488 '!=' null 2020-11-16T22:40:28.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:28.015 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:28.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:28.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:28.016 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:28.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:28.016 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:28.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:28.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:28.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:28.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:28.017 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:28.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:28.017 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:28.018 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:28.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:28.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:28.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:28.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:28.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:28.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:28.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:28.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:28.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:28.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712884 2020-11-16T22:40:28.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712884 '!=' null 2020-11-16T22:40:28.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:28.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.019 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:28.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:28.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:28.020 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:28.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:28.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:28.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:28.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:28.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:28.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:28.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:28.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:28.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:28.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:28.021 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:28.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:28.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:28.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:28.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:28.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:28.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:28.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:28.022 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:28.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850599 2020-11-16T22:40:28.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850599 '!=' null 2020-11-16T22:40:28.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:28.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:28.023 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:28.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:28.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:28.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:28.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:28.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:28.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:28.024 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:28.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:28.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:28.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:28.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:28.025 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:28.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:28.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:28.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:28.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:28.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:28.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:28.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:28.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:28.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:28.026 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850922 2020-11-16T22:40:28.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850922 '!=' null 2020-11-16T22:40:28.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:28.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:28.027 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:28.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:28.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:28.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:28.057 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:28.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:28.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:28.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:28.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:28.059 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:28.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:28.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:28.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:28.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:28.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:28.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:28.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:28.914 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:28.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:28.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:28.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:28.915 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:28.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850922 2020-11-16T22:40:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850922 '!=' null 2020-11-16T22:40:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:29.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:29.451 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:29.550 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:29.550 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:29.551 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:29.551 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:29.551 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:29.551 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:29.551 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:29.551 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:29.551 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:29.552 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:29.552 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:29.552 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:29.552 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:29.552 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:29.552 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:29.552 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:29.553 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:29.553 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:29.553 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:29.553 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:29.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:29.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:29.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:29.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:29.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:29.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:29.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:29.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:29.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:30.047 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:30.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:30.048 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:30.049 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:30.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:30.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:30.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:30.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:30.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:30.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:30.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:30.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:30.517 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:31.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22713367 2020-11-16T22:40:31.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22713367 '!=' null 2020-11-16T22:40:31.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:31.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:31.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:31.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:31.467 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:31.467 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:31.468 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:31.468 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:31.468 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:31.468 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:31.468 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:31.468 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:31.468 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:31.469 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:31.469 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:31.469 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:31.469 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:31.469 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:31.469 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:31.469 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:31.469 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:31.470 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:31.470 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:31.470 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:31.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:31.577 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:31.578 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:31.578 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:31.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:31.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:31.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:31.579 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:31.580 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:31.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:31.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:31.979 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:31.980 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:32.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:32.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:32.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:32.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:32.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:32.443 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:32.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:32.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:32.444 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:32.974 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850753 2020-11-16T22:40:32.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850753 '!=' null 2020-11-16T22:40:32.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:32.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:32.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:32.975 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:33.077 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:33.077 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:33.077 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:33.077 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:33.078 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:33.078 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:33.078 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:33.078 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:33.078 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:33.078 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:33.078 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:33.079 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:33.079 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:33.079 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:33.079 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:33.079 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:33.079 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:33.079 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:33.079 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:33.080 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:33.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:33.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:33.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:33.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:33.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:33.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:33.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:33.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:33.190 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:33.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:33.574 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:33.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:33.575 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:34.425 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:34.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:34.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:34.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:34.426 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:34.427 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:35.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586335 2020-11-16T22:40:35.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586335 '!=' null 2020-11-16T22:40:35.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:35.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:35.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:35.273 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:35.374 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:35.374 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:35.374 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:35.375 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:35.375 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:35.375 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:35.375 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:35.375 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:35.375 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:35.375 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:35.376 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:35.376 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:35.376 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:35.376 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:35.376 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:35.376 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:35.376 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:35.376 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:35.377 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:35.377 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:35.485 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:35.486 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:35.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:35.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:35.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:35.487 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:35.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:35.488 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:35.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:35.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:35.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:35.948 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:36.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:36.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:36.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:36.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:36.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:36.445 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:36.446 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:36.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:36.446 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:37.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597982 2020-11-16T22:40:37.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597982 '!=' null 2020-11-16T22:40:37.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:37.483 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:37.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:37.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:37.586 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:37.586 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:37.586 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:37.586 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:37.586 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:37.586 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:37.587 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:37.587 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:37.587 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:37.587 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:37.587 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:37.587 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:37.587 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:37.588 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:37.588 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:37.588 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:37.588 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:37.588 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:37.588 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:37.588 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:37.596 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:37.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:37.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:37.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:37.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:37.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:37.699 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:37.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:37.700 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:38.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:38.085 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:38.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:38.086 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:38.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:38.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:38.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:38.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:38.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:38.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:38.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:38.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:38.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16598276 2020-11-16T22:40:39.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16598276 '!=' null 2020-11-16T22:40:39.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:39.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:39.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:39.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:39.499 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:39.499 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:39.499 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:39.499 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:39.500 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:39.500 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:39.500 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:39.500 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:39.500 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:39.500 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:39.500 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:39.501 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:39.501 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:39.501 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:39.501 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:39.501 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:39.501 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:39.501 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:39.502 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:39.502 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:39.506 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:39.608 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:39.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:39.609 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:39.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:39.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:39.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:39.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:39.610 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:40.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:40.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:40.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:40.672 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:40.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:40.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:40.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:40.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:40.673 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:40.674 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:40.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:40.674 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:41.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19219019 2020-11-16T22:40:41.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19219019 '!=' null 2020-11-16T22:40:41.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:41.219 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:41.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:41.220 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:41.319 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:41.319 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:41.319 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:41.320 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:41.320 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:41.320 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:41.320 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:41.320 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:41.320 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:41.320 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:41.321 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:41.321 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:41.321 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:41.321 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:41.321 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:41.321 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:41.321 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:41.322 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:41.322 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:41.322 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:41.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:41.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:41.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:41.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:41.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:41.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:41.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:41.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:41.432 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:41.808 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:41.809 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:41.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:41.810 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:42.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:42.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:42.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:42.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:42.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:42.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:42.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:42.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:42.435 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:42.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=11356603 2020-11-16T22:40:42.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 11356603 '!=' null 2020-11-16T22:40:42.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:42.912 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:42.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:42.912 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:43.015 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:43.015 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:43.016 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:43.016 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:43.016 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:43.016 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:43.016 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:43.016 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:43.016 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:43.017 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:43.017 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:43.017 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:43.017 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:43.017 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:43.017 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:43.017 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:43.017 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:43.018 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:43.018 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:43.018 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:43.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:43.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:43.126 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:43.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:43.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:43.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:43.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:43.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:43.128 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:43.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:43.761 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:43.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:43.762 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:44.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:44.238 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:44.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:44.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:44.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:44.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:44.239 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:44.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:44.240 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:44.733 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19219203 2020-11-16T22:40:44.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19219203 '!=' null 2020-11-16T22:40:44.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:44.734 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:44.734 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:44.735 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:44.825 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:44.825 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:44.825 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:44.825 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:44.825 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:44.825 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:44.826 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:44.826 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:44.826 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:44.826 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:44.826 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:44.826 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:44.826 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:44.827 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:44.827 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:44.827 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:44.827 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:44.827 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:44.827 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:44.827 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:44.833 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:44.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:44.934 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:44.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:44.935 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:44.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:44.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:44.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:44.936 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:45.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:45.291 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:45.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:45.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:45.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:45.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:45.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:45.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:45.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:45.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:45.839 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:45.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:45.840 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:46.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=10483245 2020-11-16T22:40:46.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 10483245 '!=' null 2020-11-16T22:40:46.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:46.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:46.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:46.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:46.507 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:46.507 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:46.507 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:46.507 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:46.507 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:46.507 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:46.508 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:46.508 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:46.508 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:46.508 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:46.508 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:46.508 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:46.508 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:46.509 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:46.509 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:46.509 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:46.509 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:46.509 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:46.509 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:46.509 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:46.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:46.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:46.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:46.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:46.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:46.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:46.620 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:46.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:46.621 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:47.004 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:47.004 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:47.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:47.005 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:47.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:47.504 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:47.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=10483178 2020-11-16T22:40:47.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 10483178 '!=' null 2020-11-16T22:40:47.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:47.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:47.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:47.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:48.067 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:48.067 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:48.068 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:48.068 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:48.068 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:48.068 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:48.068 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:48.068 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:48.068 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:48.069 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:48.069 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:48.069 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:48.069 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:48.069 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:48.069 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:48.069 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:48.070 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:48.070 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:48.070 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:48.070 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:48.076 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:48.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:48.178 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:48.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:48.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:48.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:48.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:48.179 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:48.180 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:48.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:48.533 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:48.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:48.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:49.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:49.018 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:49.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:49.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:49.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:49.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:49.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:49.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:49.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:49.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19218962 2020-11-16T22:40:49.695 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19218962 '!=' null 2020-11-16T22:40:49.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:49.696 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:49.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:49.696 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:49.797 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:49.798 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:49.798 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:49.798 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:49.798 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:49.798 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:49.799 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:49.799 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:49.799 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:49.799 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:49.799 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:49.799 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:49.799 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:49.799 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:49.800 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:49.800 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:49.800 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:49.800 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:49.800 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:49.800 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:49.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:49.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:49.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:49.909 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:49.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:49.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:49.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:49.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:49.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:50.321 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:50.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:50.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:50.323 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:50.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:50.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:50.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:50.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:50.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:50.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:50.835 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:50.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:50.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:51.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19218962 2020-11-16T22:40:51.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19218962 '!=' null 2020-11-16T22:40:51.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:51.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:51.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:51.329 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:51.429 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:51.429 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:51.429 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:51.429 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:51.430 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:51.430 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:51.430 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:51.430 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:51.430 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:51.430 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:51.430 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:51.431 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:51.431 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:51.431 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:51.431 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:51.431 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:51.431 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:51.431 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:51.432 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:51.432 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:51.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:51.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:51.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:51.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:51.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:51.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:51.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:51.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:51.541 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:51.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:51.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:51.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:51.919 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:52.475 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:52.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:52.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:52.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:52.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:52.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:52.476 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:52.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:52.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:52.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597827 2020-11-16T22:40:52.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597827 '!=' null 2020-11-16T22:40:52.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:52.960 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:52.960 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:52.961 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:53.054 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:53.055 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:53.055 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:53.055 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:53.055 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:53.055 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:53.055 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:53.055 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:53.056 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:53.056 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:53.056 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:53.056 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:53.056 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:53.056 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:53.056 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:53.056 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:53.057 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:53.057 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:53.057 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:53.057 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:53.063 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:53.164 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:53.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:53.165 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:53.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:53.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:53.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:53.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:53.166 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:53.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:53.539 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:53.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:53.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:54.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:54.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:54.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:54.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:54.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:54.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:54.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:54.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:54.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:54.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24460094 2020-11-16T22:40:54.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24460094 '!=' null 2020-11-16T22:40:54.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:54.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:54.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:54.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:54.636 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:54.636 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:54.637 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:54.637 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:54.637 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:54.637 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:54.637 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:54.637 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:54.637 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:54.638 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:54.638 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:54.638 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:54.638 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:54.638 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:54.638 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:54.638 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:54.639 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:54.639 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:54.639 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:54.639 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:54.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:54.746 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:54.747 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:54.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:54.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:54.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:54.748 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:54.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:54.749 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:55.128 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:55.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:55.129 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:55.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:55.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:55.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:55.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:55.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:55.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:55.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:55.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:55.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:55.665 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:56.249 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597931 2020-11-16T22:40:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597931 '!=' null 2020-11-16T22:40:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:56.250 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:56.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:56.250 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:56.350 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:56.350 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:56.351 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:56.351 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:56.351 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:56.351 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:56.351 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:56.351 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:56.351 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:56.352 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:56.352 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:56.352 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:56.352 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:56.352 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:56.352 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:56.352 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:56.353 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:56.353 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:56.353 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:56.353 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:56.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:56.461 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:56.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:56.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:56.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:56.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:56.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:56.463 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:56.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:56.865 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:56.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:56.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:57.364 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:57.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:57.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:57.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:57.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:57.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:57.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:57.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:57.366 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:57.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16598002 2020-11-16T22:40:57.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16598002 '!=' null 2020-11-16T22:40:57.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:57.898 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:57.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:57.898 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:57.996 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:57.996 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:57.996 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:57.997 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:57.997 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:57.997 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:57.997 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:57.997 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:57.997 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:57.997 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:57.998 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:57.998 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:57.998 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:57.998 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:57.998 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:57.998 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:57.998 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:57.999 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:57.999 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:57.999 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:58.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:58.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:58.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:58.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:58.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:58.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:58.107 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:58.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:58.108 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:40:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:40:58.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:40:58.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:40:58.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:40:58.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:40:58.932 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:40:58.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:40:58.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:40:58.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:40:58.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:40:58.933 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:40:58.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:40:58.934 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:40:59.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16598002 2020-11-16T22:40:59.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16598002 '!=' null 2020-11-16T22:40:59.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:40:59.434 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:59.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:40:59.434 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:40:59.533 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:40:59.533 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:40:59.534 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:59.534 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:59.534 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:59.534 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:40:59.534 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:40:59.534 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:40:59.534 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:40:59.535 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:40:59.535 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:59.535 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:40:59.535 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:40:59.535 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:40:59.535 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:40:59.535 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:40:59.536 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:40:59.536 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:40:59.536 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:40:59.536 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:40:59.542 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:40:59.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:40:59.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:40:59.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:40:59.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:40:59.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:40:59.645 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:40:59.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:40:59.646 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:00.050 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:00.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:00.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:00.052 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:00.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:00.532 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:00.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:00.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:00.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:00.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:00.533 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:00.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:00.534 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586868 2020-11-16T22:41:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586868 '!=' null 2020-11-16T22:41:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:00.991 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:00.992 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:01.091 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:01.092 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:01.092 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:01.092 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:01.092 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:01.092 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:01.093 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:01.093 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:01.093 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:01.093 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:01.093 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:01.093 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:01.093 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:01.093 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:01.094 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:01.094 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:01.094 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:01.094 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:01.094 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:01.094 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:01.100 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:01.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:01.202 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:01.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:01.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:01.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:01.203 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:01.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:01.204 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:01.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:01.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:01.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:01.586 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:02.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:02.061 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:02.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:02.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:02.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:02.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:02.062 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:02.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:02.063 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:02.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850993 2020-11-16T22:41:02.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850993 '!=' null 2020-11-16T22:41:02.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:02.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:02.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:02.540 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:02.639 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:02.640 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:02.641 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:02.642 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:02.642 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:02.642 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:02.642 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:02.642 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:02.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:02.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:02.750 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:02.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:02.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:02.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:02.751 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:02.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:02.752 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:03.132 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:03.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:03.133 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:03.134 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:03.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:03.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:03.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:03.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:03.623 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:03.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:03.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:03.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:03.624 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:04.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851092 2020-11-16T22:41:04.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851092 '!=' null 2020-11-16T22:41:04.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:04.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:04.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:04.198 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:04.300 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:04.300 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:04.300 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:04.300 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:04.301 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:04.302 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:04.303 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:04.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:04.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:04.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:04.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:04.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:04.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:04.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:04.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:04.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:04.793 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:04.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:04.794 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:04.795 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:05.429 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:05.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:05.431 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:05.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724713 2020-11-16T22:41:05.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724713 '!=' null 2020-11-16T22:41:05.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:05.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:05.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:05.956 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:06.055 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:06.056 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:06.056 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:06.056 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:06.056 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:06.056 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:06.056 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:06.056 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:06.057 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:06.057 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:06.057 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:06.057 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:06.057 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:06.057 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:06.057 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:06.058 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:06.058 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:06.058 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:06.058 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:06.058 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:06.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:06.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:06.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:06.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:06.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:06.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:06.167 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:06.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:06.168 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:06.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:06.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:06.546 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:06.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:07.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:07.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:07.011 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:07.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:07.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:07.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:07.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:07.012 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:07.013 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:07.558 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724713 2020-11-16T22:41:07.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724713 '!=' null 2020-11-16T22:41:07.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:07.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:07.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:07.559 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:07.661 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:07.661 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:07.661 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:07.662 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:07.662 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:07.662 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:07.662 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:07.662 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:07.662 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:07.662 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:07.663 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:07.663 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:07.663 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:07.663 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:07.663 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:07.663 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:07.663 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:07.663 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:07.664 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:07.664 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:07.670 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:07.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:07.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:07.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:07.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:07.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:07.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:07.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:07.774 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:08.138 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:08.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:08.139 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:08.140 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:08.591 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:08.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:08.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:08.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:08.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:08.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:08.592 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:08.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:08.593 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:09.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724988 2020-11-16T22:41:09.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724988 '!=' null 2020-11-16T22:41:09.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:09.078 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:09.078 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:09.078 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:09.175 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:09.176 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:09.176 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:09.176 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:09.176 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:09.176 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:09.177 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:09.177 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:09.177 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:09.177 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:09.177 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:09.177 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:09.177 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:09.177 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:09.178 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:09.178 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:09.178 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:09.178 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:09.178 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:09.178 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:09.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:09.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:09.285 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:09.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:09.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:09.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:09.286 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:09.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:09.287 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:09.671 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:09.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:09.672 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:09.673 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:10.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:10.176 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:10.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:10.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:10.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:10.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:10.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:10.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:10.178 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:10.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22713891 2020-11-16T22:41:10.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22713891 '!=' null 2020-11-16T22:41:10.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:10.653 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:10.653 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:10.654 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:10.755 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:10.756 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:10.756 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:10.756 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:10.756 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:10.756 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:10.757 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:10.757 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:10.757 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:10.757 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:10.757 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:10.757 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:10.757 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:10.758 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:10.758 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:10.758 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:10.758 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:10.758 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:10.758 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:10.758 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:10.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:10.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:10.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:10.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:10.866 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:10.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:10.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:10.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:10.867 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:11.231 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:11.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:11.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:11.232 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:11.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:11.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:11.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:11.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:11.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:11.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:11.702 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:11.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:11.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:12.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724588 2020-11-16T22:41:12.196 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724588 '!=' null 2020-11-16T22:41:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:12.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:12.197 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:12.297 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:12.297 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:12.298 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:12.298 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:12.298 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:12.298 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:12.298 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:12.298 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:12.298 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:12.299 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:12.299 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:12.299 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:12.299 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:12.299 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:12.299 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:12.299 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:12.300 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:12.300 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:12.300 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:12.300 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:12.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:12.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:12.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:12.409 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:12.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:12.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:12.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:12.410 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:12.411 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:12.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:12.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:12.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:12.773 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:13.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:13.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:13.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:13.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:13.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:13.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:13.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:13.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:13.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:13.915 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724693 2020-11-16T22:41:13.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724693 '!=' null 2020-11-16T22:41:13.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:13.916 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:13.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:13.916 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:14.014 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:14.014 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:14.014 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:14.014 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:14.014 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:14.015 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:14.015 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:14.015 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:14.015 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:14.015 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:14.015 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:14.015 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:14.016 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:14.016 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:14.016 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:14.016 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:14.016 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:14.016 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:14.016 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:14.016 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:14.022 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:14.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:14.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:14.125 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:14.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:14.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:14.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:14.126 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:14.127 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:14.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:14.509 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:14.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:14.510 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:15.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:15.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:15.103 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:15.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:15.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:15.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:15.104 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:15.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:15.105 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724693 2020-11-16T22:41:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724693 '!=' null 2020-11-16T22:41:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:15.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:15.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:15.709 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:15.811 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:15.811 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:15.811 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:15.811 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:15.811 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:15.811 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:15.812 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:15.812 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:15.812 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:15.812 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:15.812 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:15.812 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:15.812 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:15.813 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:15.813 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:15.813 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:15.813 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:15.813 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:15.813 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:15.813 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:15.820 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:15.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:15.922 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:15.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:15.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:15.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:15.923 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:15.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:15.924 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:16.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:16.343 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:16.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:16.344 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:16.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:16.828 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:16.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:16.828 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:17.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23587070 2020-11-16T22:41:17.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23587070 '!=' null 2020-11-16T22:41:17.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:17.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:17.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:17.303 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:17.402 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:17.402 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:17.402 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:17.402 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:17.402 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:17.402 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:17.403 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:17.403 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:17.403 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:17.403 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:17.403 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:17.403 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:17.403 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:17.404 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:17.404 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:17.404 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:17.404 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:17.404 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:17.404 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:17.404 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:17.410 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:17.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:17.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:17.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:17.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:17.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:17.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:17.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:17.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:17.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:17.917 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:17.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:17.918 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:18.404 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:18.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:18.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:18.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:18.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:18.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:18.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:18.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:18.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:18.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850945 2020-11-16T22:41:18.886 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850945 '!=' null 2020-11-16T22:41:18.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:18.887 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:18.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:18.887 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:18.988 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:18.989 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:18.989 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:18.989 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:18.989 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:18.989 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:18.989 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:18.990 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:18.990 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:18.990 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:18.990 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:18.990 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:18.990 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:18.990 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:18.991 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:18.991 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:18.991 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:18.991 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:18.991 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:18.991 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:18.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:19.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:19.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:19.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:19.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:19.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:19.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:19.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:19.101 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:19.482 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:19.482 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:19.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:19.483 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:20.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:20.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:20.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:20.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:20.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:20.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:20.013 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:20.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:20.014 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:20.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21839273 2020-11-16T22:41:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21839273 '!=' null 2020-11-16T22:41:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:20.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:20.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:20.479 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:20.573 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:20.574 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:20.574 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:20.574 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:20.574 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:20.574 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:20.574 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:20.575 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:20.575 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:20.575 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:20.575 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:20.575 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:20.575 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:20.575 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:20.576 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:20.576 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:20.576 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:20.576 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:20.576 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:20.576 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:20.582 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:20.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:20.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:20.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:20.685 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:20.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:20.686 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:21.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:21.045 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:21.046 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:21.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:21.521 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:21.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:21.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:21.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:21.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:21.522 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:21.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:21.523 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:22.039 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850506 2020-11-16T22:41:22.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850506 '!=' null 2020-11-16T22:41:22.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:22.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:22.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:22.040 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:22.145 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:22.146 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:22.146 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:22.146 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:22.146 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:22.146 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:22.146 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:22.147 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:22.147 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:22.147 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:22.147 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:22.147 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:22.147 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:22.147 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:22.147 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:22.148 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:22.148 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:22.148 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:22.148 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:22.148 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:22.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:22.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:22.255 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:22.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:22.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:22.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:22.256 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:22.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:22.257 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:22.640 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:22.641 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:22.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:22.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:23.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:23.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:23.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:23.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:23.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:23.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:23.160 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:23.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:23.160 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:23.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850506 2020-11-16T22:41:23.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850506 '!=' null 2020-11-16T22:41:23.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:23.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:23.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:23.739 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:23.839 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:23.869 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:23.869 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:23.869 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:23.870 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:23.870 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:23.870 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:23.870 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:23.870 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:23.870 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:23.870 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:23.871 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:23.871 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:23.871 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:23.871 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:23.871 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:23.871 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:23.871 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:23.872 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:23.872 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:23.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:23.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:23.950 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:23.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:23.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:23.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:23.951 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:23.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:23.952 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:24.350 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:24.350 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:24.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:24.351 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:24.814 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:24.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:24.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:24.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:24.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:24.815 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:24.816 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:24.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:24.816 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:25.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850724 2020-11-16T22:41:25.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850724 '!=' null 2020-11-16T22:41:25.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:25.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:25.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:25.826 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:25.937 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:25.937 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:25.937 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:25.937 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:25.938 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:25.938 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:25.938 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:25.938 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:25.938 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:25.938 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:25.938 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:25.939 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:25.939 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:25.939 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:25.939 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:25.939 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:25.939 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:25.939 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:25.939 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:25.940 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:25.947 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:26.049 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:26.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:26.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:26.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:26.050 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:26.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:26.051 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:26.416 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:26.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:26.417 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:26.418 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:26.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:26.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:26.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:26.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:26.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:26.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:26.894 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:26.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:26.895 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:27.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586195 2020-11-16T22:41:27.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586195 '!=' null 2020-11-16T22:41:27.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:27.405 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:27.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:27.406 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:27.501 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:27.501 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:27.502 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:27.503 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:27.504 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:27.504 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:27.504 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:27.504 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:27.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:27.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:27.611 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:27.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:27.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:27.612 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:27.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:27.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:27.613 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:27.981 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:27.982 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:27.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:27.983 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:28.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:28.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:28.755 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:28.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:28.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:28.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:28.756 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:28.756 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:28.757 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:29.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724056 2020-11-16T22:41:29.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724056 '!=' null 2020-11-16T22:41:29.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:29.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:29.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:29.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:29.344 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:29.344 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:29.345 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:29.345 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:29.345 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:29.345 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:29.345 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:29.345 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:29.345 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:29.345 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:29.346 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:29.346 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:29.346 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:29.346 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:29.346 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:29.346 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:29.346 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:29.347 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:29.347 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:29.347 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:29.353 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:29.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:29.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:29.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:29.455 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:29.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:29.456 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:29.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:29.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:29.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:29.823 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:30.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:30.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:30.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:30.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:30.359 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:30.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:30.360 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:30.360 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:30.361 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:30.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586047 2020-11-16T22:41:30.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586047 '!=' null 2020-11-16T22:41:30.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:30.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:30.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:30.837 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:30.931 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:30.932 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:30.932 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:30.932 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:30.932 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:30.932 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:30.932 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:30.932 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:30.933 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:30.933 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:30.933 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:30.933 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:30.933 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:30.933 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:30.933 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:30.933 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:30.934 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:30.934 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:30.934 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:30.934 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:30.938 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:31.415 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:31.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:31.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:31.416 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:31.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:31.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:31.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:31.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:31.989 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:31.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:31.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:31.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:31.990 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:32.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597526 2020-11-16T22:41:32.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597526 '!=' null 2020-11-16T22:41:32.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:32.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:32.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:32.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:32.762 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:32.762 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:32.763 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:32.763 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:32.763 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:32.763 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:32.763 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:32.763 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:32.763 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:32.764 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:32.764 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:32.764 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:32.764 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:32.764 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:32.764 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:32.764 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:32.764 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:32.765 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:32.765 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:32.765 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:32.771 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:32.873 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:32.874 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:32.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:32.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:32.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:32.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:32.875 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:33.246 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:33.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:33.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:33.248 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:33.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:33.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:33.789 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:33.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16597852 2020-11-16T22:41:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16597852 '!=' null 2020-11-16T22:41:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:34.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:34.371 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:34.459 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:34.459 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:34.459 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:34.459 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:34.460 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:34.460 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:34.460 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:34.460 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:34.460 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:34.460 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:34.460 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:34.461 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:34.461 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:34.461 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:34.461 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:34.461 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:34.461 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:34.461 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:34.461 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:34.462 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:34.467 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:34.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:34.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:34.569 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:34.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:34.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:34.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:34.570 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:34.571 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:34.949 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:34.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:34.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:34.950 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:35.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:35.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:35.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:35.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:35.407 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:35.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:35.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:35.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:35.408 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:35.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20965506 2020-11-16T22:41:35.900 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20965506 '!=' null 2020-11-16T22:41:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:35.901 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:35.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:35.901 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:35.992 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:35.993 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:35.993 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:35.993 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:35.993 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:35.993 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:35.993 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:35.993 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:35.994 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:35.994 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:35.994 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:35.994 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:35.994 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:35.994 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:35.994 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:35.995 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:35.995 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:35.995 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:35.995 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:35.995 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:36.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:36.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:36.102 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:36.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:36.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:36.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:36.103 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:36.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:36.104 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:36.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:36.477 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:36.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:36.478 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:36.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:36.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:36.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:36.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:36.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:36.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:36.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:36.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:36.965 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:37.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20965506 2020-11-16T22:41:37.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20965506 '!=' null 2020-11-16T22:41:37.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:37.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:37.448 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:37.449 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:37.550 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:37.551 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:37.551 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:37.551 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:37.551 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:37.551 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:37.551 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:37.551 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:37.552 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:37.552 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:37.552 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:37.552 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:37.552 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:37.552 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:37.552 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:37.553 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:37.553 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:37.553 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:37.553 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:37.553 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:37.559 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:37.660 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:37.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:37.661 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:37.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:37.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:37.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:37.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:37.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:38.228 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:38.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:38.229 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:38.230 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:38.691 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:38.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:38.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:38.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:38.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:38.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:38.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:38.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:38.693 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12229925 2020-11-16T22:41:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12229925 '!=' null 2020-11-16T22:41:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:39.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:39.292 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:39.388 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:39.388 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:39.389 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:39.389 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:39.389 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:39.389 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:39.389 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:39.389 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:39.389 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:39.390 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:39.390 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:39.390 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:39.390 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:39.390 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:39.390 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:39.390 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:39.391 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:39.391 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:39.391 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:39.391 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:39.498 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:39.499 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:39.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:39.499 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:39.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:39.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:39.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:39.500 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:39.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:39.891 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:39.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:39.892 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:40.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:40.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:40.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:40.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:40.346 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:40.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:40.347 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:40.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:40.347 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:40.926 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20092039 2020-11-16T22:41:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20092039 '!=' null 2020-11-16T22:41:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:40.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:40.927 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:41.020 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:41.020 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:41.020 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:41.021 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:41.021 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:41.021 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:41.021 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:41.021 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:41.021 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:41.021 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:41.022 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:41.022 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:41.022 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:41.022 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:41.022 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:41.022 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:41.022 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:41.023 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:41.023 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:41.023 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:41.028 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:41.129 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:41.130 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:41.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:41.130 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:41.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:41.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:41.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:41.131 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:41.512 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:41.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:41.513 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:41.514 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:42.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:42.024 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:42.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:42.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:42.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:42.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:42.025 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:42.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:42.026 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:42.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12229950 2020-11-16T22:41:42.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12229950 '!=' null 2020-11-16T22:41:42.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:42.519 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:42.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:42.519 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:42.617 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:42.618 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:42.618 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:42.618 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:42.618 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:42.618 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:42.619 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:42.619 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:42.619 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:42.619 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:42.619 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:42.619 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:42.619 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:42.620 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:42.620 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:42.620 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:42.620 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:42.620 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:42.620 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:42.620 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:42.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:42.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:42.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:42.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:42.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:42.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:42.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:42.731 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:43.116 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:43.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:43.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:43.117 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:43.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:43.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:43.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:43.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:43.961 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:43.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:43.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:43.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:44.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12230065 2020-11-16T22:41:44.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12230065 '!=' null 2020-11-16T22:41:44.432 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:44.433 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:44.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:44.433 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:44.531 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:44.531 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:44.531 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:44.531 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:44.531 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:44.532 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:44.532 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:44.532 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:44.532 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:44.532 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:44.532 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:44.532 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:44.533 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:44.533 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:44.533 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:44.533 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:44.533 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:44.533 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:44.533 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:44.533 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:44.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:44.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:44.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:44.642 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:44.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:44.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:44.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:44.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:44.643 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:45.019 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:45.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:45.020 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:45.021 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:45.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:45.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:45.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:45.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:45.515 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:45.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:45.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:45.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:45.516 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20091921 2020-11-16T22:41:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20091921 '!=' null 2020-11-16T22:41:46.031 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:46.032 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:46.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:46.032 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:46.133 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:46.133 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:46.133 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:46.133 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:46.133 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:46.133 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:46.134 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:46.134 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:46.134 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:46.134 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:46.134 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:46.134 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:46.135 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:46.135 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:46.135 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:46.135 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:46.135 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:46.135 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:46.135 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:46.136 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:46.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:46.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:46.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:46.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:46.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:46.244 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:46.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:46.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:46.245 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:46.703 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:46.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:46.704 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:46.705 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:47.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:47.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:47.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:47.328 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:47.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724145 2020-11-16T22:41:47.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724145 '!=' null 2020-11-16T22:41:47.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:47.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:47.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:47.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:47.932 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:47.932 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:47.932 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:47.933 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:47.933 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:47.933 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:47.933 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:47.933 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:47.933 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:47.933 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:47.933 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:47.934 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:47.934 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:47.934 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:47.934 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:47.934 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:47.934 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:47.934 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:47.935 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:47.935 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:47.940 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:48.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:48.043 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:48.043 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:48.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:48.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:48.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:48.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:48.044 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:48.528 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:48.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:48.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:48.529 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:48.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:48.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:48.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:48.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:48.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:48.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:48.985 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:48.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:48.986 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:49.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15724145 2020-11-16T22:41:49.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15724145 '!=' null 2020-11-16T22:41:49.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:49.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:49.491 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:49.492 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:49.591 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:49.591 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:49.591 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:49.592 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:49.592 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:49.592 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:49.592 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:49.592 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:49.592 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:49.592 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:49.593 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:49.593 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:49.593 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:49.593 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:49.593 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:49.593 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:49.593 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:49.593 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:49.594 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:49.594 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:49.600 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:49.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:49.701 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:49.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:49.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:49.702 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:49.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:49.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:49.703 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:50.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:50.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:50.175 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:50.176 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:50.649 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:50.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:50.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:50.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:50.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:50.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:50.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:50.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:50.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:51.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22712541 2020-11-16T22:41:51.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22712541 '!=' null 2020-11-16T22:41:51.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:51.205 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:51.205 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:51.206 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:51.301 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:51.302 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:51.303 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:51.304 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:51.304 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:51.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:51.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:51.411 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:51.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:51.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:51.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:51.412 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:51.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:51.413 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:51.802 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:51.803 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:51.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:51.804 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:52.286 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:52.287 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:52.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:52.288 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:53.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850633 2020-11-16T22:41:53.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850633 '!=' null 2020-11-16T22:41:53.005 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:53.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:53.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:53.006 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:53.102 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:53.102 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:53.103 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:53.103 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:53.103 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:53.103 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [ 2020-11-16T22:41:53.103 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:41:53.103 INFO:tasks.workunit.client.0.smithi189.stdout: "item": "3.0", 2020-11-16T22:41:53.103 INFO:tasks.workunit.client.0.smithi189.stdout: "prio": 190, 2020-11-16T22:41:53.104 INFO:tasks.workunit.client.0.smithi189.stdout: "can_preempt": true 2020-11-16T22:41:53.104 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:53.104 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:41:53.104 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:53.104 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:53.104 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:53.104 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:53.104 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:53.105 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:53.105 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:53.105 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:53.110 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:53.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:53.212 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:53.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:53.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:53.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:53.213 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:53.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:53.214 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:53.603 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:53.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:53.604 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:53.605 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:54.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:54.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:54.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:54.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:54.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:54.099 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:54.100 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14850791 2020-11-16T22:41:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14850791 '!=' null 2020-11-16T22:41:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:54.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:54.585 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:54.679 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:54.679 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:54.679 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:54.680 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:54.680 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:54.680 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:54.680 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:54.680 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:54.680 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:54.680 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:54.681 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:54.681 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:54.681 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:54.681 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:54.687 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:54.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:54.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:54.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:54.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:54.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:54.790 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:54.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:54.791 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:55.187 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:55.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:55.188 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:55.189 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:55.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:55.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:55.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:55.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:55.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:55.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:55.864 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:55.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:55.864 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23586529 2020-11-16T22:41:56.335 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23586529 '!=' null 2020-11-16T22:41:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:56.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:56.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:56.336 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:56.434 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:56.434 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:56.434 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:56.434 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:56.435 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:56.435 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:56.435 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:56.435 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:56.435 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:56.435 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:56.435 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:56.436 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:56.436 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:56.436 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:56.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:56.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:56.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:56.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:56.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:56.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:56.547 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:56.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:56.548 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:56.908 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:56.909 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:56.909 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:56.910 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:57.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:57.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:57.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:57.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:57.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:57.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:57.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:57.382 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:57.383 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:57.857 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13103919 2020-11-16T22:41:57.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13103919 '!=' null 2020-11-16T22:41:57.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:57.858 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:57.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:57.858 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:57.954 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:57.955 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:57.955 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:57.955 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:57.955 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:57.955 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:57.955 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:57.956 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:57.956 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:57.956 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:57.956 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:57.956 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:57.956 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:57.956 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:57.962 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:58.065 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:58.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:58.065 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:58.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:58.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:58.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:58.066 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2020-11-16T22:41:58.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:58.453 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:58.454 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:41:58.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2020-11-16T22:41:58.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2020-11-16T22:41:58.928 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-11-16T22:41:58.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-11-16T22:41:58.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-11-16T22:41:58.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-11-16T22:41:58.929 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-11-16T22:41:58.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-11-16T22:41:58.930 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-11-16T22:41:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13103919 2020-11-16T22:41:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13103919 '!=' null 2020-11-16T22:41:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-11-16T22:41:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:59.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2020-11-16T22:41:59.384 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_recovery_reservations 2020-11-16T22:41:59.482 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:41:59.482 INFO:tasks.workunit.client.0.smithi189.stdout: "local_reservations": { 2020-11-16T22:41:59.482 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:59.483 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:59.483 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:59.483 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:59.483 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:41:59.483 INFO:tasks.workunit.client.0.smithi189.stdout: "remote_reservations": { 2020-11-16T22:41:59.483 INFO:tasks.workunit.client.0.smithi189.stdout: "max_allowed": 1, 2020-11-16T22:41:59.483 INFO:tasks.workunit.client.0.smithi189.stdout: "min_priority": 0, 2020-11-16T22:41:59.483 INFO:tasks.workunit.client.0.smithi189.stdout: "queues": [], 2020-11-16T22:41:59.484 INFO:tasks.workunit.client.0.smithi189.stdout: "in_progress": [] 2020-11-16T22:41:59.484 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:41:59.484 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:41:59.491 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-11-16T22:41:59.593 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-11-16T22:41:59.594 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:41:59.594 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:41:59.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:41:59.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:41:59.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:41:59.595 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:41:59.596 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:41:59.967 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2020-11-16T22:41:59.968 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:41:59.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:41:59.969 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:42:00.452 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2020-11-16T22:42:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:42:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:42:00.453 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:295: TEST_recovery_priority: ceph pg dump pgs 2020-11-16T22:42:00.816 INFO:tasks.workunit.client.0.smithi189.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-11-16T22:42:00.816 INFO:tasks.workunit.client.0.smithi189.stdout:7.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2020-11-16T22:38:06.625679+0000 91'600 117:1086 [1,4] 1 [1,4] 1 0'0 2020-11-16T22:29:00.696185+0000 0'0 2020-11-16T22:29:00.696185+0000 0 2020-11-16T22:42:00.816 INFO:tasks.workunit.client.0.smithi189.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2020-11-16T22:39:59.126330+0000 91'600 117:1102 [1,2] 1 [1,2] 1 0'0 2020-11-16T22:28:45.457031+0000 0'0 2020-11-16T22:28:45.457031+0000 0 2020-11-16T22:42:00.816 INFO:tasks.workunit.client.0.smithi189.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2020-11-16T22:41:55.184332+0000 91'600 118:1107 [1,2] 1 [1,2] 1 0'0 2020-11-16T22:28:50.266785+0000 0'0 2020-11-16T22:28:50.266785+0000 0 2020-11-16T22:42:00.817 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-16T22:42:00.817 INFO:tasks.workunit.client.0.smithi189.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-11-16T22:42:00.818 INFO:tasks.workunit.client.0.smithi189.stderr:dumped pgs 2020-11-16T22:42:00.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:297: TEST_recovery_priority: get_asok_path osd.1 2020-11-16T22:42:00.829 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-11-16T22:42:00.830 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-11-16T22:42:00.830 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:42:00.830 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:42:00.831 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:42:00.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679603/ceph-osd.1.asok 2020-11-16T22:42:00.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:297: TEST_recovery_priority: CEPH_ARGS= 2020-11-16T22:42:00.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:297: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.679603/ceph-osd.1.asok dump_pgstate_history 2020-11-16T22:42:00.932 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2020-11-16T22:42:00.932 INFO:tasks.workunit.client.0.smithi189.stdout: "pgs": [ 2020-11-16T22:42:00.932 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.932 INFO:tasks.workunit.client.0.smithi189.stdout: "pg": "7.0", 2020-11-16T22:42:00.933 INFO:tasks.workunit.client.0.smithi189.stdout: "currently": "Started/Primary/Active/Clean", 2020-11-16T22:42:00.933 INFO:tasks.workunit.client.0.smithi189.stdout: "history": [ 2020-11-16T22:42:00.933 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.933 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "60", 2020-11-16T22:42:00.933 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.933 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.933 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:00.934 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:01.223481+0000", 2020-11-16T22:42:00.934 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:01.223491+0000" 2020-11-16T22:42:00.934 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.934 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.934 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:00.934 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:01.223497+0000", 2020-11-16T22:42:00.934 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:01.245096+0000" 2020-11-16T22:42:00.935 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.935 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.935 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:00.935 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:01.245097+0000", 2020-11-16T22:42:00.935 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:01.245406+0000" 2020-11-16T22:42:00.935 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.935 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.935 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T22:42:00.936 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:01.245408+0000", 2020-11-16T22:42:00.936 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:01.245482+0000" 2020-11-16T22:42:00.936 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.936 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.936 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T22:42:00.936 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:01.245483+0000", 2020-11-16T22:42:00.936 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:01.541929+0000" 2020-11-16T22:42:00.937 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.937 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.937 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:00.937 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:01.223494+0000", 2020-11-16T22:42:00.937 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:01.541948+0000" 2020-11-16T22:42:00.937 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.937 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.938 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T22:42:00.938 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:01.542356+0000", 2020-11-16T22:42:00.938 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:02.686820+0000" 2020-11-16T22:42:00.938 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.938 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.938 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T22:42:00.938 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:01.541949+0000", 2020-11-16T22:42:00.938 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:02.686860+0000" 2020-11-16T22:42:00.939 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.939 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.939 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:00.939 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:01.223491+0000", 2020-11-16T22:42:00.939 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:02.686918+0000" 2020-11-16T22:42:00.939 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.939 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.940 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:00.940 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:01.223477+0000", 2020-11-16T22:42:00.940 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:02.686966+0000" 2020-11-16T22:42:00.940 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.940 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.940 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.940 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.940 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "60", 2020-11-16T22:42:00.941 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.941 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.941 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:00.941 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:02.686973+0000", 2020-11-16T22:42:00.941 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:02.687391+0000" 2020-11-16T22:42:00.941 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.941 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.942 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.942 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.942 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "89", 2020-11-16T22:42:00.942 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.942 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.942 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:00.942 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:02.687399+0000", 2020-11-16T22:42:00.943 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:02.687427+0000" 2020-11-16T22:42:00.943 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.943 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.943 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:00.943 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:02.687444+0000", 2020-11-16T22:42:00.943 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:02.813060+0000" 2020-11-16T22:42:00.943 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.944 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.944 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:00.944 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:02.813061+0000", 2020-11-16T22:42:00.944 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:02.813126+0000" 2020-11-16T22:42:00.944 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.944 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.944 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T22:42:00.944 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:02.813127+0000", 2020-11-16T22:42:00.945 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:02.813140+0000" 2020-11-16T22:42:00.945 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.945 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.945 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T22:42:00.945 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:02.813143+0000", 2020-11-16T22:42:00.945 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:03.124057+0000" 2020-11-16T22:42:00.945 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.945 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.946 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:00.946 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:02.687436+0000", 2020-11-16T22:42:00.946 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:03.124065+0000" 2020-11-16T22:42:00.946 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.946 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.946 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T22:42:00.946 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:03.124200+0000", 2020-11-16T22:42:00.947 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:03.674549+0000" 2020-11-16T22:42:00.947 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.947 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.947 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T22:42:00.947 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:03.674549+0000", 2020-11-16T22:42:00.947 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:03.674634+0000" 2020-11-16T22:42:00.947 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.948 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.948 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T22:42:00.948 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:03.674635+0000", 2020-11-16T22:42:00.948 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.388909+0000" 2020-11-16T22:42:00.948 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.948 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.948 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T22:42:00.949 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:03.124066+0000", 2020-11-16T22:42:00.949 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.388957+0000" 2020-11-16T22:42:00.949 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.949 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.949 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:00.949 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:02.687427+0000", 2020-11-16T22:42:00.949 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.388987+0000" 2020-11-16T22:42:00.950 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.950 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.950 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:00.950 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:02.687392+0000", 2020-11-16T22:42:00.950 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.388995+0000" 2020-11-16T22:42:00.950 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.950 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.950 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.951 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.951 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "90", 2020-11-16T22:42:00.951 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.951 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.951 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:00.957 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.388996+0000", 2020-11-16T22:42:00.957 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.389553+0000" 2020-11-16T22:42:00.957 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.957 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.957 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.958 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.958 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "96", 2020-11-16T22:42:00.958 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.958 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.958 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:00.958 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.389565+0000", 2020-11-16T22:42:00.958 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.389599+0000" 2020-11-16T22:42:00.958 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.959 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.959 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:00.959 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.389621+0000", 2020-11-16T22:42:00.959 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.390018+0000" 2020-11-16T22:42:00.959 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.959 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.959 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:00.960 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.390020+0000", 2020-11-16T22:42:00.960 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.390280+0000" 2020-11-16T22:42:00.960 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.960 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.960 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T22:42:00.960 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.390281+0000", 2020-11-16T22:42:00.960 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.390342+0000" 2020-11-16T22:42:00.960 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.961 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.961 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T22:42:00.961 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.390343+0000", 2020-11-16T22:42:00.961 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.688400+0000" 2020-11-16T22:42:00.961 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.961 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.961 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:00.962 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.389608+0000", 2020-11-16T22:42:00.962 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.688450+0000" 2020-11-16T22:42:00.962 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.962 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.962 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T22:42:00.962 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.688615+0000", 2020-11-16T22:42:00.962 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.813030+0000" 2020-11-16T22:42:00.963 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.963 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.963 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T22:42:00.963 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.813045+0000", 2020-11-16T22:42:00.963 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:29.813302+0000" 2020-11-16T22:42:00.963 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.963 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.963 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T22:42:00.964 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.813304+0000", 2020-11-16T22:42:00.964 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.265569+0000" 2020-11-16T22:42:00.964 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.964 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.964 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T22:42:00.964 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.688452+0000", 2020-11-16T22:42:00.964 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.265641+0000" 2020-11-16T22:42:00.965 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.965 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.965 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:00.965 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.389600+0000", 2020-11-16T22:42:00.965 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.265852+0000" 2020-11-16T22:42:00.965 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.965 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.965 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:00.966 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:29.389553+0000", 2020-11-16T22:42:00.966 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.265875+0000" 2020-11-16T22:42:00.966 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.966 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.966 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.966 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.966 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "97", 2020-11-16T22:42:00.967 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.967 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.967 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:00.967 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.265889+0000", 2020-11-16T22:42:00.967 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.267205+0000" 2020-11-16T22:42:00.967 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.967 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.967 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.968 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.968 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "99", 2020-11-16T22:42:00.968 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.968 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.968 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:00.968 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.267227+0000", 2020-11-16T22:42:00.968 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.267309+0000" 2020-11-16T22:42:00.969 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.969 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.969 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:00.969 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.267351+0000", 2020-11-16T22:42:00.969 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.449921+0000" 2020-11-16T22:42:00.969 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.969 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.969 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:00.970 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.449922+0000", 2020-11-16T22:42:00.970 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.450081+0000" 2020-11-16T22:42:00.970 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.970 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.970 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:00.970 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.267330+0000", 2020-11-16T22:42:00.970 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.450102+0000" 2020-11-16T22:42:00.971 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.971 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.971 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/WaitActingChange", 2020-11-16T22:42:00.971 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.450102+0000", 2020-11-16T22:42:00.971 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.910103+0000" 2020-11-16T22:42:00.971 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.971 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.971 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:00.972 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.267310+0000", 2020-11-16T22:42:00.972 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.910152+0000" 2020-11-16T22:42:00.972 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.972 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.972 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:00.972 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.267206+0000", 2020-11-16T22:42:00.972 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.910163+0000" 2020-11-16T22:42:00.972 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.973 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.973 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.973 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.973 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "99", 2020-11-16T22:42:00.973 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.973 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.973 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:00.974 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.910164+0000", 2020-11-16T22:42:00.974 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.910762+0000" 2020-11-16T22:42:00.974 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.974 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.974 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.974 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.974 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "113", 2020-11-16T22:42:00.974 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.975 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.975 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:00.975 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.910778+0000", 2020-11-16T22:42:00.975 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.910820+0000" 2020-11-16T22:42:00.975 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.975 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.975 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:00.976 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.910842+0000", 2020-11-16T22:42:00.976 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.912112+0000" 2020-11-16T22:42:00.976 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.976 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.976 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:00.976 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.912113+0000", 2020-11-16T22:42:00.976 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.912310+0000" 2020-11-16T22:42:00.976 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.977 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.977 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T22:42:00.977 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.912311+0000", 2020-11-16T22:42:00.977 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:22.912364+0000" 2020-11-16T22:42:00.977 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.977 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.977 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T22:42:00.977 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.912365+0000", 2020-11-16T22:42:00.978 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:23.169994+0000" 2020-11-16T22:42:00.978 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.978 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.978 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:00.978 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.910832+0000", 2020-11-16T22:42:00.978 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:23.170019+0000" 2020-11-16T22:42:00.978 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.979 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.979 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T22:42:00.979 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:23.173138+0000", 2020-11-16T22:42:00.979 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:23.564312+0000" 2020-11-16T22:42:00.979 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.979 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.979 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2020-11-16T22:42:00.979 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:23.564313+0000", 2020-11-16T22:42:00.980 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:23.564497+0000" 2020-11-16T22:42:00.980 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.980 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.980 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2020-11-16T22:42:00.980 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:23.564498+0000", 2020-11-16T22:42:00.980 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:23.565039+0000" 2020-11-16T22:42:00.980 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.981 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.981 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovering", 2020-11-16T22:42:00.981 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:23.565040+0000", 2020-11-16T22:42:00.981 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:55.888544+0000" 2020-11-16T22:42:00.981 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.981 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.981 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/NotRecovering", 2020-11-16T22:42:00.981 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:55.888546+0000", 2020-11-16T22:42:00.982 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:55.889128+0000" 2020-11-16T22:42:00.982 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.982 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.982 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2020-11-16T22:42:00.982 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:55.889129+0000", 2020-11-16T22:42:00.982 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:36:08.477885+0000" 2020-11-16T22:42:00.982 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.983 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.983 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2020-11-16T22:42:00.983 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:36:08.477885+0000", 2020-11-16T22:42:00.983 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:36:08.478543+0000" 2020-11-16T22:42:00.983 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.983 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.983 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovering", 2020-11-16T22:42:00.983 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:36:08.478544+0000", 2020-11-16T22:42:00.984 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:38:05.028711+0000" 2020-11-16T22:42:00.984 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.984 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.984 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T22:42:00.984 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:38:05.028742+0000", 2020-11-16T22:42:00.984 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:38:05.420766+0000" 2020-11-16T22:42:00.984 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.985 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.985 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T22:42:00.985 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:23.170020+0000", 2020-11-16T22:42:00.985 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:38:05.420833+0000" 2020-11-16T22:42:00.985 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.985 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.985 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:00.985 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.910821+0000", 2020-11-16T22:42:00.986 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:38:05.420876+0000" 2020-11-16T22:42:00.986 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.986 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.986 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:00.986 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:22.910763+0000", 2020-11-16T22:42:00.986 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:38:05.420890+0000" 2020-11-16T22:42:00.986 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.987 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.987 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.987 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.987 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "113", 2020-11-16T22:42:00.987 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.987 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.987 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:00.987 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:38:05.420892+0000", 2020-11-16T22:42:00.988 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:38:05.422528+0000" 2020-11-16T22:42:00.988 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.988 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.988 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.988 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.988 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.988 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.989 INFO:tasks.workunit.client.0.smithi189.stdout: "pg": "3.0", 2020-11-16T22:42:00.989 INFO:tasks.workunit.client.0.smithi189.stdout: "currently": "Started/Primary/Active/Clean", 2020-11-16T22:42:00.989 INFO:tasks.workunit.client.0.smithi189.stdout: "history": [ 2020-11-16T22:42:00.989 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.989 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "38", 2020-11-16T22:42:00.989 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.989 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.989 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:00.990 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:50.560667+0000", 2020-11-16T22:42:00.990 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:50.560694+0000" 2020-11-16T22:42:00.990 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.990 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.990 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:00.990 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:50.560711+0000", 2020-11-16T22:42:00.990 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:50.711025+0000" 2020-11-16T22:42:00.991 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.991 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.991 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:00.991 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:50.711026+0000", 2020-11-16T22:42:00.991 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:50.711098+0000" 2020-11-16T22:42:00.991 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.991 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.992 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T22:42:00.992 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:50.711099+0000", 2020-11-16T22:42:00.992 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:50.711115+0000" 2020-11-16T22:42:00.992 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.992 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.992 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T22:42:00.992 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:50.711115+0000", 2020-11-16T22:42:00.992 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:50.996332+0000" 2020-11-16T22:42:00.993 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.993 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.993 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:00.993 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:50.560703+0000", 2020-11-16T22:42:00.993 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:50.996360+0000" 2020-11-16T22:42:00.993 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.993 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.993 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T22:42:00.994 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:50.996917+0000", 2020-11-16T22:42:00.994 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:51.366010+0000" 2020-11-16T22:42:00.994 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.994 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.994 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T22:42:00.994 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:51.366011+0000", 2020-11-16T22:42:00.994 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:51.366140+0000" 2020-11-16T22:42:00.995 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.995 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.995 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T22:42:00.995 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:51.366141+0000", 2020-11-16T22:42:00.995 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.205526+0000" 2020-11-16T22:42:00.995 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.995 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.996 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T22:42:00.996 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:50.996361+0000", 2020-11-16T22:42:00.996 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.205558+0000" 2020-11-16T22:42:00.996 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.996 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.996 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:00.996 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:50.560695+0000", 2020-11-16T22:42:00.996 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.205575+0000" 2020-11-16T22:42:00.997 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.997 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.997 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:00.997 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:50.560659+0000", 2020-11-16T22:42:00.997 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.205579+0000" 2020-11-16T22:42:00.997 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.997 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.998 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.998 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.998 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "38", 2020-11-16T22:42:00.998 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.998 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.998 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:00.998 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.205580+0000", 2020-11-16T22:42:00.999 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.205771+0000" 2020-11-16T22:42:00.999 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:00.999 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:00.999 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:00.999 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:00.999 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "87", 2020-11-16T22:42:00.999 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:00.999 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.000 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:01.000 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.205775+0000", 2020-11-16T22:42:01.000 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.205785+0000" 2020-11-16T22:42:01.000 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.000 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.000 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:01.000 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.205791+0000", 2020-11-16T22:42:01.001 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.207157+0000" 2020-11-16T22:42:01.001 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.001 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.001 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:01.001 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.207158+0000", 2020-11-16T22:42:01.001 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.207212+0000" 2020-11-16T22:42:01.001 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.001 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.002 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T22:42:01.002 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.207212+0000", 2020-11-16T22:42:01.002 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.207223+0000" 2020-11-16T22:42:01.002 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.002 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.002 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T22:42:01.002 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.207223+0000", 2020-11-16T22:42:01.003 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.644868+0000" 2020-11-16T22:42:01.003 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.003 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.003 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:01.003 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.205788+0000", 2020-11-16T22:42:01.003 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.644893+0000" 2020-11-16T22:42:01.003 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.003 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.004 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T22:42:01.004 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.645220+0000", 2020-11-16T22:42:01.004 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.737589+0000" 2020-11-16T22:42:01.004 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.004 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.004 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T22:42:01.004 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.737590+0000", 2020-11-16T22:42:01.005 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:52.737634+0000" 2020-11-16T22:42:01.005 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.005 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.005 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T22:42:01.005 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.737634+0000", 2020-11-16T22:42:01.005 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.281865+0000" 2020-11-16T22:42:01.005 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.005 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.006 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T22:42:01.006 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.644894+0000", 2020-11-16T22:42:01.006 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.281913+0000" 2020-11-16T22:42:01.006 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.006 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.006 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:01.006 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.205785+0000", 2020-11-16T22:42:01.007 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.281967+0000" 2020-11-16T22:42:01.007 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.007 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.007 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:01.007 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:52.205771+0000", 2020-11-16T22:42:01.007 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.281977+0000" 2020-11-16T22:42:01.007 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.008 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.008 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.008 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.008 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "87", 2020-11-16T22:42:01.008 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.008 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.008 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:01.008 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.281988+0000", 2020-11-16T22:42:01.009 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.282587+0000" 2020-11-16T22:42:01.009 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.009 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.009 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.009 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.009 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "105", 2020-11-16T22:42:01.009 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.009 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.010 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:01.010 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.282600+0000", 2020-11-16T22:42:01.010 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.282637+0000" 2020-11-16T22:42:01.010 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.010 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.010 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:01.010 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.282662+0000", 2020-11-16T22:42:01.011 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.400361+0000" 2020-11-16T22:42:01.011 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.011 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.011 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:01.011 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.400362+0000", 2020-11-16T22:42:01.011 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.400502+0000" 2020-11-16T22:42:01.011 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.011 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.012 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T22:42:01.012 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.400503+0000", 2020-11-16T22:42:01.012 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.400540+0000" 2020-11-16T22:42:01.012 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.012 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.012 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T22:42:01.012 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.400540+0000", 2020-11-16T22:42:01.013 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.617084+0000" 2020-11-16T22:42:01.013 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.013 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.013 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:01.013 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.282650+0000", 2020-11-16T22:42:01.013 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.617101+0000" 2020-11-16T22:42:01.013 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.013 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.014 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T22:42:01.014 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.617230+0000", 2020-11-16T22:42:01.014 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.658942+0000" 2020-11-16T22:42:01.014 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.014 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.014 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T22:42:01.014 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.658948+0000", 2020-11-16T22:42:01.015 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:29:28.659035+0000" 2020-11-16T22:42:01.015 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.015 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.015 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T22:42:01.015 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.659036+0000", 2020-11-16T22:42:01.015 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:44.984312+0000" 2020-11-16T22:42:01.015 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.015 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.016 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T22:42:01.016 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.617101+0000", 2020-11-16T22:42:01.016 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:44.984378+0000" 2020-11-16T22:42:01.016 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.016 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.016 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:01.016 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.282638+0000", 2020-11-16T22:42:01.017 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:44.984578+0000" 2020-11-16T22:42:01.017 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.017 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.017 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:01.017 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:29:28.282587+0000", 2020-11-16T22:42:01.017 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:44.984589+0000" 2020-11-16T22:42:01.017 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.017 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.018 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.018 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.018 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "106", 2020-11-16T22:42:01.018 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.018 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.018 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:01.018 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:44.984590+0000", 2020-11-16T22:42:01.019 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:44.985444+0000" 2020-11-16T22:42:01.019 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.019 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.019 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.019 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.019 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "107", 2020-11-16T22:42:01.019 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.019 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.020 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:01.020 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:44.985458+0000", 2020-11-16T22:42:01.020 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:44.985495+0000" 2020-11-16T22:42:01.020 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.020 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.020 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:01.020 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:44.985520+0000", 2020-11-16T22:42:01.020 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:44.986979+0000" 2020-11-16T22:42:01.021 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.021 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.021 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:01.021 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:44.986980+0000", 2020-11-16T22:42:01.021 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:44.987122+0000" 2020-11-16T22:42:01.021 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.021 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.022 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:01.022 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:44.985507+0000", 2020-11-16T22:42:01.022 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:44.987146+0000" 2020-11-16T22:42:01.022 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.022 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.022 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/WaitActingChange", 2020-11-16T22:42:01.022 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:44.987146+0000", 2020-11-16T22:42:01.022 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:45.209891+0000" 2020-11-16T22:42:01.023 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.023 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.023 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:01.023 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:44.985497+0000", 2020-11-16T22:42:01.023 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:45.209906+0000" 2020-11-16T22:42:01.023 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.023 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.024 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:01.024 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:44.985444+0000", 2020-11-16T22:42:01.024 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:45.209910+0000" 2020-11-16T22:42:01.024 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.024 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.024 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.024 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.024 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "107", 2020-11-16T22:42:01.025 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.025 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.025 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:01.025 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.209910+0000", 2020-11-16T22:42:01.025 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:45.210108+0000" 2020-11-16T22:42:01.025 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.025 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.026 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.026 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.026 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "117", 2020-11-16T22:42:01.026 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.026 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.026 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:01.026 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.210112+0000", 2020-11-16T22:42:01.026 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:45.210127+0000" 2020-11-16T22:42:01.027 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.027 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.027 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:01.027 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.210138+0000", 2020-11-16T22:42:01.027 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:45.310545+0000" 2020-11-16T22:42:01.027 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.027 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.027 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:01.028 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.310546+0000", 2020-11-16T22:42:01.028 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:45.310675+0000" 2020-11-16T22:42:01.028 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.028 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.028 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T22:42:01.028 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.310676+0000", 2020-11-16T22:42:01.028 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:45.310704+0000" 2020-11-16T22:42:01.029 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.029 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.029 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T22:42:01.029 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.310704+0000", 2020-11-16T22:42:01.029 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:45.528193+0000" 2020-11-16T22:42:01.029 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.029 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.029 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:01.030 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.210134+0000", 2020-11-16T22:42:01.030 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:45.528207+0000" 2020-11-16T22:42:01.030 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.030 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.030 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T22:42:01.030 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.529263+0000", 2020-11-16T22:42:01.030 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:45.855265+0000" 2020-11-16T22:42:01.031 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.031 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.031 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2020-11-16T22:42:01.031 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.855266+0000", 2020-11-16T22:42:01.031 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:55.888997+0000" 2020-11-16T22:42:01.031 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.031 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.031 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2020-11-16T22:42:01.032 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:55.888998+0000", 2020-11-16T22:42:01.032 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:55.889639+0000" 2020-11-16T22:42:01.032 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.032 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.032 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovering", 2020-11-16T22:42:01.032 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:55.889640+0000", 2020-11-16T22:42:01.032 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:36:08.477733+0000" 2020-11-16T22:42:01.033 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.033 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.033 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/NotRecovering", 2020-11-16T22:42:01.033 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:36:08.477734+0000", 2020-11-16T22:42:01.033 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:36:08.477938+0000" 2020-11-16T22:42:01.033 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.033 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.033 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2020-11-16T22:42:01.034 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:36:08.477939+0000", 2020-11-16T22:42:01.034 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:39:57.026238+0000" 2020-11-16T22:42:01.034 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.034 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.034 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2020-11-16T22:42:01.034 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:39:57.026238+0000", 2020-11-16T22:42:01.034 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:39:57.026769+0000" 2020-11-16T22:42:01.035 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.035 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.035 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovering", 2020-11-16T22:42:01.035 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:39:57.026770+0000", 2020-11-16T22:42:01.035 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:41:54.487475+0000" 2020-11-16T22:42:01.035 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.035 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.035 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T22:42:01.036 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:41:54.487476+0000", 2020-11-16T22:42:01.036 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:41:54.729209+0000" 2020-11-16T22:42:01.036 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.036 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.036 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T22:42:01.036 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.528207+0000", 2020-11-16T22:42:01.036 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:41:54.729265+0000" 2020-11-16T22:42:01.036 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.037 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.037 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:01.037 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.210128+0000", 2020-11-16T22:42:01.037 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:41:54.729318+0000" 2020-11-16T22:42:01.037 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.037 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.037 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:01.038 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:45.210108+0000", 2020-11-16T22:42:01.038 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:41:54.729337+0000" 2020-11-16T22:42:01.038 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.038 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.038 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.038 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.038 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "117", 2020-11-16T22:42:01.038 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.039 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.039 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:01.039 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:41:54.729339+0000", 2020-11-16T22:42:01.039 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:41:54.730192+0000" 2020-11-16T22:42:01.039 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.039 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.039 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.040 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.040 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.040 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.040 INFO:tasks.workunit.client.0.smithi189.stdout: "pg": "1.0", 2020-11-16T22:42:01.040 INFO:tasks.workunit.client.0.smithi189.stdout: "currently": "Started/Primary/Active/Clean", 2020-11-16T22:42:01.040 INFO:tasks.workunit.client.0.smithi189.stdout: "history": [ 2020-11-16T22:42:01.041 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.041 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "24", 2020-11-16T22:42:01.041 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.041 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.042 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Initial", 2020-11-16T22:42:01.042 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.643154+0000", 2020-11-16T22:42:01.042 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:45.643291+0000" 2020-11-16T22:42:01.043 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.043 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.043 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.043 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.044 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "24", 2020-11-16T22:42:01.044 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.044 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.045 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:01.045 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.643292+0000", 2020-11-16T22:42:01.045 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:45.643324+0000" 2020-11-16T22:42:01.045 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.045 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.046 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.046 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.046 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "27", 2020-11-16T22:42:01.046 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.046 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.046 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:01.046 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.643328+0000", 2020-11-16T22:42:01.047 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:45.643337+0000" 2020-11-16T22:42:01.047 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.047 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.047 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:01.047 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.643344+0000", 2020-11-16T22:42:01.047 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:45.792700+0000" 2020-11-16T22:42:01.047 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.048 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.048 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:01.048 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.792701+0000", 2020-11-16T22:42:01.048 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:45.792915+0000" 2020-11-16T22:42:01.048 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.048 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.048 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T22:42:01.049 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.792916+0000", 2020-11-16T22:42:01.049 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:45.792949+0000" 2020-11-16T22:42:01.049 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.049 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.049 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T22:42:01.049 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.792950+0000", 2020-11-16T22:42:01.049 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:45.951229+0000" 2020-11-16T22:42:01.049 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.050 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.050 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:01.050 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.643341+0000", 2020-11-16T22:42:01.050 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:45.951257+0000" 2020-11-16T22:42:01.050 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.050 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.050 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T22:42:01.051 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.951699+0000", 2020-11-16T22:42:01.051 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:46.679873+0000" 2020-11-16T22:42:01.051 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.051 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.051 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T22:42:01.051 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:46.679874+0000", 2020-11-16T22:42:01.051 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:46.680085+0000" 2020-11-16T22:42:01.051 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.052 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.052 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T22:42:01.052 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:46.680086+0000", 2020-11-16T22:42:01.052 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:47.306054+0000" 2020-11-16T22:42:01.052 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.052 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.052 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T22:42:01.053 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.951259+0000", 2020-11-16T22:42:01.053 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:47.306083+0000" 2020-11-16T22:42:01.053 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.053 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.053 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:01.053 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.643338+0000", 2020-11-16T22:42:01.053 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:47.306126+0000" 2020-11-16T22:42:01.053 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.054 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.054 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:01.054 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:45.643325+0000", 2020-11-16T22:42:01.054 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:47.306130+0000" 2020-11-16T22:42:01.054 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.054 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.054 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.055 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.055 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "27", 2020-11-16T22:42:01.055 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.055 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.055 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:01.055 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:47.306131+0000", 2020-11-16T22:42:01.055 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:47.306366+0000" 2020-11-16T22:42:01.056 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.056 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.056 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.056 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.056 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "101", 2020-11-16T22:42:01.056 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.056 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.056 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:01.057 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:47.306370+0000", 2020-11-16T22:42:01.057 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:47.306380+0000" 2020-11-16T22:42:01.057 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.057 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.057 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:01.057 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:47.306389+0000", 2020-11-16T22:42:01.057 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:47.461705+0000" 2020-11-16T22:42:01.058 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.058 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.058 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:01.058 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:47.461706+0000", 2020-11-16T22:42:01.058 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:47.461835+0000" 2020-11-16T22:42:01.058 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.058 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.058 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T22:42:01.059 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:47.461836+0000", 2020-11-16T22:42:01.059 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:47.461862+0000" 2020-11-16T22:42:01.059 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.059 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.059 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T22:42:01.059 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:47.461863+0000", 2020-11-16T22:42:01.059 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:47.833686+0000" 2020-11-16T22:42:01.059 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.060 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.060 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:01.060 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:47.306386+0000", 2020-11-16T22:42:01.060 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:47.833702+0000" 2020-11-16T22:42:01.060 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.060 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.060 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T22:42:01.061 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:47.833987+0000", 2020-11-16T22:42:01.061 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:48.277796+0000" 2020-11-16T22:42:01.061 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.061 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.061 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T22:42:01.061 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:48.277797+0000", 2020-11-16T22:42:01.061 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:28:48.277878+0000" 2020-11-16T22:42:01.061 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.062 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.062 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Clean", 2020-11-16T22:42:01.062 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:48.277878+0000", 2020-11-16T22:42:01.062 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:34.736436+0000" 2020-11-16T22:42:01.062 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.062 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.062 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T22:42:01.063 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:47.833703+0000", 2020-11-16T22:42:01.063 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:34.736498+0000" 2020-11-16T22:42:01.063 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.063 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.063 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:01.063 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:47.306380+0000", 2020-11-16T22:42:01.063 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:34.736706+0000" 2020-11-16T22:42:01.063 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.064 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.064 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:01.064 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:28:47.306367+0000", 2020-11-16T22:42:01.064 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:34.736720+0000" 2020-11-16T22:42:01.064 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.064 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.064 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.065 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.065 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "101", 2020-11-16T22:42:01.065 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.065 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.065 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:01.065 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:34.736721+0000", 2020-11-16T22:42:01.065 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:34.737530+0000" 2020-11-16T22:42:01.065 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.066 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.066 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.066 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.066 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "103", 2020-11-16T22:42:01.066 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.066 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.066 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:01.067 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:34.737544+0000", 2020-11-16T22:42:01.067 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:34.737585+0000" 2020-11-16T22:42:01.067 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.067 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.067 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:01.067 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:34.737612+0000", 2020-11-16T22:42:01.067 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:34.879701+0000" 2020-11-16T22:42:01.067 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.068 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.068 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:01.068 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:34.879703+0000", 2020-11-16T22:42:01.068 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:34.880047+0000" 2020-11-16T22:42:01.068 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.068 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.068 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:01.068 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:34.737599+0000", 2020-11-16T22:42:01.069 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:34.880098+0000" 2020-11-16T22:42:01.069 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.069 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.069 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/WaitActingChange", 2020-11-16T22:42:01.069 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:34.880100+0000", 2020-11-16T22:42:01.069 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:35.256100+0000" 2020-11-16T22:42:01.069 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.070 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.070 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:01.070 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:34.737586+0000", 2020-11-16T22:42:01.070 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:35.256147+0000" 2020-11-16T22:42:01.070 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.070 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.070 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:01.070 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:34.737531+0000", 2020-11-16T22:42:01.071 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:35.256159+0000" 2020-11-16T22:42:01.071 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.071 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.071 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.071 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.071 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "103", 2020-11-16T22:42:01.071 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.072 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.072 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:01.072 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.256160+0000", 2020-11-16T22:42:01.072 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:35.256757+0000" 2020-11-16T22:42:01.072 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.072 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.072 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.072 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.073 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "115", 2020-11-16T22:42:01.073 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.073 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.073 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Start", 2020-11-16T22:42:01.073 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.256772+0000", 2020-11-16T22:42:01.073 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:35.256816+0000" 2020-11-16T22:42:01.073 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.073 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.074 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-11-16T22:42:01.074 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.256841+0000", 2020-11-16T22:42:01.074 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:35.412634+0000" 2020-11-16T22:42:01.074 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.074 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.074 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetLog", 2020-11-16T22:42:01.074 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.412636+0000", 2020-11-16T22:42:01.075 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:35.412869+0000" 2020-11-16T22:42:01.075 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.075 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.075 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-11-16T22:42:01.075 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.412870+0000", 2020-11-16T22:42:01.075 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:35.412922+0000" 2020-11-16T22:42:01.075 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.075 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.076 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-11-16T22:42:01.076 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.412923+0000", 2020-11-16T22:42:01.076 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:35.641461+0000" 2020-11-16T22:42:01.076 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.076 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.076 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Peering", 2020-11-16T22:42:01.076 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.256827+0000", 2020-11-16T22:42:01.076 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:35.641492+0000" 2020-11-16T22:42:01.077 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.077 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.077 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Activating", 2020-11-16T22:42:01.077 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.644376+0000", 2020-11-16T22:42:01.077 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:35:35.986588+0000" 2020-11-16T22:42:01.077 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.077 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.078 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2020-11-16T22:42:01.078 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.986591+0000", 2020-11-16T22:42:01.078 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:38:05.029252+0000" 2020-11-16T22:42:01.078 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.078 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.078 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2020-11-16T22:42:01.078 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:38:05.029253+0000", 2020-11-16T22:42:01.078 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:38:05.029923+0000" 2020-11-16T22:42:01.079 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.079 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.079 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovering", 2020-11-16T22:42:01.079 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:38:05.029928+0000", 2020-11-16T22:42:01.079 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:39:57.025650+0000" 2020-11-16T22:42:01.079 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.080 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.080 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active/Recovered", 2020-11-16T22:42:01.080 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:39:57.025652+0000", 2020-11-16T22:42:01.080 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:39:57.419748+0000" 2020-11-16T22:42:01.080 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.080 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.080 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary/Active", 2020-11-16T22:42:01.080 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.641493+0000", 2020-11-16T22:42:01.081 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:39:57.419772+0000" 2020-11-16T22:42:01.081 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.081 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.081 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started/Primary", 2020-11-16T22:42:01.081 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.256817+0000", 2020-11-16T22:42:01.081 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:39:57.419790+0000" 2020-11-16T22:42:01.081 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.082 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.082 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Started", 2020-11-16T22:42:01.082 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:35:35.256758+0000", 2020-11-16T22:42:01.082 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:39:57.419816+0000" 2020-11-16T22:42:01.082 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.082 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.082 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2020-11-16T22:42:01.082 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.083 INFO:tasks.workunit.client.0.smithi189.stdout: "epoch": "115", 2020-11-16T22:42:01.083 INFO:tasks.workunit.client.0.smithi189.stdout: "states": [ 2020-11-16T22:42:01.083 INFO:tasks.workunit.client.0.smithi189.stdout: { 2020-11-16T22:42:01.083 INFO:tasks.workunit.client.0.smithi189.stdout: "state": "Reset", 2020-11-16T22:42:01.083 INFO:tasks.workunit.client.0.smithi189.stdout: "enter": "2020-11-16T22:39:57.419817+0000", 2020-11-16T22:42:01.083 INFO:tasks.workunit.client.0.smithi189.stdout: "exit": "2020-11-16T22:39:57.420168+0000" 2020-11-16T22:42:01.083 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.084 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.084 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.084 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.084 INFO:tasks.workunit.client.0.smithi189.stdout: } 2020-11-16T22:42:01.084 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2020-11-16T22:42:01.084 INFO:tasks.workunit.client.0.smithi189.stdout:} 2020-11-16T22:42:01.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:299: TEST_recovery_priority: '[' 0 '!=' 0 ']' 2020-11-16T22:42:01.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:303: TEST_recovery_priority: echo TEST PASSED 2020-11-16T22:42:01.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:306: TEST_recovery_priority: delete_pool test1 2020-11-16T22:42:01.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2020-11-16T22:42:01.086 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-11-16T22:42:01.086 INFO:tasks.workunit.client.0.smithi189.stdout:TEST PASSED 2020-11-16T22:42:01.484 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' does not exist 2020-11-16T22:42:01.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:307: TEST_recovery_priority: delete_pool test3 2020-11-16T22:42:01.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2020-11-16T22:42:01.494 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-11-16T22:42:02.098 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test3' does not exist 2020-11-16T22:42:02.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:308: TEST_recovery_priority: delete_pool test7 2020-11-16T22:42:02.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test7 2020-11-16T22:42:02.108 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-11-16T22:42:02.726 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test7' does not exist 2020-11-16T22:42:02.735 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:309: TEST_recovery_priority: kill_daemons td/osd-recovery-prio 2020-11-16T22:42:02.737 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:42:02.737 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:42:02.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:42:02.737 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:42:02.738 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:42:03.056 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:42:03.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:310: TEST_recovery_priority: return 0 2020-11-16T22:42:03.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: teardown td/osd-recovery-prio 2020-11-16T22:42:03.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2020-11-16T22:42:03.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:42:03.057 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2020-11-16T22:42:03.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:42:03.058 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:42:03.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:42:03.058 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:42:03.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:42:03.072 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:42:03.072 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:42:03.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:42:03.074 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:42:03.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:42:03.075 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:42:03.076 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:42:03.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:42:03.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:42:03.077 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:42:03.078 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:42:03.079 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:42:03.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:42:03.080 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2020-11-16T22:42:03.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:42:03.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:42:03.143 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:42:03.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.679603 2020-11-16T22:42:03.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:42:03.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:42:03.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-11-16T22:42:03.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-recovery-prio 0 2020-11-16T22:42:03.145 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2020-11-16T22:42:03.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-11-16T22:42:03.146 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2020-11-16T22:42:03.146 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:42:03.147 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:42:03.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:42:03.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:42:03.147 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:42:03.150 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:42:03.151 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:42:03.151 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:42:03.152 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:42:03.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:42:03.154 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:42:03.155 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:42:03.155 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:42:03.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:42:03.156 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:42:03.157 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:42:03.157 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:42:03.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-11-16T22:42:03.159 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2020-11-16T22:42:03.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:42:03.161 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:42:03.162 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679603 2020-11-16T22:42:03.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.679603 2020-11-16T22:42:03.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:42:03.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:42:03.163 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-11-16T22:42:03.165 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-16T22:42:03.165 INFO:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-16T22:42:03.221 INFO:tasks.workunit:Running workunit osd/osd-recovery-space.sh... 2020-11-16T22:42:03.222 INFO:teuthology.orchestra.run.smithi189:workunit test osd/osd-recovery-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=911cc35794d871071a8bebfb0919482263b1d18b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2020-11-16T22:42:03.283 INFO:tasks.workunit.client.0.smithi189.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-11-16T22:42:03.298 INFO:tasks.workunit.client.0.smithi189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-11-16T22:42:03.298 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:42:03.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2020-11-16T22:42:03.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-11-16T22:42:03.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-11-16T22:42:03.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-11-16T22:42:03.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-11-16T22:42:03.299 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-11-16T22:42:03.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-11-16T22:42:03.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-recovery-space 2020-11-16T22:42:03.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:21: run: local dir=td/osd-recovery-space 2020-11-16T22:42:03.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:22: run: shift 2020-11-16T22:42:03.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: export CEPH_MON=127.0.0.1:7221 2020-11-16T22:42:03.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: CEPH_MON=127.0.0.1:7221 2020-11-16T22:42:03.300 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:25: run: export CEPH_ARGS 2020-11-16T22:42:03.300 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: uuidgen 2020-11-16T22:42:03.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: CEPH_ARGS+='--fsid=774cd8aa-52d0-4a74-bd4f-e53178324a08 --auth-supported=none ' 2020-11-16T22:42:03.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7221 ' 2020-11-16T22:42:03.301 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:28: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2020-11-16T22:42:03.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: export objects=600 2020-11-16T22:42:03.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: objects=600 2020-11-16T22:42:03.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: export poolprefix=test 2020-11-16T22:42:03.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: poolprefix=test 2020-11-16T22:42:03.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:32: run: set 2020-11-16T22:42:03.302 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:32: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-11-16T22:42:03.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:32: run: local funcs=TEST_recovery_test_simple 2020-11-16T22:42:03.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: for func in $funcs 2020-11-16T22:42:03.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:34: run: setup td/osd-recovery-space 2020-11-16T22:42:03.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-space 2020-11-16T22:42:03.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-space 2020-11-16T22:42:03.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2020-11-16T22:42:03.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-11-16T22:42:03.305 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2020-11-16T22:42:03.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-11-16T22:42:03.306 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-11-16T22:42:03.306 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-11-16T22:42:03.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-11-16T22:42:03.307 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-11-16T22:42:03.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-11-16T22:42:03.309 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-11-16T22:42:03.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-11-16T22:42:03.310 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-11-16T22:42:03.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-11-16T22:42:03.312 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-11-16T22:42:03.313 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-11-16T22:42:03.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:42:03.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-11-16T22:42:03.314 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-11-16T22:42:03.315 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:42:03.315 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-11-16T22:42:03.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-11-16T22:42:03.317 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2020-11-16T22:42:03.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-11-16T22:42:03.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:42:03.319 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.737250 2020-11-16T22:42:03.319 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.737250 2020-11-16T22:42:03.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-11-16T22:42:03.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-11-16T22:42:03.320 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-space 2020-11-16T22:42:03.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-11-16T22:42:03.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:42:03.322 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.737250 2020-11-16T22:42:03.323 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.737250 2020-11-16T22:42:03.325 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-11-16T22:42:03.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-11-16T22:42:03.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-11-16T22:42:03.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-11-16T22:42:03.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-space 1' TERM HUP INT 2020-11-16T22:42:03.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:35: run: TEST_recovery_test_simple td/osd-recovery-space 2020-11-16T22:42:03.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:88: TEST_recovery_test_simple: local dir=td/osd-recovery-space 2020-11-16T22:42:03.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:89: TEST_recovery_test_simple: local pools=1 2020-11-16T22:42:03.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:90: TEST_recovery_test_simple: local OSDS=2 2020-11-16T22:42:03.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:92: TEST_recovery_test_simple: run_mon td/osd-recovery-space a 2020-11-16T22:42:03.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-space 2020-11-16T22:42:03.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-11-16T22:42:03.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-11-16T22:42:03.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-11-16T22:42:03.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-space/a 2020-11-16T22:42:03.328 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-space/a --run-dir=td/osd-recovery-space 2020-11-16T22:42:03.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-11-16T22:42:03.625 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:42:03.626 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:42:03.626 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:42:03.626 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:42:03.627 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.737250 2020-11-16T22:42:03.627 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.737250/$cluster-$name.asok' 2020-11-16T22:42:03.627 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-space/a '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.737250/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-space/log --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-11-16T22:42:03.830 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-11-16T22:42:03.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-11-16T22:42:03.831 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:42:03.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:42:03.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-11-16T22:42:03.832 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-11-16T22:42:03.832 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:42:03.833 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:42:03.833 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:42:03.833 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:42:03.833 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:42:03.834 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.737250 2020-11-16T22:42:03.834 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.737250/ceph-mon.a.asok 2020-11-16T22:42:03.834 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:42:03.835 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.737250/ceph-mon.a.asok config get fsid 2020-11-16T22:42:03.941 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-11-16T22:42:03.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-11-16T22:42:03.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-11-16T22:42:03.942 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-11-16T22:42:03.943 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-11-16T22:42:03.943 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-11-16T22:42:03.943 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-11-16T22:42:03.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-11-16T22:42:03.944 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-11-16T22:42:03.944 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:42:03.944 INFO:tasks.workunit.client.0.smithi189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.737250 2020-11-16T22:42:03.944 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.737250/ceph-mon.a.asok 2020-11-16T22:42:03.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-11-16T22:42:03.944 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.737250/ceph-mon.a.asok config get mon_host 2020-11-16T22:42:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:93: TEST_recovery_test_simple: run_mgr td/osd-recovery-space x 2020-11-16T22:42:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-space 2020-11-16T22:42:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-11-16T22:42:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-11-16T22:42:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-11-16T22:42:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-space/x 2020-11-16T22:42:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-11-16T22:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-11-16T22:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.737250 2020-11-16T22:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.737250/$cluster-$name.asok' 2020-11-16T22:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-space/x '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.737250/$cluster-$name.asok' --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-11-16T22:42:04.990 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:94: TEST_recovery_test_simple: export CEPH_ARGS 2020-11-16T22:42:04.991 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: expr 2 - 1 2020-11-16T22:42:04.993 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: seq 0 1 2020-11-16T22:42:04.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T22:42:04.994 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:98: TEST_recovery_test_simple: run_osd td/osd-recovery-space 0 2020-11-16T22:42:04.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-space 2020-11-16T22:42:04.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:42:04.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-11-16T22:42:04.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:42:04.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-space/0 2020-11-16T22:42:04.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=774cd8aa-52d0-4a74-bd4f-e53178324a08 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 ' 2020-11-16T22:42:04.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:42:04.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:42:04.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:42:04.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/0' 2020-11-16T22:42:04.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/0/journal' 2020-11-16T22:42:04.997 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:42:04.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:42:04.998 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2020-11-16T22:42:04.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:42:04.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:42:04.998 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:42:04.998 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:42:04.998 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:42:04.999 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.737250 2020-11-16T22:42:04.999 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.737250/$cluster-$name.asok' 2020-11-16T22:42:05.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.737250/$cluster-$name.asok' 2020-11-16T22:42:05.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:42:05.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:42:05.000 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:42:05.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2020-11-16T22:42:05.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2020-11-16T22:42:05.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:42:05.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:42:05.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:42:05.001 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:42:05.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:42:05.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-space/0 2020-11-16T22:42:05.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:42:05.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=bae8cdce-50e6-4038-8484-c8b19a88e5f7 2020-11-16T22:42:05.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 bae8cdce-50e6-4038-8484-c8b19a88e5f7' 2020-11-16T22:42:05.002 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:42:05.003 INFO:tasks.workunit.client.0.smithi189.stdout:add osd0 bae8cdce-50e6-4038-8484-c8b19a88e5f7 2020-11-16T22:42:05.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA9ALNfo+QsAhAAuRh7fHhO/wcEpH14C/DnYg== 2020-11-16T22:42:05.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA9ALNfo+QsAhAAuRh7fHhO/wcEpH14C/DnYg=="}' 2020-11-16T22:42:05.023 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new bae8cdce-50e6-4038-8484-c8b19a88e5f7 -i td/osd-recovery-space/0/new.json 2020-11-16T22:42:05.370 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:42:05.380 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-space/0/new.json 2020-11-16T22:42:05.382 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=774cd8aa-52d0-4a74-bd4f-e53178324a08 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.737250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA9ALNfo+QsAhAAuRh7fHhO/wcEpH14C/DnYg== --osd-uuid bae8cdce-50e6-4038-8484-c8b19a88e5f7 2020-11-16T22:42:05.404 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:05.413+0000 7f8908d4fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:05.406 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:05.417+0000 7f8908d4fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:05.407 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:05.417+0000 7f8908d4fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:05.408 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:05.417+0000 7f8908d4fd80 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2020-11-16T22:42:05.408 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:05.417+0000 7f8908d4fd80 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2020-11-16T22:42:05.408 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:05.417+0000 7f8908d4fd80 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2020-11-16T22:42:05.446 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:05.457+0000 7f8908d4fd80 -1 bluestore(td/osd-recovery-space/0) _read_fsid unparsable uuid 2020-11-16T22:42:09.982 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-space/0/keyring 2020-11-16T22:42:09.983 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:42:09.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-11-16T22:42:09.984 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:42:09.985 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd0 key to auth repository 2020-11-16T22:42:10.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-11-16T22:42:10.463 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.0 2020-11-16T22:42:10.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=774cd8aa-52d0-4a74-bd4f-e53178324a08 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.737250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:42:10.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:42:10.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:42:10.464 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:42:10.491 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:10.501+0000 7f10eb7e5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:10.494 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:10.505+0000 7f10eb7e5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:10.495 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:10.505+0000 7f10eb7e5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:10.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-11-16T22:42:10.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:42:10.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-11-16T22:42:10.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:42:10.837 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:42:10.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:42:10.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:42:10.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:42:10.838 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:42:10.838 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:42:11.039 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:11.049+0000 7f10eb7e5d80 -1 Falling back to public interface 2020-11-16T22:42:11.245 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:42:11.893 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:11.901+0000 7f10eb7e5d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-16T22:42:12.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:42:12.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:42:12.247 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:42:12.247 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:42:12.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:42:12.248 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-11-16T22:42:12.656 INFO:tasks.workunit.client.0.smithi189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/738106,v1:127.0.0.1:6803/738106] [v2:127.0.0.1:6804/738106,v1:127.0.0.1:6805/738106] exists,up bae8cdce-50e6-4038-8484-c8b19a88e5f7 2020-11-16T22:42:12.656 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:42:12.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:42:12.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:42:12.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-11-16T22:42:12.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:98: TEST_recovery_test_simple: run_osd td/osd-recovery-space 1 2020-11-16T22:42:12.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-space 2020-11-16T22:42:12.657 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-11-16T22:42:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-11-16T22:42:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-11-16T22:42:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-space/1 2020-11-16T22:42:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=774cd8aa-52d0-4a74-bd4f-e53178324a08 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 ' 2020-11-16T22:42:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-11-16T22:42:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-11-16T22:42:12.658 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-11-16T22:42:12.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/1' 2020-11-16T22:42:12.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/1/journal' 2020-11-16T22:42:12.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-11-16T22:42:12.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-11-16T22:42:12.659 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2020-11-16T22:42:12.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-11-16T22:42:12.659 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-11-16T22:42:12.660 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-11-16T22:42:12.661 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-11-16T22:42:12.661 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-11-16T22:42:12.661 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.737250 2020-11-16T22:42:12.662 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.737250/$cluster-$name.asok' 2020-11-16T22:42:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.737250/$cluster-$name.asok' 2020-11-16T22:42:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-11-16T22:42:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-11-16T22:42:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-11-16T22:42:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2020-11-16T22:42:12.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2020-11-16T22:42:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-11-16T22:42:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-11-16T22:42:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:42:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-11-16T22:42:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-11-16T22:42:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-space/1 2020-11-16T22:42:12.663 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-11-16T22:42:12.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3370cfe0-c260-4585-a627-f87d65786b8c 2020-11-16T22:42:12.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 3370cfe0-c260-4585-a627-f87d65786b8c' 2020-11-16T22:42:12.664 INFO:tasks.workunit.client.0.smithi189.stdout:add osd1 3370cfe0-c260-4585-a627-f87d65786b8c 2020-11-16T22:42:12.664 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-11-16T22:42:12.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBEALNfh2liKRAATVW/IXLaZc86By6I+cIV/w== 2020-11-16T22:42:12.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBEALNfh2liKRAATVW/IXLaZc86By6I+cIV/w=="}' 2020-11-16T22:42:12.680 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3370cfe0-c260-4585-a627-f87d65786b8c -i td/osd-recovery-space/1/new.json 2020-11-16T22:42:13.176 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:42:13.188 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-space/1/new.json 2020-11-16T22:42:13.190 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=774cd8aa-52d0-4a74-bd4f-e53178324a08 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.737250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBEALNfh2liKRAATVW/IXLaZc86By6I+cIV/w== --osd-uuid 3370cfe0-c260-4585-a627-f87d65786b8c 2020-11-16T22:42:13.218 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:13.229+0000 7f586af94d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:13.220 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:13.229+0000 7f586af94d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:13.221 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:13.233+0000 7f586af94d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:13.222 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:13.233+0000 7f586af94d80 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2020-11-16T22:42:13.222 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:13.233+0000 7f586af94d80 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2020-11-16T22:42:13.222 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:13.233+0000 7f586af94d80 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2020-11-16T22:42:13.298 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:13.309+0000 7f586af94d80 -1 bluestore(td/osd-recovery-space/1) _read_fsid unparsable uuid 2020-11-16T22:42:17.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-space/1/keyring 2020-11-16T22:42:17.978 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-11-16T22:42:17.979 INFO:tasks.workunit.client.0.smithi189.stdout:adding osd1 key to auth repository 2020-11-16T22:42:17.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-11-16T22:42:17.980 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-11-16T22:42:18.477 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-11-16T22:42:18.478 INFO:tasks.workunit.client.0.smithi189.stdout:start osd.1 2020-11-16T22:42:18.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-11-16T22:42:18.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=774cd8aa-52d0-4a74-bd4f-e53178324a08 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.737250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-11-16T22:42:18.478 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-11-16T22:42:18.479 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-11-16T22:42:18.504 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:18.513+0000 7f79a3064d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:18.507 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:18.517+0000 7f79a3064d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:18.508 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:18.517+0000 7f79a3064d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-11-16T22:42:18.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-11-16T22:42:18.954 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-11-16T22:42:18.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-11-16T22:42:18.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-11-16T22:42:18.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-11-16T22:42:18.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:42:18.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-11-16T22:42:18.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:42:18.955 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:42:18.956 INFO:tasks.workunit.client.0.smithi189.stdout:0 2020-11-16T22:42:19.070 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:19.081+0000 7f79a3064d80 -1 Falling back to public interface 2020-11-16T22:42:19.308 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-11-16T22:42:19.930 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-16T22:42:19.941+0000 7f79a3064d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-16T22:42:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-11-16T22:42:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-11-16T22:42:20.309 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-11-16T22:42:20.310 INFO:tasks.workunit.client.0.smithi189.stdout:1 2020-11-16T22:42:20.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-11-16T22:42:20.310 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-11-16T22:42:20.726 INFO:tasks.workunit.client.0.smithi189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/739100,v1:127.0.0.1:6811/739100] [v2:127.0.0.1:6812/739100,v1:127.0.0.1:6813/739100] exists,up 3370cfe0-c260-4585-a627-f87d65786b8c 2020-11-16T22:42:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-11-16T22:42:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-11-16T22:42:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-11-16T22:42:20.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:101: TEST_recovery_test_simple: ceph osd set-nearfull-ratio .40 2020-11-16T22:42:21.577 INFO:tasks.workunit.client.0.smithi189.stderr:osd set-nearfull-ratio 0.4 2020-11-16T22:42:21.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:102: TEST_recovery_test_simple: ceph osd set-backfillfull-ratio .45 2020-11-16T22:42:22.395 INFO:tasks.workunit.client.0.smithi189.stderr:osd set-backfillfull-ratio 0.45 2020-11-16T22:42:22.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: ceph osd set-full-ratio .50 2020-11-16T22:42:23.148 INFO:tasks.workunit.client.0.smithi189.stderr:osd set-full-ratio 0.5 2020-11-16T22:42:23.164 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:105: TEST_recovery_test_simple: seq 1 1 2020-11-16T22:42:23.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:105: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2020-11-16T22:42:23.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:107: TEST_recovery_test_simple: create_pool test1 1 1 2020-11-16T22:42:23.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2020-11-16T22:42:23.754 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'test1' already exists 2020-11-16T22:42:23.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-11-16T22:42:24.766 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:108: TEST_recovery_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2020-11-16T22:42:25.454 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 1 size to 1 2020-11-16T22:42:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:111: TEST_recovery_test_simple: wait_for_clean 2020-11-16T22:42:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-11-16T22:42:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-11-16T22:42:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-11-16T22:42:25.465 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-11-16T22:42:25.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-11-16T22:42:25.467 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-11-16T22:42:25.468 INFO:tasks.workunit.client.0.smithi189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-11-16T22:42:25.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-11-16T22:42:25.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-11-16T22:42:25.468 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-11-16T22:42:25.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-11-16T22:42:25.650 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-11-16T22:42:25.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-11-16T22:42:25.651 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-11-16T22:42:25.651 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-11-16T22:42:26.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-11-16T22:42:26.141 INFO:tasks.workunit.client.0.smithi189.stderr:1' 2020-11-16T22:42:26.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-11-16T22:42:26.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:42:26.142 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-11-16T22:42:26.302 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836485 2020-11-16T22:42:26.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836485 2020-11-16T22:42:26.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836485' 2020-11-16T22:42:26.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-11-16T22:42:26.304 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-11-16T22:42:26.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705668 2020-11-16T22:42:26.459 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705668 2020-11-16T22:42:26.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836485 1-38654705668' 2020-11-16T22:42:26.460 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:42:26.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836485 2020-11-16T22:42:26.462 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:42:26.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-11-16T22:42:26.466 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836485 2020-11-16T22:42:26.467 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:42:26.468 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836485 2020-11-16T22:42:26.469 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2020-11-16T22:42:26.469 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.0 seq 21474836485 2020-11-16T22:42:26.471 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-11-16T22:42:26.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836485 2020-11-16T22:42:26.851 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-11-16T22:42:26.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705668 2020-11-16T22:42:26.855 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-11-16T22:42:26.856 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-11-16T22:42:26.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705668 2020-11-16T22:42:26.859 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-11-16T22:42:26.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705668 2020-11-16T22:42:26.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2020-11-16T22:42:26.861 INFO:tasks.workunit.client.0.smithi189.stdout:waiting osd.1 seq 38654705668 2020-11-16T22:42:26.862 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-11-16T22:42:27.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705668 -lt 38654705668 2020-11-16T22:42:27.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-11-16T22:42:27.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:42:27.247 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:42:27.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2020-11-16T22:42:27.727 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-11-16T22:42:27.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-11-16T22:42:27.729 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-11-16T22:42:27.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-11-16T22:42:27.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-11-16T22:42:27.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-11-16T22:42:27.730 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-11-16T22:42:28.106 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2020-11-16T22:42:28.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-11-16T22:42:28.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-11-16T22:42:28.109 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-11-16T22:42:28.569 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2020-11-16T22:42:28.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-11-16T22:42:28.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-11-16T22:42:28.570 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:113: TEST_recovery_test_simple: dd if=/dev/urandom of=td/osd-recovery-space/datafile bs=1024 count=5 2020-11-16T22:42:28.570 INFO:tasks.workunit.client.0.smithi189.stderr:5+0 records in 2020-11-16T22:42:28.571 INFO:tasks.workunit.client.0.smithi189.stderr:5+0 records out 2020-11-16T22:42:28.571 INFO:tasks.workunit.client.0.smithi189.stderr:5120 bytes (5.1 kB, 5.0 KiB) copied, 0.000255842 s, 20.0 MB/s 2020-11-16T22:42:28.573 INFO:tasks.workunit.client.0.smithi189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: seq 1 600 2020-11-16T22:42:28.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:28.575 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj1 td/osd-recovery-space/datafile 2020-11-16T22:42:28.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:28.689 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj2 td/osd-recovery-space/datafile 2020-11-16T22:42:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj3 td/osd-recovery-space/datafile 2020-11-16T22:42:28.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:28.897 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj4 td/osd-recovery-space/datafile 2020-11-16T22:42:29.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:29.094 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj5 td/osd-recovery-space/datafile 2020-11-16T22:42:29.166 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:29.167 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj6 td/osd-recovery-space/datafile 2020-11-16T22:42:29.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:29.258 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj7 td/osd-recovery-space/datafile 2020-11-16T22:42:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj8 td/osd-recovery-space/datafile 2020-11-16T22:42:29.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:29.426 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj9 td/osd-recovery-space/datafile 2020-11-16T22:42:29.517 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:29.518 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj10 td/osd-recovery-space/datafile 2020-11-16T22:42:29.609 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:29.610 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj11 td/osd-recovery-space/datafile 2020-11-16T22:42:29.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:29.692 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj12 td/osd-recovery-space/datafile 2020-11-16T22:42:29.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:29.784 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj13 td/osd-recovery-space/datafile 2020-11-16T22:42:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj14 td/osd-recovery-space/datafile 2020-11-16T22:42:29.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:29.952 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj15 td/osd-recovery-space/datafile 2020-11-16T22:42:30.052 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:30.053 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj16 td/osd-recovery-space/datafile 2020-11-16T22:42:30.161 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:30.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj17 td/osd-recovery-space/datafile 2020-11-16T22:42:30.260 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:30.261 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj18 td/osd-recovery-space/datafile 2020-11-16T22:42:30.361 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:30.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj19 td/osd-recovery-space/datafile 2020-11-16T22:42:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:30.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj20 td/osd-recovery-space/datafile 2020-11-16T22:42:30.545 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:30.546 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj21 td/osd-recovery-space/datafile 2020-11-16T22:42:30.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:30.645 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj22 td/osd-recovery-space/datafile 2020-11-16T22:42:30.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:30.763 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj23 td/osd-recovery-space/datafile 2020-11-16T22:42:30.862 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:30.863 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj24 td/osd-recovery-space/datafile 2020-11-16T22:42:30.963 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:30.964 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj25 td/osd-recovery-space/datafile 2020-11-16T22:42:31.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:31.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj26 td/osd-recovery-space/datafile 2020-11-16T22:42:31.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:31.165 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj27 td/osd-recovery-space/datafile 2020-11-16T22:42:31.256 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:31.257 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj28 td/osd-recovery-space/datafile 2020-11-16T22:42:31.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:31.357 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj29 td/osd-recovery-space/datafile 2020-11-16T22:42:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj30 td/osd-recovery-space/datafile 2020-11-16T22:42:31.539 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:31.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj31 td/osd-recovery-space/datafile 2020-11-16T22:42:31.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:31.641 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj32 td/osd-recovery-space/datafile 2020-11-16T22:42:31.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:31.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj33 td/osd-recovery-space/datafile 2020-11-16T22:42:31.841 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:31.842 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj34 td/osd-recovery-space/datafile 2020-11-16T22:42:31.941 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:31.942 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj35 td/osd-recovery-space/datafile 2020-11-16T22:42:32.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:32.033 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj36 td/osd-recovery-space/datafile 2020-11-16T22:42:32.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:32.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj37 td/osd-recovery-space/datafile 2020-11-16T22:42:32.224 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:32.225 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj38 td/osd-recovery-space/datafile 2020-11-16T22:42:32.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:32.318 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj39 td/osd-recovery-space/datafile 2020-11-16T22:42:32.408 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:32.409 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj40 td/osd-recovery-space/datafile 2020-11-16T22:42:32.509 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:32.510 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj41 td/osd-recovery-space/datafile 2020-11-16T22:42:32.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:32.619 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj42 td/osd-recovery-space/datafile 2020-11-16T22:42:32.744 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:32.745 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj43 td/osd-recovery-space/datafile 2020-11-16T22:42:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:32.836 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj44 td/osd-recovery-space/datafile 2020-11-16T22:42:32.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:32.919 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj45 td/osd-recovery-space/datafile 2020-11-16T22:42:33.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:33.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj46 td/osd-recovery-space/datafile 2020-11-16T22:42:33.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:33.162 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj47 td/osd-recovery-space/datafile 2020-11-16T22:42:33.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:33.254 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj48 td/osd-recovery-space/datafile 2020-11-16T22:42:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:33.345 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj49 td/osd-recovery-space/datafile 2020-11-16T22:42:33.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:33.438 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj50 td/osd-recovery-space/datafile 2020-11-16T22:42:33.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:33.530 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj51 td/osd-recovery-space/datafile 2020-11-16T22:42:33.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:33.630 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj52 td/osd-recovery-space/datafile 2020-11-16T22:42:33.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:33.729 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj53 td/osd-recovery-space/datafile 2020-11-16T22:42:33.822 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:33.823 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj54 td/osd-recovery-space/datafile 2020-11-16T22:42:34.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj55 td/osd-recovery-space/datafile 2020-11-16T22:42:34.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:34.349 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj56 td/osd-recovery-space/datafile 2020-11-16T22:42:34.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:34.439 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj57 td/osd-recovery-space/datafile 2020-11-16T22:42:34.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:34.523 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj58 td/osd-recovery-space/datafile 2020-11-16T22:42:34.606 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:34.607 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj59 td/osd-recovery-space/datafile 2020-11-16T22:42:34.741 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:34.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj60 td/osd-recovery-space/datafile 2020-11-16T22:42:34.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:34.825 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj61 td/osd-recovery-space/datafile 2020-11-16T22:42:34.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:34.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj62 td/osd-recovery-space/datafile 2020-11-16T22:42:35.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:35.009 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj63 td/osd-recovery-space/datafile 2020-11-16T22:42:35.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:35.141 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj64 td/osd-recovery-space/datafile 2020-11-16T22:42:35.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:35.234 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj65 td/osd-recovery-space/datafile 2020-11-16T22:42:35.326 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:35.327 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj66 td/osd-recovery-space/datafile 2020-11-16T22:42:35.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:35.427 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj67 td/osd-recovery-space/datafile 2020-11-16T22:42:35.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:35.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj68 td/osd-recovery-space/datafile 2020-11-16T22:42:35.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:35.618 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj69 td/osd-recovery-space/datafile 2020-11-16T22:42:35.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:35.711 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj70 td/osd-recovery-space/datafile 2020-11-16T22:42:35.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:35.810 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj71 td/osd-recovery-space/datafile 2020-11-16T22:42:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:35.903 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj72 td/osd-recovery-space/datafile 2020-11-16T22:42:36.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:36.012 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj73 td/osd-recovery-space/datafile 2020-11-16T22:42:36.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:36.121 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj74 td/osd-recovery-space/datafile 2020-11-16T22:42:36.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:36.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj75 td/osd-recovery-space/datafile 2020-11-16T22:42:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:36.337 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj76 td/osd-recovery-space/datafile 2020-11-16T22:42:36.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:36.447 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj77 td/osd-recovery-space/datafile 2020-11-16T22:42:36.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:36.547 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj78 td/osd-recovery-space/datafile 2020-11-16T22:42:36.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:36.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj79 td/osd-recovery-space/datafile 2020-11-16T22:42:36.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:36.764 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj80 td/osd-recovery-space/datafile 2020-11-16T22:42:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:36.865 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj81 td/osd-recovery-space/datafile 2020-11-16T22:42:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:36.956 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj82 td/osd-recovery-space/datafile 2020-11-16T22:42:37.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:37.040 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj83 td/osd-recovery-space/datafile 2020-11-16T22:42:37.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:37.148 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj84 td/osd-recovery-space/datafile 2020-11-16T22:42:37.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:37.232 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj85 td/osd-recovery-space/datafile 2020-11-16T22:42:37.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:37.333 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj86 td/osd-recovery-space/datafile 2020-11-16T22:42:37.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:37.424 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj87 td/osd-recovery-space/datafile 2020-11-16T22:42:37.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:37.516 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj88 td/osd-recovery-space/datafile 2020-11-16T22:42:37.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:37.617 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj89 td/osd-recovery-space/datafile 2020-11-16T22:42:37.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:37.709 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj90 td/osd-recovery-space/datafile 2020-11-16T22:42:37.800 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:37.801 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj91 td/osd-recovery-space/datafile 2020-11-16T22:42:37.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:37.893 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj92 td/osd-recovery-space/datafile 2020-11-16T22:42:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:37.993 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj93 td/osd-recovery-space/datafile 2020-11-16T22:42:38.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:38.085 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj94 td/osd-recovery-space/datafile 2020-11-16T22:42:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:38.177 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj95 td/osd-recovery-space/datafile 2020-11-16T22:42:38.268 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:38.269 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj96 td/osd-recovery-space/datafile 2020-11-16T22:42:38.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:38.351 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj97 td/osd-recovery-space/datafile 2020-11-16T22:42:38.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:38.444 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj98 td/osd-recovery-space/datafile 2020-11-16T22:42:38.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:38.527 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj99 td/osd-recovery-space/datafile 2020-11-16T22:42:38.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:38.628 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj100 td/osd-recovery-space/datafile 2020-11-16T22:42:38.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:38.761 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj101 td/osd-recovery-space/datafile 2020-11-16T22:42:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:38.871 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj102 td/osd-recovery-space/datafile 2020-11-16T22:42:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:38.979 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj103 td/osd-recovery-space/datafile 2020-11-16T22:42:39.124 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:39.125 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj104 td/osd-recovery-space/datafile 2020-11-16T22:42:39.229 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:39.230 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj105 td/osd-recovery-space/datafile 2020-11-16T22:42:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:39.372 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj106 td/osd-recovery-space/datafile 2020-11-16T22:42:39.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:39.489 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj107 td/osd-recovery-space/datafile 2020-11-16T22:42:39.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:39.589 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj108 td/osd-recovery-space/datafile 2020-11-16T22:42:39.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:39.698 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj109 td/osd-recovery-space/datafile 2020-11-16T22:42:39.788 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:39.789 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj110 td/osd-recovery-space/datafile 2020-11-16T22:42:39.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:39.890 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj111 td/osd-recovery-space/datafile 2020-11-16T22:42:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj112 td/osd-recovery-space/datafile 2020-11-16T22:42:40.098 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:40.099 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj113 td/osd-recovery-space/datafile 2020-11-16T22:42:40.182 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:40.183 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj114 td/osd-recovery-space/datafile 2020-11-16T22:42:40.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:40.274 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj115 td/osd-recovery-space/datafile 2020-11-16T22:42:40.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:40.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj116 td/osd-recovery-space/datafile 2020-11-16T22:42:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj117 td/osd-recovery-space/datafile 2020-11-16T22:42:40.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:40.557 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj118 td/osd-recovery-space/datafile 2020-11-16T22:42:40.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:40.642 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj119 td/osd-recovery-space/datafile 2020-11-16T22:42:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj120 td/osd-recovery-space/datafile 2020-11-16T22:42:40.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:40.826 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj121 td/osd-recovery-space/datafile 2020-11-16T22:42:40.917 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:40.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj122 td/osd-recovery-space/datafile 2020-11-16T22:42:41.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:41.027 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj123 td/osd-recovery-space/datafile 2020-11-16T22:42:41.152 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:41.153 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj124 td/osd-recovery-space/datafile 2020-11-16T22:42:41.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:41.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj125 td/osd-recovery-space/datafile 2020-11-16T22:42:41.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:41.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj126 td/osd-recovery-space/datafile 2020-11-16T22:42:41.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:41.437 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj127 td/osd-recovery-space/datafile 2020-11-16T22:42:41.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:41.529 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj128 td/osd-recovery-space/datafile 2020-11-16T22:42:41.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:41.612 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj129 td/osd-recovery-space/datafile 2020-11-16T22:42:41.704 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:41.705 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj130 td/osd-recovery-space/datafile 2020-11-16T22:42:41.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:41.795 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj131 td/osd-recovery-space/datafile 2020-11-16T22:42:41.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:41.895 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj132 td/osd-recovery-space/datafile 2020-11-16T22:42:41.995 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:41.996 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj133 td/osd-recovery-space/datafile 2020-11-16T22:42:42.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:42.096 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj134 td/osd-recovery-space/datafile 2020-11-16T22:42:42.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:42.197 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj135 td/osd-recovery-space/datafile 2020-11-16T22:42:42.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:42.289 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj136 td/osd-recovery-space/datafile 2020-11-16T22:42:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj137 td/osd-recovery-space/datafile 2020-11-16T22:42:42.472 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:42.473 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj138 td/osd-recovery-space/datafile 2020-11-16T22:42:42.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:42.573 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj139 td/osd-recovery-space/datafile 2020-11-16T22:42:42.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:42.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj140 td/osd-recovery-space/datafile 2020-11-16T22:42:42.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:42.765 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj141 td/osd-recovery-space/datafile 2020-11-16T22:42:42.866 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:42.867 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj142 td/osd-recovery-space/datafile 2020-11-16T22:42:42.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:42.966 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj143 td/osd-recovery-space/datafile 2020-11-16T22:42:43.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:43.073 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj144 td/osd-recovery-space/datafile 2020-11-16T22:42:43.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:43.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj145 td/osd-recovery-space/datafile 2020-11-16T22:42:43.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:43.267 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj146 td/osd-recovery-space/datafile 2020-11-16T22:42:43.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:43.366 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj147 td/osd-recovery-space/datafile 2020-11-16T22:42:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj148 td/osd-recovery-space/datafile 2020-11-16T22:42:43.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:43.550 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj149 td/osd-recovery-space/datafile 2020-11-16T22:42:43.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:43.643 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj150 td/osd-recovery-space/datafile 2020-11-16T22:42:43.725 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:43.726 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj151 td/osd-recovery-space/datafile 2020-11-16T22:42:43.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:43.827 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj152 td/osd-recovery-space/datafile 2020-11-16T22:42:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:43.911 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj153 td/osd-recovery-space/datafile 2020-11-16T22:42:44.002 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:44.003 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj154 td/osd-recovery-space/datafile 2020-11-16T22:42:44.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:44.120 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj155 td/osd-recovery-space/datafile 2020-11-16T22:42:44.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:44.211 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj156 td/osd-recovery-space/datafile 2020-11-16T22:42:44.303 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:44.304 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj157 td/osd-recovery-space/datafile 2020-11-16T22:42:44.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:44.454 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj158 td/osd-recovery-space/datafile 2020-11-16T22:42:44.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:44.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj159 td/osd-recovery-space/datafile 2020-11-16T22:42:44.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:44.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj160 td/osd-recovery-space/datafile 2020-11-16T22:42:44.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:44.772 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj161 td/osd-recovery-space/datafile 2020-11-16T22:42:44.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:44.872 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj162 td/osd-recovery-space/datafile 2020-11-16T22:42:44.972 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:44.973 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj163 td/osd-recovery-space/datafile 2020-11-16T22:42:45.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:45.064 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj164 td/osd-recovery-space/datafile 2020-11-16T22:42:45.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:45.172 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj165 td/osd-recovery-space/datafile 2020-11-16T22:42:45.272 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:45.273 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj166 td/osd-recovery-space/datafile 2020-11-16T22:42:45.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:45.365 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj167 td/osd-recovery-space/datafile 2020-11-16T22:42:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj168 td/osd-recovery-space/datafile 2020-11-16T22:42:45.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:45.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj169 td/osd-recovery-space/datafile 2020-11-16T22:42:45.624 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:45.625 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj170 td/osd-recovery-space/datafile 2020-11-16T22:42:45.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:45.708 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj171 td/osd-recovery-space/datafile 2020-11-16T22:42:45.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:45.799 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj172 td/osd-recovery-space/datafile 2020-11-16T22:42:45.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:45.882 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj173 td/osd-recovery-space/datafile 2020-11-16T22:42:45.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:45.975 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj174 td/osd-recovery-space/datafile 2020-11-16T22:42:46.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:46.059 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj175 td/osd-recovery-space/datafile 2020-11-16T22:42:46.142 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:46.143 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj176 td/osd-recovery-space/datafile 2020-11-16T22:42:46.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:46.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj177 td/osd-recovery-space/datafile 2020-11-16T22:42:46.342 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:46.343 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj178 td/osd-recovery-space/datafile 2020-11-16T22:42:46.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:46.435 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj179 td/osd-recovery-space/datafile 2020-11-16T22:42:46.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:46.534 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj180 td/osd-recovery-space/datafile 2020-11-16T22:42:46.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:46.644 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj181 td/osd-recovery-space/datafile 2020-11-16T22:42:46.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:46.762 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj182 td/osd-recovery-space/datafile 2020-11-16T22:42:46.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:46.861 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj183 td/osd-recovery-space/datafile 2020-11-16T22:42:46.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:46.945 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj184 td/osd-recovery-space/datafile 2020-11-16T22:42:47.045 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:47.046 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj185 td/osd-recovery-space/datafile 2020-11-16T22:42:47.179 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj186 td/osd-recovery-space/datafile 2020-11-16T22:42:47.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:47.270 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj187 td/osd-recovery-space/datafile 2020-11-16T22:42:47.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:47.362 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj188 td/osd-recovery-space/datafile 2020-11-16T22:42:47.462 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:47.463 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj189 td/osd-recovery-space/datafile 2020-11-16T22:42:47.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:47.554 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj190 td/osd-recovery-space/datafile 2020-11-16T22:42:47.647 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:47.648 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj191 td/osd-recovery-space/datafile 2020-11-16T22:42:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj192 td/osd-recovery-space/datafile 2020-11-16T22:42:47.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:47.831 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj193 td/osd-recovery-space/datafile 2020-11-16T22:42:47.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:47.923 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj194 td/osd-recovery-space/datafile 2020-11-16T22:42:48.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:48.007 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj195 td/osd-recovery-space/datafile 2020-11-16T22:42:48.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:48.090 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj196 td/osd-recovery-space/datafile 2020-11-16T22:42:48.174 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj197 td/osd-recovery-space/datafile 2020-11-16T22:42:48.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:48.266 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj198 td/osd-recovery-space/datafile 2020-11-16T22:42:48.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:48.358 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj199 td/osd-recovery-space/datafile 2020-11-16T22:42:48.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:48.457 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj200 td/osd-recovery-space/datafile 2020-11-16T22:42:48.540 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:48.541 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj201 td/osd-recovery-space/datafile 2020-11-16T22:42:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj202 td/osd-recovery-space/datafile 2020-11-16T22:42:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:48.743 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj203 td/osd-recovery-space/datafile 2020-11-16T22:42:48.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:48.834 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj204 td/osd-recovery-space/datafile 2020-11-16T22:42:48.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:48.918 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj205 td/osd-recovery-space/datafile 2020-11-16T22:42:49.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:49.010 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj206 td/osd-recovery-space/datafile 2020-11-16T22:42:49.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:49.133 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj207 td/osd-recovery-space/datafile 2020-11-16T22:42:49.243 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:42:49.244 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj208 td/osd-recovery-space/datafile 2020-11-16T22:42:49.336 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.402 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj209 td/osd-recovery-space/datafile 2020-11-16T22:59:59.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.661 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj210 td/osd-recovery-space/datafile 2020-11-16T22:59:59.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj211 td/osd-recovery-space/datafile 2020-11-16T22:59:59.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj212 td/osd-recovery-space/datafile 2020-11-16T22:59:59.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.662 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj213 td/osd-recovery-space/datafile 2020-11-16T22:59:59.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj214 td/osd-recovery-space/datafile 2020-11-16T22:59:59.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj215 td/osd-recovery-space/datafile 2020-11-16T22:59:59.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.663 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj216 td/osd-recovery-space/datafile 2020-11-16T22:59:59.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj217 td/osd-recovery-space/datafile 2020-11-16T22:59:59.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj218 td/osd-recovery-space/datafile 2020-11-16T22:59:59.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj219 td/osd-recovery-space/datafile 2020-11-16T22:59:59.664 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj220 td/osd-recovery-space/datafile 2020-11-16T22:59:59.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj221 td/osd-recovery-space/datafile 2020-11-16T22:59:59.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj222 td/osd-recovery-space/datafile 2020-11-16T22:59:59.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.665 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj223 td/osd-recovery-space/datafile 2020-11-16T22:59:59.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj224 td/osd-recovery-space/datafile 2020-11-16T22:59:59.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj225 td/osd-recovery-space/datafile 2020-11-16T22:59:59.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.666 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj226 td/osd-recovery-space/datafile 2020-11-16T22:59:59.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2020-11-16T22:59:59.667 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj227 td/osd-recovery-space/datafile 2020-11-16T22:59:59.870 INFO:tasks.workunit.client.0.smithi189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects)